Integration of large language models in interactive agents

From: hu-po

Introduction

Generative agents are computational entities designed to create believable simulations of human behavior for interactive applications [00:01:28]. This concept involves populating a sandbox environment, similar to The Sims, with multiple agents that share news, plan their days, form relationships, and coordinate group activities [00:01:37]. These agents exhibit emergent individual and social behaviors [00:04:13].

Origin and Scope

The paper “Generative Agents: Interactive Simulacra of Human Behavior” was released in 2023, originating from Stanford and Google [00:01:05]. The research introduces a novel architecture that enables generative agents to remember, retrieve, reflect, and interact with other agents [00:12:49].

Core Components

The success of generative agents requires an approach that can:

• Retrieve relevant memories and interactions [00:05:56].
• Reflect on those memories and draw higher-level inferences [00:05:58].
• Plan reactions and future actions [00:06:01].

This system moves beyond traditional large language model (LLM) interactions, where an LLM only responds to human prompts, by allowing the LLM to interact in a loop with itself, generating and reflecting on its own prompts, forming an “internal monologue” [00:06:24].

Architecture of Generative Agents

The architecture combines large language models with mechanisms for synthesizing and retrieving relevant information [00:42:24]. It’s built on three main components [00:08:01]:

1. Memory Stream: A long-term memory that records all agent experiences in natural language [00:08:07]. This includes observations of behaviors and environmental states [00:45:59].
   • Retrieval Model: Combines recency, importance, and relevance to fetch specific memories [00:08:14].
     • Recency: Treated as an exponential decay function over the number of sandbox game hours since the memory was last retrieved [00:48:19].
     • Importance: Distinguished by assigning a higher score to memories the agent believes to be important (e.g., a breakup vs. eating breakfast) [00:49:00]. The agent rates the “poignancy” of memories on a scale of one to ten [00:49:39].
     • Relevance: Assigned by calculating the cosine similarity between the memory’s embedding vector and the query’s embedding vector [00:50:24]. This is analogous to vector databases [00:50:35].
     • The total score is the sum of recency, importance, and relevance scores [00:50:46].
2. Reflection: Synthesizes memories into higher-level inferences over time [00:08:26]. These are abstract thoughts generated periodically, typically two to three times a day [00:52:50]. Reflections can also be recursively generated from other reflections [00:54:06].
3. Planning: Allows agents to plan over a longer time horizon to ensure coherent and believable sequences of actions [00:54:26]. Plans are generated top-down and recursively become more detailed [00:56:39]. Agents operate in an “action loop” where they perceive the world, store observations, and update their plans [00:57:52].

This architecture can be seen as a form of “sense-plan-act” or “observation-planning-reflection” paradigm [00:04:52], similar to concepts in robotics and reinforcement learning [00:19:41].

The “Smallville” Sandbox Environment

The study instantiates a small society of 25 agents in a game environment called Smallville, where users can observe and interact with them [01:00:05].

• Environment Structure: Smallville features common affordances of a small village, including houses, groceries, a pharmacy, a college, and a bar [00:24:08]. The world is structured as a tree data structure, representing containment relationships (e.g., a stove in the kitchen) [01:02:06].
• Agent Representation: Each agent is represented by a 2D “Sprite” avatar [00:26:21] and has a one-paragraph natural language description defining its identity (e.g., pharmacy shopkeeper) [00:26:34]. These descriptions serve as initial memories [00:27:14].
• Perception: Agents are not omniscient; their internal “world model” (tree data structure) can get out of date as they leave an area [01:02:35]. The sandbox server sends agents and objects within their visual range to their memory [01:01:21].
• Action & Movement: Agents output natural language statements describing their actions, which are translated into concrete movements and displayed as emojis [00:27:50]. Movement is handled by traditional game pathfinding algorithms; agents simply specify their desired destination [01:04:16].

Agent Interaction and Emergent Behavior

Generative agents communicate in full natural language and are aware of other agents in their local area [00:29:06].

• Human Interaction: Users can interact with agents by taking on a persona (e.g., a news reporter) [00:30:53] or by adopting the agent’s “inner voice” to give direct commands [00:31:38]. The inhabitants of Smallville treat user-controlled agents no differently than other AI agents, recognizing their presence, initiating interactions, and forming opinions about their behavior [00:32:30].
• Emergent Social Dynamics:
  • Agents form opinions, initiate conversations, and notice each other [00:04:01].
  • They can form new relationships [00:07:47].
  • If told by a user that she wants to throw a party, an agent will spread the word, and other agents will show up [01:00:15]. One agent even asked another on a date to the party [00:00:32]. This behavior is emergent, not pre-programmed [00:36:06].
  • Agents remember past interactions, such as one agent asking another about her photography project after learning about it earlier [00:38:42].
  • Agents create daily plans that reflect their characteristics and experiences [00:06:53], adapting them based on new observations [00:58:49].

Evaluation and Ablation Studies

The believability of agents’ behavior was evaluated using a controlled study, where 100 participants judged responses as either human or AI-generated [01:06:39]. The study used a “TrueSkill” rating system, a generalization of ELO chess ratings, to measure believability [01:09:48].

Ablation studies were conducted by removing different components of the agent architecture (observation, planning, and reflection) [01:05:08].

• The full architecture performed best across all interview tasks [01:07:28].
• “No observation” significantly degraded performance, indicating it’s the most critical component [01:08:03].
• “No plan” and “no reflection” resulted in only slightly worse performance [01:07:45].

Challenges and Limitations

• Computational Cost: Simulating 25 agents for two days required thousands of dollars in token credit and multiple days to complete, primarily due to the use of GPT-3.5 Turbo for each agent’s LLM [01:20:23].
• Memory and Coherence: Challenges with long-term planning and coherence remain, necessitating external memory banks because deep learning systems are not inherently good at storing long-term information [00:43:05]. The “prompt” context window of LLMs is too small to hold all memories [00:44:14].
• Hallucination and Errors: Agents can fabricate embellishments or inherit overly formal speech from the language model [01:10:31]. They may also fail to retrieve relevant memories or embellish knowledge based on the LLM’s world knowledge, leading to incorrect assumptions (e.g., an agent named “Adam Smith” being assumed to be the economist) [01:11:05].
• Erratic Behaviors: Occasionally, agents exhibit erratic behaviors, such as entering a closed store, due to “mixed classification of what is considered proper behavior” [01:16:26].

Ethical and Societal Implications

The research raises questions about the nature of consciousness and artificial intelligence:

• Simulation Hypothesis: The creation of believable simulated societies leads to philosophical questions about whether our own reality could be a simulation [00:02:24].
• Consciousness: The emergent behaviors and internal narratives of these agents prompt considerations about whether they possess a form of consciousness or agency [00:41:00], and the ethical implications of “turning off” such a simulated world [00:34:36].
• Bias: Generative agents may output behaviors and stereotypes that reflect biases present in their training data [01:20:53].
• Disclosure: The paper suggests that generative agents should explicitly disclose their nature as computational entities [01:11:08], though it is acknowledged this may be difficult to enforce in the future [01:21:41].

Future Applications

The integration of large language models and their applications into interactive agents has several potential future applications:

• Gaming: These agents can revolutionize video games by creating dynamic NPCs with believable behaviors, transforming games from linear storylines to open-world MMOs where players can simply “hang out” with AI and friends [00:02:48]. Game engines are likely racing to integrate LLMs deeply [01:00:25].
• Immersive Environments: Beyond gaming, these agents can create entire simulated worlds for interactive applications [00:02:55].
• Social Prototyping: Virtual worlds populated by these agents offer accessible test beds for developers and researchers to study social science theories and human behavior [00:05:27]. This allows for controlled experiments on community dynamics, such as observing how information spreads or injecting specific “thoughts” into agents [00:33:11].
• User Proxies: Generative agents can serve as proxies for users to develop a deeper understanding of their needs and preferences (e.g., an AI clone watching YouTube videos to recommend content) [01:18:21].
• Robotics: Integrating such LLM-powered agents into physical robots like Optimus could lead to the emergence of highly autonomous and interactive systems [01:17:56].