Building effective agents

From: aidotengineer

Barry introduces the topic of building effective agents [00:00:28]. Approximately two months prior, he and Eric co-authored a blog post titled “Building Effective Agents” [00:00:31]. This post shared opinions on what an agent is and isn’t, alongside practical insights gained during their work [00:00:35]. The discussion expands on three core ideas from that blog post [00:00:44]:

1. Don’t build agents for everything [00:00:52].
2. Keep it simple [00:00:55].
3. Think like your agents [00:00:58].

Evolution of AI Systems

The journey to agentic systems began with simple features like summarization, classification, and extraction, which were considered “magic” a few years ago but are now commonplace [00:01:07]. As products matured, developers became more creative, often requiring multiple model calls [00:01:19]. This led to orchestrating these calls in predefined control flows, known as workflows [00:01:26]. Workflows allowed a trade-off between cost and latency for improved performance, marking the beginning of agentic systems [00:01:32].

Currently, models are more capable, leading to the emergence of domain-specific agents in production [00:01:44]. Unlike workflows, agents can determine their own trajectory and operate nearly independently based on environmental feedback [00:01:52].

The future of agentic systems might involve single agents becoming more general-purpose or the development of multi-agent collaborations with delegation [00:02:04]. As systems are given more agency, they become more useful and capable, but also incur higher costs, increased latency, and greater consequences for errors [00:02:17].

When to Build Agents: A Checklist

Agents are ideal for scaling complex and valuable tasks, not as a universal upgrade for every use case [00:02:38]. Workflows remain a valuable and concrete method for delivering value today [00:02:49].

[!INFO] Checklist for Building Agents

• Complexity of Task: Agents excel in ambiguous problem spaces [00:03:01]. If a decision tree can be easily mapped, explicitly building and optimizing it is more cost-effective and provides greater control [00:03:08].
• Value of Task: The exploration involved in agentic tasks consumes many tokens, so the task’s value must justify the cost [00:03:21]. For high-volume, low-budget tasks (e.g., customer support), a workflow for common scenarios might capture most of the value [00:03:31].
• Derisk Critical Capabilities: Ensure there are no significant bottlenecks in the agent’s trajectory [00:04:02]. For a coding agent, this means ensuring it can write good code, debug, and recover from errors [00:04:09]. Bottlenecks multiply cost and latency, suggesting a need to reduce scope and simplify the task [00:04:18].
• Cost of Error and Error Discovery: If errors are high-stakes and difficult to discover, it becomes challenging to trust the agent with autonomy [00:04:32]. Mitigation strategies include limiting scope (e.g., read-only access) or incorporating more human-in-the-loop interaction, though this also limits scalability [00:04:44].

Coding as an Agent Use Case

Coding is an excellent use case for agents [00:05:00]:

• The process from design document to pull request is ambiguous and complex [00:05:03].
• Good code holds significant value [00:05:11].
• Models like Claude are proficient in many parts of the coding workflow [00:05:16].
• Coding output is easily verifiable through unit tests and Continuous Integration (CI) [00:05:23]. This verifiability is a key reason for the proliferation of creative and successful coding agents [00:05:31].

Keep It Simple

Once a suitable use case for agents is identified, the second core idea is to maintain simplicity [00:05:41]. Agents are fundamentally models utilizing tools in a loop [00:05:51].

The three defining components of an agent are [00:05:57]:

1. Environment: The system in which the agent operates [00:06:02].
2. Tools: An interface that allows the agent to take action and receive feedback [00:06:06].
3. System Prompt: Defines the agent’s goals, constraints, and ideal behavior within the environment [00:06:11].

The model is then called in a loop [00:06:20]. It is crucial to keep this structure simple, as complexity upfront significantly hinders iteration speed [00:06:27]. Iterating on these three basic components yields the highest return on investment (ROI), with optimizations to follow later [00:06:31].

Different agent use cases may appear distinct in product surface, scope, and capability, but often share nearly identical backbones and code [00:06:41]. The environment depends on the use case, leaving the choice of tools and system prompt as the primary design decisions [00:07:01].

Once these basic components are established, various optimizations can be applied [00:07:31]:

• Caching trajectories for coding and computer use to reduce cost [00:07:37].
• Parallelizing tool calls for search-heavy agents to reduce latency [00:07:41].
• Presenting agent progress clearly to build user trust [00:07:47].

The advice remains: keep it as simple as possible during iteration, focusing on these three core components first, and then optimize once desired behaviors are achieved [00:07:54].

Think Like Your Agents

A common mistake in developing agents is to approach them from a human perspective, leading to confusion when agents make unexpected errors [00:08:06]. The recommendation is to place oneself in the agent’s context window [00:08:20].

While agents can exhibit sophisticated and complex behaviors, at each step, the model is merely running inference on a highly limited set of contexts [00:08:27]. Everything the model knows about the current state of the world is contained within 10-20k tokens [00:08:37]. Limiting one’s own understanding to this context helps determine if it’s sufficient and coherent, offering a better grasp of how agents perceive the world [00:08:43].

An Agent’s Perspective: Computer Use Case

Imagine being a computer use agent [00:09:00]:

• Input is a static screenshot and a brief, often poorly written, description (e.g., “You are a computer use agent. You have a set of tools and you have a task.“) [00:09:06].
• Despite internal reasoning, only actions taken through tools affect the environment [00:09:19].
• During inference and tool execution, it’s akin to “closing eyes” for 3-5 seconds and operating in the dark [00:09:30].
• Opening “eyes” reveals a new screenshot, but the outcome of the previous action (success or failure) is unknown [00:09:41]. This “lethal phase” perpetuates the cycle [00:09:48].

Performing a full task from an agent’s perspective reveals crucial contextual needs, such as screen resolution for accurate clicks or recommended actions and limitations to guide exploration and prevent unnecessary steps [00:10:04].

Fortunately, since systems speak our language, one can directly query the model (e.g., Claude) [00:10:41]:

• Ask if system prompt instructions are ambiguous or make sense [00:10:47].
• Inquire if the agent understands how to use a tool, or if it needs more/fewer parameters [00:10:52].
• Feed the entire agent’s trajectory back to the model and ask why a decision was made, or how to facilitate better future decisions [00:11:02].

This process, while not replacing human understanding, helps bridge the gap between human and agent perceptions, aiding in building AI agents more effectively [00:11:14].

Personal Musings and Open Questions

Barry shares personal reflections on the evolution of AI agents and open questions for AI engineers [00:11:35]:

• Budget-Aware Agents: Unlike workflows, controlling cost and latency for agents is challenging [00:11:47]. Defining and enforcing budgets (time, money, tokens) is crucial for deploying agents in production and enabling more use cases [00:11:56]. This relates to challenges in creating effective AI agents.
• Self-Evolving Tools: Models are already assisting in iterating on tool descriptions [00:12:14]. This concept could generalize into a meta-tool allowing agents to design and improve their own tool ergonomics [00:12:21]. This would make agents more general-purpose by enabling them to adapt tools for each use case, impacting designing and optimizing agent environments and building AI agents using primitives.
• Multi-Agent Collaborations: There is a strong conviction that multi-agent collaborations will be widespread in production by the end of the year [00:12:38]. These systems offer benefits like parallelism, separation of concerns, and protection of main agent context windows via sub-agents [00:12:46]. A key open question is how these agents will communicate beyond rigid synchronous user-assistant turns, exploring asynchronous communication and enabling more diverse roles for agent interaction [00:12:59]. This is critical for developing AI agents for productivity and agentic workflows.

Key Takeaways

The three core takeaways for building effective agents are [00:13:38]:

1. Don’t build agents for everything [00:13:41].
2. If you find a good use case, keep the agent as simple as possible for as long as possible [00:13:46].
3. As you iterate, think like your agent, gain their perspective, and help them do their job [00:13:51].