Challenges and benefits of AI agents

From: aidotengineer

AI agents are generating significant discussion, with experts holding varied opinions on their capabilities and future. While some are highly optimistic about their potential, others voice concerns about their current limitations [00:00:43].

Defining AI Agents

An AI agent is not a new concept [00:01:08]. The recent increase in their power is largely due to the advancement of large language models (LLMs) [00:01:11]. AI agents operate through a cyclical process:

1. Perception: Agents must understand their environment by sensing information from various sources like text, images, audio, video, and touch [00:01:20].
2. Reasoning: After perceiving information, agents process it to understand how to complete tasks, break them down into steps, and identify appropriate tools or actions [00:01:37]. This inner planning often involves a “chain of thought” reasoning, typically powered by LLMs [00:01:59].
3. Reflection (Meta-reasoning): Agents can perform meta-reasoning steps, asking themselves if a choice was correct after executing an action and, if not, going back to reconsider [00:02:10].
4. Actions: These are any performances an agent executes, such as talking to a human or moving from point A to point B [00:02:25]. In essence, agents interact with their environment through actions [00:02:41].

Levels of Autonomy

The difficulty in deploying AI agents can be understood through an analogy to the levels of autonomy in self-driving cars [00:03:02]:

• Level 1 (Chatbot): Similar to initial chatbots (around 2017), focused on retrieving information [00:03:12].
• Level 2 (Agent Assist): An agent assists a human, like a customer service agent using an LLM to suggest responses that still require human approval before sending [00:03:20].
• Level 3 (Agent as a Service): LLMs automate AI workflows, offering services like booking meetings or writing job descriptions [00:03:35].
• Level 4 (Autonomous Agents): An agent can delegate and perform multiple, interrelated tasks, sharing components, knowledge, and resources [00:03:51].
• Level 5 (Jarvis-like): Complete trust in agents, delegating all security measures like keys, allowing agents to perform entirely on behalf of the user [00:04:16].

While self-driving cars are high-risk agents where errors can be fatal, AI agents can be separated into low-risk and high-risk tasks [00:05:08]. Low-risk tasks, such as filing reimbursements, can benefit from human supervision initially and gain trust over time for full automation [00:05:12]. Customer-facing tasks are typically considered high-risk, but the progression from back-office to front-office deployment is anticipated [00:05:27].

Benefits and Potential of AI Agents

Prominent figures are very bullish on AI agents, viewing them as a massive advancement in AI progress [00:00:32].

• Revolution in Computing: Bill Gates sees agents as the “biggest revolution in computing” [00:00:26].
• Significant AI Progress: Andrew Ng describes it as “massive AI progress” [00:00:30].
• Year of the Agent: Sam Altman from OpenAI suggested 2025 could be “the year of agent” [00:00:36].
• Improved Performance: Methods like reflection and tree search can significantly improve an agent’s ability to reason, plan, and perform tasks [00:05:41], [00:17:50].
• Self-Discovery: Agents can self-discover task-specific information, like optimal persuasion strategies in dialogue tasks, diversifying their approaches (e.g., emotional vs. logical appeal) [00:25:08].
• Reduced Human Supervision: Improvements can be achieved without explicit human supervision data, by using synthetic data and on-policy data generation [00:15:06].
• Test-Time Scaling: Existing pre-trained models can achieve better results by being given more steps or “budgets” during inference, allowing for more in-depth reasoning and reflection [00:17:27].
• Broader Application: The ability to perform complex tasks like web browsing (Visual Web Arena) and operating system tasks (OS World) demonstrates the wide applicability of these agents [00:31:54].

Challenges and Solutions in Building AI Agents

Despite the optimism, there are challenges in AI agent development. Some dismiss current agents as mere “simple wrappers of large language models” that “really can’t plan” or solve practical solutions [00:00:46].

Technical Challenges

• Self-Improvement with Smaller Models:
  • Noise Propagation: Smaller LLMs (e.g., LLaMA 7B) can generate “noisy” feedback that propagates into correction steps, hindering improvement (the “blind leading the blind” problem) [00:08:58].
  • Incompatibility of Logics: Feedback generated by larger models or verifiers might not be compatible with the internal logic or demonstration style of smaller models, rendering the feedback useless [00:09:56].
  • Solution: Distillation and Edited Feedback: One approach is to use a larger LLM to edit the smaller model’s feedback, tailoring it to the smaller model’s internal logic, or use external tools like Python scripts for more accurate feedback [00:11:10]. This process, called “self-refine” or “self-improvement,” involves iterating on feedback and corrections [00:08:33].
• Scaling Laws and Data Limitations:
  • The performance of LLMs is linearly correlated with compute, data size, and parameter size [00:16:27].
  • Data Scarcity: There’s a “million-dollar question” on how to generate more and better diversified data, as there’s effectively “one internet” of data [00:16:44].
  • Parameter Limits: Increasing parameters without abundant, diversified data makes models harder to improve [00:17:01].
• Eliciting Better Behaviors/Planning:
  • Pre-training LLMs is resource-intensive and often only feasible for large organizations like Google or OpenAI [00:17:10].
  • Solution: Test-Time Scaling: Instead of re-training, significant improvements can be achieved by giving models more steps or “budget” during inference (test time) to perform processes like “think step by step” or “reflection” [00:17:27].
  • Monte Carlo Tree Search (MCTS): Applied to sequential decision-making tasks (like dialogue or chess), MCTS allows agents to simulate future actions and outcomes to choose the best move [00:20:22].
    • In conversational settings, “Zero-training MCTS” prompts an LLM to act as a policy, simulate action outcomes, and evaluate action quality, while another LLM simulates the user [00:21:12].
    • Open-loop MCTS accounts for human response variance, allowing for stochastic sampling of simulated conversations [00:23:08].
  • Reflective MCTS (RMCTS): This algorithm extends MCTS by incorporating contrastive reflection, allowing agents to learn from past interactions and dynamically improve search efficiency. It also uses multi-agent debate for more reliable state evaluation instead of single prompts [00:28:48].
  • Memory Module: RMCTS equips the system with a memory module to improve behavior across different tasks. Experiences (successes or errors) are saved to a vector database, allowing the agent to retrieve similar reflections for future tasks [00:29:39].
• Learning from Search Processes:
  • Solution: Exploratory Learning: Instead of simple imitation learning (training on the best action found), exploratory learning treats the tree search process as a single trajectory. The model learns how to linearize the search tree traversal, motivating it to learn how to explore, backtrack, and evaluate its own actions [00:33:55]. This enables the model to improve its decision processes independently [00:34:21].

Future Development and Challenges of AI Agents

Ongoing technical challenges in AI agent development include [00:35:55]:

• Developing better reinforcement learning (RL) methods to reduce reliance on the search tree [00:35:57].
• Implementing model predictive control methods to reduce expensive environment setup and interaction [00:36:02].
• Improving control abilities and autonomous exploration for agent orchestration layers [00:36:29].

Beyond single-agent, single-task benchmarks, future challenges and solutions in building AI agents will involve system-level problems when a human wants a model to perform multiple tasks on the same computer [00:37:37]:

• Scheduling [00:37:49].
• Interacting with databases to avoid side effects [00:37:51].
• Enhanced security measures for human handover and supervision requests [00:37:55].
• Multi-user, Multi-agent Planning: This is even more complex, especially in a community setting where multiple humans interact with various agents for different tasks, leading to diverse settings that need quantification [00:38:04].

The aim is to establish more realistic benchmarks that integrate system considerations and algorithms that prioritize not only task completion but also efficiency and security, forming the basis for future applications [00:38:25].