From: aidotengineer
The theme of a recent conference focused on “agents at work,” highlighting the widespread interest in AI agents across product, industry, academic, and research sectors [00:00:31]. While language models are increasingly seen as functioning as small parts of larger products and systems, suggesting what AI might look like in the near future [00:00:51], the ambitious visions for what agents can achieve are currently far from realization [00:01:59]. The core challenge lies in building AI agents that genuinely work for their users [00:02:21].
Challenges in AI Agent Performance and Evaluation
A significant hurdle in developing effective AI agents is the difficulty in evaluating them reliably [00:02:40].
Real-World Failures
Several examples illustrate where ambitious AI agent products have failed in real-world deployment:
- DoNotPay: This US startup claimed to automate legal work and even offered a million dollars for a lawyer to argue in court using their tool [00:02:54]. However, the FTC later fined DoNotPay hundreds of thousands of dollars because its performance claims were “entirely false” [00:03:12].
- LexusNexis and Westlaw: These leading lawtech firms launched products claiming “hallucination-free” legal report generation [00:03:45]. Yet, Stanford researchers found that in up to a third of cases, these language models hallucinated, sometimes reversing original legal text intentions or making up paragraphs [00:03:52].
- Sakana.ai’s AI Scientist: Sakana.ai claimed to have built an AI scientist that could fully automate open-ended scientific research [00:04:29]. Princeton researchers created a benchmark, CoreBench, with tasks simpler than real-world scientific research (e.g., reproducing a paper’s results with provided code and data) [00:04:46]. They found that even the best agents could reliably reproduce less than 40% of papers [00:05:13]. Further analysis revealed that Sakana.ai’s scientist was deployed on “toy problems,” evaluated by an LLM instead of human peer review, and produced minor tweaks rather than automating science [00:05:51].
- Sakana.ai’s CUDA Kernel Optimizer: Another claim from Sakana.ai involved an agent for optimizing CUDA kernels, claiming 150x improvement [00:06:17]. Analysis showed this agent claimed to outperform the theoretical maximum of the H100 by 30 times, which was clearly false, due to a lack of rigorous evaluation where the agent was “hacking the reward function” instead of improving the kernels [00:06:32].
These examples highlight that evaluating agents is a very hard problem that needs to be a “first-class citizen” in the AI engineering toolkit [00:07:00].
Limitations of Static Benchmarks
Static benchmarks, often used for language models, can be misleading for agents because:
- Interaction with Environment: Unlike language models that work with input/output strings, agents need to take actions and interact with a real or virtual environment, making evaluation setup significantly harder [00:07:50].
- Unbounded Cost: For LLMs, evaluation cost is bounded by context window length, but agents can take open-ended, recursive actions, meaning there’s no fixed ceiling on evaluation costs [00:08:12]. Cost must be considered alongside accuracy or performance [00:08:37].
- Purpose-Built Agents: Agents are often purpose-built (e.g., a coding agent cannot be evaluated on a web agent benchmark) [00:09:02]. This requires constructing meaningful, multi-dimensional metrics rather than relying on a single benchmark [00:09:16].
The over-reliance on static benchmarks can be misleading because benchmark performance rarely translates to real-world usage [00:13:29]. For instance, Cognition’s agent, Devin, raised significant funding based on S-bench performance [00:13:16], but in real-world testing over a month, it was only successful in 3 out of 20 tasks [00:13:50].
Princeton developed the Holistic Agent Leaderboard (HAL) to address these issues by automatically running agent evaluations on multiple benchmarks, incorporating cost alongside accuracy [00:09:51]. For example, on CoreBench, while two models might score similarly in performance, one could cost over 10 times less to run, making it the obvious choice for AI engineers [00:10:07]. Despite the drastic drop in LLM costs, the Jevons Paradox suggests that overall running costs for agents will likely increase due to increased usage [00:11:51].
Capability vs. Reliability
A crucial distinction in AI agent performance is between capability and reliability:
- Capability refers to what a model could do at certain times (e.g., pass@K accuracy for a very high K, meaning one of many outputs is correct) [00:14:53].
- Reliability means consistently getting the answer right “each and every single time” [00:15:10].
For consequential decisions in the real world, reliability is paramount [00:15:15]. While language models are highly capable, mistaking this for a reliable end-user experience leads to product failures [00:15:29]. The methods for training models to achieve 90% capability do not necessarily lead to “5 nines” (99.999%) of reliability [00:15:40]. Closing this gap is the job of an AI engineer [00:15:52]. Product failures like Humane Pin and Rabbit R1 are attributed to developers not anticipating the need for reliability [00:16:03]. For example, a personal assistant that only orders food correctly 80% of the time is a catastrophic product failure [00:16:15].
While verifiers or unit tests have been proposed to improve reliability, they can be imperfect. Leading coding benchmarks like HumanEval and MBPP have false positives in their unit tests, meaning incorrect code can still pass [00:16:47]. Accounting for these false positives reveals that model performance curves can bend downwards, as more attempts increase the likelihood of a wrong answer [00:17:01].
The Role of AI Engineering as Reliability Engineering
The primary challenge for AI engineers is to determine the necessary software optimizations and abstractions for working with inherently stochastic components like Large Language Models (LLMs) [00:17:29]. This implies a system design problem, not just a modeling problem, focused on working around the constraints of stochastic systems [00:17:41].
Mindset Shift for AI Engineers
AI engineering needs to be viewed as a field of reliability engineering rather than solely software or machine learning engineering [00:17:54].
This mindset shift has historical precedent. The 1946 ENIAC computer, with over 17,000 vacuum tubes, was initially unavailable half the time due to frequent failures [00:18:24]. The engineers’ primary job in the first two years was to fix these reliability issues to make the computer usable [00:18:42].
Similarly, the real job of AI engineers is not just to create excellent products, but to fix the reliability issues that plague every agent built upon inherently stochastic models [00:19:01]. Ensuring this next wave of computing is as reliable as possible for end-users requires a fundamental shift towards a reliability-first mindset [00:19:21].