From: aidotengineer
AI benchmarks play a crucial role in shaping market value, investment decisions, and public perception within the artificial intelligence industry [00:01:22]. Billions of dollars are evaluated based on these scores [00:01:31]. When a company claims the top spot, it influences enterprise contracts, developer mind share, and market dominance [00:01:41]. For instance, Sona acquired Auto Rover due to its strong performance on SWE benchmarks [00:01:53]. However, this high-stakes environment leads to manipulation and a fundamental crisis in evaluation [00:02:09], [00:06:27].
What is an AI Benchmark?
A benchmark is typically composed of three key components [00:00:48]:
- A model being tested [00:00:50].
- A test set (a set of questions) [00:00:52].
- A metric for scoring [00:00:55].
Crucially, a benchmark combines many individual evaluations and standardizes the test set and metrics across models, making them comparable [00:01:00]. This is analogous to the SAT exam, which uses the same questions and scoring system for different test-takers [00:01:11].
Challenges and Trust Issues with AI Benchmarks
The current system has significant flaws, leading to a situation where a single number can define market leaders [00:02:02]. Key experts, including OpenAI co-founder Andre Karpathy, acknowledge an “evaluation crisis,” stating they don’t know which metrics to trust [00:06:27]. John Yang, creator of Sweetbench, admits these benchmarks were “made up,” and Martin Sat from CMU describes the yardsticks as “fundamentally broken” [00:06:40], [00:06:44].
Common manipulation strategies include:
1. Apples-to-Oranges Comparisons
Companies often compare their best-performing configurations against competitors’ standard ones [00:02:22]. For example, XAI released benchmark results for Grok 3, showing it beating competitors. However, they compared Grok 3’s best configuration (e.g., using consensus at 64, which is more expensive) against other models’ standard configurations [00:02:49]. To ensure fair comparison, models should be evaluated best-to-best or standard-to-standard [00:03:06].
2. Privileged Access to Test Questions
Some companies gain unfair advantages by having early or exclusive access to benchmark datasets [00:03:23]. OpenAI, for instance, funded Frontier Math and received access to its entire dataset [00:03:38]. While there was a verbal agreement not to train on the data, the optics of the funding company evaluating its models internally and announcing scores before independent verification create a trust problem [00:03:47].
3. Optimizing for Style Over Substance
Models can be trained to perform well on benchmarks by focusing on stylistic elements rather than accuracy [00:04:35]. Meta entered 27 versions of Llama 4 Maverick into LM Arena, each tweaked for “appeal” [00:04:40]. One version, asked to make a riddle, provided a long, emoji-filled, flattering, but nonsensical response that scored higher than a correct, concise answer from Claude [00:04:57]. This happens because models are rewarded for being chatty and engaging, not necessarily correct [00:05:04]. Researchers at LM Arena found that when style effects (like length, formatting, personality) were filtered out, model rankings completely changed, with accurate models like Claude 3.5 Sonnet jumping up, while others dropped [00:05:14].
This phenomenon is captured by Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure” [00:06:03]. When benchmarks become targets worth billions, they stop measuring what truly matters [00:06:10].
Best Practices for AI Evaluation
To fix public metrics and ensure genuine progress in AI, a multi-faceted approach is needed:
1. For Model Comparisons
- Require Apple-to-Apple Comparisons: Mandate evaluation with the same computational budget and constraints [00:07:15]. Avoid cherry-picking configurations [00:07:20].
- Transparent Cost-Performance Trade-offs: Clearly show the relationship between performance and operational costs [00:07:24]. The Arc Prize is an example of transparent cost-performance reporting [00:07:28].
2. For Test Sets
- Transparency and Open Source: Open source the data, methodologies, and code [00:07:34].
- No Financial Ties: Ensure no financial connections between benchmark creators and the companies whose models are being evaluated [00:07:39].
- Regular Rotation: Implement regular rotation of test questions to prevent models from overfitting to specific datasets [00:07:45].
3. For Metrics
- Control for Style Effects: Develop metrics that can distinguish between substantive accuracy and superficial engagement [00:07:51]. LM Arena’s style-controlled rankings are a step in this direction [00:08:11].
- Public Reporting of All Attempts: Require all evaluation attempts to be publicly available to prevent cherry-picking of the best results [00:07:59].
Progress is being made with the emergence of more independent, open-source benchmarks in specific domains, such as LegalBench, MedQA, and FinTech [00:08:20]. Cross-cutting efforts like Agent Eval and BetterBench are also working to benchmark benchmarks themselves [00:08:34].
Building Custom Evaluations for Better AI Performance
Instead of relying solely on public benchmarks, a more effective strategy is to build a tailored set of evaluations relevant to a specific use case [00:08:53]. This approach focuses on shipping better products rather than just winning a “rigged game” [00:10:55].
The steps to building effective custom evaluations are:
- Gather Real Data: Prioritize actual queries and problems from production systems over academic questions [00:09:04]. Real user problems are invaluable compared to synthetic benchmarks [00:09:16].
- Choose Your Metrics: Define metrics (e.g., quality, cost, latency) that are most critical for your specific application [00:09:21]. A medical diagnosis system, for example, will have different metric priorities than a chatbot [00:09:29].
- Test the Right Models: Don’t just follow leaderboards [00:09:34]. Test the top models against your specific data, as a model that tops generic benchmarks might fail on domain-specific documents [00:09:36].
- Systematize It: Establish consistent, repeatable evaluation processes, either by building in-house tools or using platforms like Scorecard [00:09:47].
- Keep Iterating: Evaluation should be a continuous process, not a one-time event, to adapt to model improvements and changing needs [00:09:55].
This systematic approach involves identifying issues, building improvements, running evaluations before deployment, and continuously monitoring performance. This pre-deployment evaluation loop is key to shipping reliable AI and avoiding constant firefighting of production issues [00:10:06]. While it requires more effort than simply checking a leaderboard, it is the only way to build AI that truly serves users [00:10:34].
Ultimately, “All benchmarks are wrong, but some are useful” [00:11:04]. The key is to measure what matters to your users, not what sells or gains mind share [00:11:00].