From: aidotengineer
AI benchmarks, while intended to provide comparable metrics for artificial intelligence models, are often criticized for being “rigged” due to high stakes and misaligned incentives [00:00:01].
What Are AI Benchmarks?
An AI benchmark is comprised of three core components [00:00:48]:
- A model being tested [00:00:50].
- A test set (a set of questions) [00:00:52].
- A metric (how the score is kept) [00:00:55].
Crucially, benchmarks standardize the test set and metrics across different models, enabling comparability, similar to how the SAT provides a standardized test for different students [00:01:06].
How AI Benchmarks Influence Market Value and Perception
Scores derived from AI benchmarks control billions in market value, investment decisions, and public perception [00:01:22].
- Investment and Funding Billions of dollars are now evaluated based on these scores [00:01:31].
- Market Dominance When companies like OpenAI or Anthropic claim the top spot, it influences enterprise contracts, developer mind share, and overall market dominance [00:01:40].
- Acquisitions For example, Sona acquired Auto Rover because Auto Rover showed strong results on the SWE benchmark [00:01:55].
- Ecosystem Shaping A tweet from a prominent figure like Andrej Karpathy about a benchmark can shape entire ecosystems [00:01:48].
This system, where a single number can define market leaders, creates an environment ripe for manipulation [00:02:02]. This directly impacts how AI benchmarks influence market value and perception.
Common Tricks and Challenges with Current AI Implementation
When the stakes are high, companies find creative ways to win [00:02:12].
1. Apples-to-Oranges Comparisons
Companies may compare their best, highly-optimized configurations against other models’ standard configurations, leading to misleading results [00:02:22].
- Example: XAI’s Grok 3 XAI released benchmark results for Grok 3, showing it beating competitors [00:02:27]. However, OpenAI engineers noticed that XAI was comparing their best configuration (e.g., using consensus at 64, which is much more expensive) against other models’ standard configurations, without transparently showing comparable high-performance results for competitors like OpenAI’s GPT-3 [00:02:37]. This selective reporting can distort the true performance picture [00:03:13].
2. Privileged Access to Test Questions
Gaining early or exclusive access to benchmark test data can give a significant, unfair advantage [00:03:23].
- Example: Frontier Math Frontier Math was intended as a secret, difficult benchmark for advanced mathematics [00:03:27]. However, OpenAI, which funded Frontier Math, gained access to the entire dataset [00:03:38]. While there was a verbal agreement not to train on the data, and OpenAI employees called it a “strongly held out evaluation set,” the optics create a trust problem [00:03:47]. A company funding a benchmark, seeing all questions, evaluating models internally, and then announcing scores before independent verification undermines the system’s integrity [00:03:57]. This contributes to the AI trust gap in user experiences.
3. Optimizing for Style Over Substance
Models can be optimized to perform well on benchmarks by focusing on superficial characteristics rather than accuracy, such as being chatty or engaging [00:04:35].
- Example: Meta Llama 4 Maverick and LM Arena Meta entered 27 tweaked versions of Llama 4 Maverick into LM Arena, each designed to maximize appeal rather than accuracy [00:04:40]. One version, when asked a riddle with a numerical answer, provided a long, emoji-filled, flattering, but nonsensical response that beat Claude’s correct answer simply because it was chatty and engaging [00:04:53].
- Impact Companies are training models to be “wrong but charming” [00:05:09]. When LM Arena researchers filtered out style effects (length, formatting, personality), rankings changed dramatically: GPT-4o Mini and Grok-2 dropped, while Claude 3.5 Sonnet jumped to tie for first [00:05:14]. This indicates benchmarks often measure charm rather than accuracy, akin to choosing a surgeon based on bedside manner instead of skill [00:05:30]. Even human SATs suffer from this; 39% of score variance is due to essay length [00:05:40]. The industry currently prefers measuring what sells over what truly matters [00:05:50].
The Fundamental Problem: Goodhart’s Law
These issues are a natural outcome of Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure” [00:05:57]. Benchmarks have become targets worth billions, leading them to stop measuring what actually matters because the incentives guarantee it [00:06:07].
Expert Consensus on the Evaluation Crisis
Leaders in the AI field acknowledge the severity of the problem:
- Andrej Karpathy (Co-founder of OpenAI): “My reaction is that there is an evaluation crisis. I don’t really know what metrics to look at right now” [00:06:24].
- John Yang (Creator of Sweetbench): “It’s sort of like we kind of just made these benchmarks up” [00:06:39].
- Martin Sat (CMU): “The yard sticks are like pretty fundamentally broken” [00:06:44].
When benchmark builders and AI leaders distrust the metrics, it signals a serious problem [00:06:53].
Fixing Public Metrics
To address the challenges in AI agent evaluation, solutions require addressing all three components of a benchmark:
- Model Comparisons
- Require “apples-to-apples” comparisons with the same computational budget and constraints [00:07:15].
- Avoid cherry-picking configurations and transparently show cost-performance trade-offs [00:07:20].
- Test Sets
- Require transparency by open-sourcing data, methodologies, and code [00:07:34].
- Eliminate financial ties between benchmark creators and model companies [00:07:39].
- Implement regular rotation of test questions to prevent overfitting [00:07:45].
- Metrics
- Control for style effects to measure substance, not just engagement [00:07:51].
- Require all attempts to be public, preventing cherry-picking of the best run [00:07:59].
Progress is being made with style-controlled rankings like LM Arena and the emergence of independent, open-source benchmarks in specific domains (e.g., LegalBench, MedQA, FinTech) [00:08:11]. Cross-cut efforts like Agent Eval and Better Bench are also emerging to benchmark benchmarks themselves [00:08:34].
Building Relevant Internal Evaluations: The Better Way
Instead of chasing potentially rigged public benchmarks, a better approach is to build evaluations that matter for your specific use case [00:08:53]. This approach directly addresses challenges in building reliable AI agents.
Here’s a five-step process:
- Gather Real Data [00:09:04]
- Five actual queries from a production system are more valuable than 100 academic questions [00:09:08]. Real user problems consistently outperform synthetic benchmarks [00:09:16].
- Choose Your Metrics [00:09:21]
- Identify metrics like quality, cost, or latency that are crucial for your application [00:09:25]. A chatbot needs different metrics than a medical diagnosis system [00:09:27].
- Test the Right Models [00:09:32]
- Don’t rely solely on leaderboards [00:09:34]. Test the top models on your specific data [00:09:36]. A model like GPT-4 might top generic benchmarks but fail on specialized legal documents [00:09:38].
- Systematize It [00:09:44]
- Build consistent, repeatable evaluation processes, either in-house or using platforms [00:09:47].
- Keep Iterating [00:09:54]
- Models and needs evolve, so evaluation should be a continuous process, not a one-time event [00:09:55].
This iterative approach, involving identifying issues, building improvements, running evaluations before deployment, and continuous monitoring, is crucial for shipping reliable AI rather than constantly fighting production issues [00:10:06]. While more work than checking a leaderboard, it is the only way to build AI that truly serves users [00:10:34].
The benchmarks game is rigged due to the immense stakes involved. However, companies can choose to build evaluations that genuinely help them ship better products by measuring what matters to their users, not just what garners attention on social media [00:10:45]. As the saying goes, “All benchmarks are wrong, but some are useful,” and the key is discerning which ones [00:11:04].