Addressing criteria drift and data set drift in evaluations

From: aidotengineer

Effective evaluation is crucial for building AI systems that deliver real-world value beyond mere demonstrations [00:01:57]. While traditional software development relies on unit and integration testing, AI applications require robust evaluation frameworks to ensure quality before deployment [00:02:11]. Many teams, despite having testing setups, often face uncertainty about what constitutes a good evaluation framework [00:02:39].

Components of AI Evaluation

To test quality before production, three key components are needed:

1. Agent [00:02:55]: This is whatever is being evaluated, ranging from an end-to-end agent or a small function within it, to a retrieval pipeline [00:02:59]. Agents, such as customer service chatbots or Q&A systems for legal contracts, have unique requirements like accuracy, compliance, explainability, and nuance [00:03:10].
2. Data Set [00:03:48]: This component is what the agent is evaluated against [00:03:53]. It should include both inputs (production queries/requests) and ideal outputs, covering not just typical scenarios but also tricky edge cases where issues might arise [00:04:12]. These examples should be written by domain experts who understand the business context and quality requirements [00:04:34].
3. Evaluators [00:04:50]: This defines how quality is measured [00:04:55].
   • Human Evaluators: Subject matter experts review and score outputs, providing feedback, but this method is slow and expensive [00:05:00].
   • Code-Based Evaluators: Effective for objective metrics like response time or latency [00:05:09].
   • LLM Evaluators: These promise to combine the nuance of human reasoning with the speed and scalability of automated systems [00:05:21].

These three components are dynamic and must evolve over time as the agent improves, the data set grows, and evaluation criteria become more sophisticated [00:05:57].

The Rise of LLM Evaluators

LLM evaluators have gained significant popularity, with teams switching their entire evaluation stack to rely on LLMs as judges [00:06:19]. Their main promises are compelling:

• Speed: Evaluations that previously took 8-10 hours with human evaluators can now be completed in under an hour [00:06:43]. For a thousand test cases, human evaluations might take a full day, while an LLM evaluator could finish in 50-60 minutes [00:06:54].
• Cost: A traditional human evaluation for a thousand ratings might cost several hundred dollars, whereas LLM evaluators range from $3to$120, representing a 10x reduction in cost [00:07:19].
• Consistency: LLM evaluators show over 80% consistency with human judgments [00:07:45]. This level of consistency is comparable to the agreement between different human evaluators [00:07:55]. Research papers like NLG Eval and SPADE show strong correlations between human judgments and LLM scores [00:08:12]. Major model providers like OpenAI and Anthropic are also increasingly using LLM evaluators for alignment [00:08:20].

Challenges and Limitations of Traditional Evaluation Methods

Despite their advantages, LLM evaluators face two major problems that can render evaluations meaningless [00:08:42]:

Criteria Drift

Criteria drift occurs when an evaluator’s notion of “good” no longer aligns with the user’s perception of quality [00:10:41]. This often happens because popular frameworks (e.g., Ragas, Prompts, LangChain) use built-in evaluation criteria designed for generalizability, not specific use cases [00:09:10].

For example, an AI startup building an LLM-based recommendation system for e-commerce found that while their evaluator checked standard boxes like context relevance and generation relevance, it missed crucial user requirements for relevance in production [00:09:29]. The evaluator focused too heavily on keyword relevance, failing to consider the broader context of product descriptions in relation to user queries [00:09:58]. Additionally, inconsistent grading can occur if the underlying LLM model for the evaluator changes or is not a stable version [00:10:30]. Research, such as the EvalGen paper by Shanker and his team at Berkeley, highlights that evaluation criteria must evolve over time to balance true positives and false positives and maximize F1 score alignment with human judgments [00:10:50].

Data Set Drift

Data set drift refers to when test data sets lack sufficient coverage and no longer represent real-world usage patterns [00:11:19]. Developers might spend weeks crafting perfect test cases with clear queries and expected answers, but these curated tests often fail to hold up when real users introduce messy, context-dependent inputs [00:11:27].

Users commonly ask questions broader than anticipated, require real-world data like search API results, or combine multiple questions in unexpected ways [00:11:55]. This means that while metrics might look good on existing test cases, they don’t reflect actual performance. It’s akin to training for a marathon on a treadmill without accounting for real-world factors like incline or surface traction [00:12:26].

Steps to Create Effective Evaluations for AI Applications

The key insight to address these challenges is that evaluators and data sets need to be iteratively aligned, similar to how an LLM application itself is aligned [00:12:59].

Here is a three-step approach for iterative improvement of evaluation processes:

1. Align Evaluators with Domain Experts:

   • Have domain experts regularly grade outputs, not just once, but continuously [00:13:25].
   • Encourage experts to critique evaluator results, identifying what the evaluator misses or overemphasizes [00:13:30].
   • Use these critiques and few-shot examples in the evaluator prompt to ground it in a real-world understanding of quality [00:13:38].
   • Continuously iterate on the evaluator prompt itself, rather than solely relying on templated metrics [00:13:47].

2. Keep Data Sets Aligned with Real-World User Queries:

   • Log real-world usage and treat the test bank as a “living, breathing” entity [00:14:13].
   • Automatically flow underperforming queries from production back into the test suite, either manually or via automation [00:14:19]. These real-world failures are “golden” opportunities to improve the test bank and identify where the evaluation system falls short [00:16:32].
   • Add ground truth labels to these new test cases to continuously improve the test bank [00:16:47].

3. Measure and Track Alignment Over Time:

   • Use concrete metrics like F1 score for binary judgments or correlation coefficients for Likert scales [00:14:35].
   • Track how well the evaluator matches human judgment with every iteration [00:14:44]. This systematic tracking informs whether the evaluator is truly improving or regressing [00:14:50].
   • Set up a simple dashboard to monitor alignment scores [00:17:30].

Practical Implementation Steps

• Customize LLM Evaluator Prompts: Avoid relying solely on templated metrics [00:15:16]. Carefully tune evaluation criteria, add few-shot examples of critiques from domain experts, and choose between binary or Likert scales (binary often recommended) [00:15:22]. Ensure that what is being measured is meaningful to the specific use case, application, and business context [00:15:38].
• Involve Domain Experts Early: Get domain experts to evaluate the evaluator itself [00:15:53]. Even starting with 20 examples in a spreadsheet can provide a good sense of whether evaluator judgments align with expert opinions and inform necessary changes [00:16:00].
• Iterate LLM Evaluator Prompts: Evaluator prompts are not static; they must evolve over time [00:16:55]. Test new versions against the expanding test bank and make them more specific [00:17:03].
• Invest in an Eval Console: Build or utilize a tool that allows domain experts to iterate on evaluator prompts and assess agreement with evaluator critiques and judgments [00:17:08].

The goal is not perfection, but continuous improvement [00:18:00]. LLM evaluations are only as effective as their alignment with real-world usage [00:18:07]. Teams should avoid static evaluation approaches and instead integrate iterative feedback loops into their development processes to achieve significant payoffs in evaluation quality [00:18:24].

NOTE

For tools to implement this workflow, consider platforms like HoneyHive.ai [00:18:40].