Data set and evaluation alignment

From: aidotengineer

LM evaluations are a critical component of building effective AI systems, ensuring that applications deliver real-world value beyond being mere “fancy demos” [02:03:00]. However, many existing evaluation methods, even those with robust testing frameworks, can lead to meaningless evaluations [00:15:00]. This article explores the challenges and offers solutions for improving AI evaluation methods by ensuring iterative alignment between data sets and evaluators [00:17:00].

The Problem with Current AI Evaluations

Despite the common belief that existing testing frameworks are sufficient, working with hundreds of teams has revealed recurring patterns of issues with evaluation that standard frameworks cannot handle [00:33:00]. Many teams express uncertainty about what constitutes a good evaluation framework [02:44:00].

Fundamentals of Evaluation

An evaluation fundamentally involves testing the quality of an AI system before it’s deployed to production [02:49:00], similar to unit and integration testing in traditional software [02:11:00]. Three key components are necessary for effective evaluation:

1. Agent [02:55:00]: This is the system being evaluated, which could be an end-to-end agent, a small function, or a retrieval pipeline [02:59:00]. Agents vary widely (e.g., customer service chatbot, Q&A agent for legal contracts) and each has unique requirements. For instance, a document Q&A system may need to be accurate, compliant with regulations, explain its reasoning, and understand domain nuances [03:22:00]. Your evaluation needs to account for all these aspects [03:43:00].

2. Data Set [03:48:00]: This is what the agent is evaluated against and is considered the most important component [03:51:00]. Many teams struggle here, relying on limited, hand-written test cases that don’t cover all use cases [03:57:00]. A robust data set must include:

   • Inputs: Queries and requests the system will receive in production [04:12:00].
   • Ideal Outputs: What good responses should look like [04:19:00].
   • Coverage: Not just “happy paths,” but also tricky edge cases where things might go wrong [04:26:00].
   • Domain Expert Input: Examples should be written by experts who understand the business context and can define quality requirements [04:34:00].

3. Evaluators [04:52:00]: This determines how quality is measured [04:55:00].

   • Human Evaluators: Subject matter experts review and score outputs, providing feedback. This is effective but slow and expensive [04:57:00].
   • Code-based Evaluators: Good for objective metrics like response time, latency, or even metrics like ROUGE/BLEU (though their effectiveness is debated) [05:09:00].
   • LM Evaluators: Promise to combine nuanced reasoning with the speed and scalability of automated systems [05:21:00].

These components are dynamic and must evolve over time as the agent improves, data sets need to include more challenging cases, and evaluation criteria become more sophisticated [06:00:00].

The Rise of LM Evaluators

LM evaluators have gained significant popularity, with teams switching their entire evaluation stack to rely on “LM as a judge” [06:19:00]. Their compelling promise includes:

• Speed: Evaluations that previously took 8-10 hours with human evaluation can now be completed in under an hour [06:43:00]. A thousand test cases could be evaluated in 50-60 minutes, a significant improvement over a full day of human work [06:54:00].
• Cost: A traditional human evaluation for 1,000 ratings could cost several hundred dollars via platforms like Mechanical Turk, whereas LM evaluators range from $3to$120, representing a 10x cost reduction [07:19:00].
• Consistency: LM evaluators show over 80% consistency with human judgments, a level comparable to agreement between different human evaluators [07:41:00]. Research papers like “nlg eval” and “Spade” demonstrate strong correlations between human judgments and LLM scores [08:09:00]. Major model providers like OpenAI and Anthropic are increasingly using this approach for alignment [08:20:00].

Major Problems with LM Evaluators

Despite their benefits, LM evaluators have two very major problems:

1. Criteria Drift [08:49:00]: This occurs when an evaluator’s notion of “good” no longer aligns with the user’s [10:41:00]. Popular frameworks often use generalized evaluation criteria not tailored to unique use cases [09:56:00].

   • Example: An LM-based e-commerce recommendation system performed well in testing using standard criteria like context and generation relevance. However, in production, user complaints arose because the evaluator over-indexed on keyword relevance and missed the broader context of product descriptions relevant to user queries, failing to catch real relevance issues [09:21:00].
   • Root Cause: Underlying LM models used for evaluation might change, leading to inconsistent grading [10:30:00].
   • Research: The “EvalGen” paper by Shanker and team at Berkeley highlighted that evaluation criteria must evolve, balancing true and false positives to maximize F1 score alignment with human judgments [10:50:00].

2. Data Set Drift [11:19:00]: This refers to test data sets lacking sufficient coverage of real-world scenarios [11:21:00].

   • Problem: Hand-crafted test cases, while seemingly perfect, often fail to represent the messy, context-dependent, and often complex inputs users provide in production [11:27:00]. Users might ask broader topics, require real-world data, or combine multiple questions unexpectedly [11:53:00].
   • Analogy: It’s like training for a marathon on a treadmill; metrics look good, but the real race includes inclines and varying surfaces that weren’t accounted for [12:26:00]. Your test cases no longer reflect reality [12:40:00].

Solutions: Iterative Alignment for Meaningful Evals

To fix these problems and make evals work for ourselves, the core insight is that evaluators and data sets must be iteratively aligned, much like aligning an LLM application itself [12:56:00].

Here’s a three-step approach for getting your evaluation right:

1. Align Evaluators with Domain Experts [13:19:00]:

   • Have domain experts regularly grade outputs and critique evaluator results [13:25:00].
   • Use their critiques and few-shot examples in the evaluator prompt to ground the evaluator with a real-world notion of what’s good and bad [13:36:00].
   • Iterate on the underlying evaluator prompt, rather than relying on templated library metrics, until satisfactory agreement is reached [13:47:00].

2. Keep Data Sets Aligned with Real-World User Queries [14:09:00]:

   • Log all queries and treat your test bank as a “living, breathing thing” [14:11:00].
   • Automatically (or manually) flow underperforming queries from production back into the test suite, as these “real-world failures” are invaluable for improving the test bank [14:19:00].
   • Continuously add these test cases and their ground truth labels to the test bank [16:45:00].

3. Measure and Track Alignment Over Time [14:31:00]:

   • Use concrete metrics like F1 score for binary judgments or correlation coefficients for Likert scales [14:35:00].
   • Track how well your evaluator matches human judgment with every iteration to understand if it’s truly improving or regressing [14:44:00].

Practical Implementation Steps:

• Customize the LM evaluator Prompt [15:11:00]:

  • Tailor your evaluation criteria carefully [15:22:00].
  • Add few-shot examples of critiques from domain experts [15:25:00].
  • Choose between binary or Likert scales for ratings (binary is highly recommended) [15:31:00].
  • Ensure you are measuring something genuinely meaningful to your use case and business context, not just generic out-of-the-box metrics [15:38:00].

• Involve Domain Experts Early [15:50:00]:

  • Get them to evaluate the evaluator itself [15:53:00].
  • Starting with even 20 examples in a spreadsheet can provide a good sense of alignment between evaluator judgments and expert opinions [16:00:00].

• Log and Iterate [16:17:00]:

  • Continuously add real-world failures from production to your test bank [16:26:00].
  • Iterate on your LM evaluator prompts; they are not static [16:55:00]. Make them more specific to your use case [17:06:00].
  • Consider investing in an “eval console” tool (or building one internally) to allow domain experts to directly iterate on and assess evaluator critiques [17:08:00].

• Continuous Measurement [17:23:00]:

  • Set up a simple dashboard to track alignment scores (F1, correlation metrics) over time [17:28:00]. This systematic tracking informs whether evaluator templates are improving [17:44:00].

Conclusion

The goal is continuous improvement, not perfection [18:00:00]. LM evaluations are only as good as their alignment with real-world usage [18:07:00]. Avoid the trap of static evaluation; LLMs require iterative feedback loops in development processes to ensure meaningful improvement [18:13:00].