Iterative improvement of evaluation processes

From: aidotengineer

Evaluations are fundamental to building AI systems that deliver real-world value, moving beyond mere “fancy demos” [01:57:00]. Just as traditional software relies on unit and integration testing before deployment, AI applications require robust evaluations to ensure quality before pushing changes to production [02:09:00].

Components of an Effective Evaluation Framework

To test the quality of an AI application before production, three key components are needed [02:49:00]:

1. Agent: This is the system or function being evaluated, which could be an end-to-end agent, a small function within an agent, or a retrieval pipeline [02:55:00]. Different agents, such as a customer service chatbot or a Q&A agent for legal contracts, have unique requirements and challenges [03:08:00]. For example, a document Q&A system might need to be accurate, compliant with regulations, explain its reasoning, and understand nuanced financial accounting standards, all of which the evaluation must account for [03:22:00].
2. Data Set: The dataset is what the agent is evaluated against [03:48:00]. It must include both inputs (types of queries/requests the system will receive in production) and ideal outputs (what good responses should look like) [04:10:00]. These datasets should cover not only “happy paths” but also tricky edge cases where things might go wrong, and ideally, be written by domain experts who understand the business context [04:24:00].
3. Evaluators: This component determines how quality is measured [04:52:00].
   • Human Evaluators: Traditionally, subject matter experts review and score outputs, providing feedback, but this is slow and expensive [04:57:00].
   • Code-based Evaluators: Effective for objective metrics like response time or latency [05:09:00].
   • LLM Evaluators: These promise to combine the nuance of human reasoning with the speed and scalability of automated systems [05:21:00].

The Rise of LLM Evaluators

LLM evaluators have gained significant popularity, with many teams switching their entire evaluation stack to rely on LLMs as judges [06:19:00]. Their main promises are compelling:

• Speed: Evaluations that took 8-10 hours with human evaluators can now be completed in under an hour [06:43:00]. For example, 1,000 test cases that might take a full day with Mechanical Turk could be evaluated in 50-60 minutes with an LLM evaluator [06:54:00].
• Cost: A traditional human evaluation through Mechanical Turk for 1,000 ratings could cost several hundred dollars, whereas LLM evaluators range from $3to$120, representing a 10x cost reduction [07:19:00].
• Consistency: LLM evaluators show over 80% consistency with human judgments [07:41:00]. Research papers like NLG Eval and SPADE show strong correlations between human judgments and LLM scores [08:09:00], with major model providers increasingly using this direction for alignment [08:20:00].

Challenges and Limitations of LLM Evaluators

Despite their advantages, LLM evaluators face two major problems [08:42:00]:

Criteria Drift

This occurs when an evaluator’s notion of what constitutes “good” no longer aligns with the user’s perception of quality [10:41:00]. Popular frameworks often use generalized evaluation criteria that may not measure what is crucial for a unique use case [09:10:00]. For instance, an e-commerce recommendation system might pass initial evaluations based on context or generation relevance, but fail in production because the evaluator over-indexed on keyword relevance instead of the broader product description and user query context, leading to user complaints [09:21:00]. Criteria drift can also happen if the underlying LLM model used for evaluation changes, leading to inconsistent grading [10:30:00]. Research suggests that evaluation criteria must evolve over time to balance true positives with false positives and maximize alignment with human judgments [10:50:00].

Data Set Drift

This problem arises when datasets lack sufficient test coverage, meaning they don’t accurately represent real-world usage [11:19:00]. Hand-written test cases, even if initially “golden,” often don’t hold up in beta or production when real users provide messy, context-dependent inputs or ask broader, more complex, or combined questions [11:27:00]. The system’s metrics might still look good on the existing test cases, but it fails in reality because the test cases no longer reflect actual conditions [12:15:00].

Strategies for AI Evaluation and Troubleshooting: Iterative Alignment

The solution lies in ensuring that evaluators and datasets are iteratively aligned, similar to how an LLM application itself is aligned [12:53:00].

Three-Step Approach for Iterative Improvement:

1. Align Evaluators with Domain Experts: Have domain experts regularly grade outputs and critique evaluator results [13:19:00]. Incorporate their feedback and “few-shot examples” into the evaluator prompt to ground it in a real-world understanding of good and bad [13:36:00]. Continuously iterate on the evaluator prompt, moving beyond templated metrics, until a satisfactory level of agreement is reached [13:47:00].
2. Keep Data Sets Aligned with Real-World User Queries: Treat the test bank as a “living, breathing thing” [14:09:00]. Log underperforming queries from production and automatically or manually flow them back into the test suite [14:19:00].
3. Measure and Track Alignment Over Time: Use concrete metrics like F1 score for binary judgments or correlation coefficients for Likert scales [14:31:00]. Tracking how well the evaluator matches human judgment with every iteration helps determine if the evaluator is truly improving or regressing [14:44:00].

Practical Implementation Steps:

• Customize the LLM Evaluator Prompt: This is considered the most important step [15:11:00]. Tailor evaluation criteria, add few-shot examples of critiques from domain experts, and decide between binary or Likert scales (binary is highly recommended) [15:22:00]. Ensure that the prompt measures what is truly meaningful to the specific use case, application, and business context, rather than relying on out-of-the-box metrics [15:38:00].
• Involve Domain Experts Early: Have domain experts evaluate the evaluator itself, even starting with 20 examples in a spreadsheet, to gauge alignment with their judgments and inform future prompt changes [15:52:00].
• Log and Update Test Bank: Continuously log underperforming queries from production and add them, along with ground truth labels, to the test bank [16:17:00]. These real-world failures are invaluable for identifying where the evaluation system falls short [16:32:00].
• Iterate LLM Evaluator Prompts: Evaluator prompts are not static; they must evolve [16:55:00]. Test new versions against the expanding test bank, making them more specific to the use case [17:01:00]. Investing in or building an “eval console” can help domain experts iterate on prompts and assess agreement with evaluator judgments [17:10:00].
• Systematic Measurement: It’s crucial to track alignment scores (F1, correlation metrics) over time using a dashboard to systematically monitor evaluator improvement [17:23:00]. This process mirrors how the original LLM application’s prompt is tested [17:50:00].

Conclusion

The ultimate goal is continuous improvement, not perfection [18:00:00]. LLM evaluations are only as effective as their alignment with real-world usage [18:07:00]. Avoid static evaluation—LLMs don’t work with a “set it and forget it” approach [18:14:00]. Instead, build iterative feedback loops into the development process for significant payoff [18:24:00].