Tubegraph

LM evaluation challenges

7 min read

From: aidotengineer

Evaluating Large Language Models (LLMs) is crucial for building AI systems that deliver real-world value beyond fancy demos [01:57:00]. Despite common beliefs that existing evaluation frameworks are robust, recurring patterns suggest that many evaluations might be meaningless [00:00:15]. This article discusses the fundamental components of evaluation, the rise of LM evaluators, and the significant challenges they pose, along with practical solutions for overcoming them.

Fundamentals of Evaluation

An evaluation in AI applications is analogous to unit and integration testing in traditional software development [02:09:00]. Just as changes aren’t pushed to production without tests, AI applications shouldn’t be updated without proper evaluations [02:17:00].

A robust evaluation framework requires three key components to test quality before deployment:

  1. Agent [02:55:00] This is the AI system or component being evaluated, which could be an end-to-end agent, a small function, or a retrieval pipeline [02:59:00]. Examples include customer service chatbots, Q&A agents for legal contracts, or document Q&A systems [03:10:00]. Each agent has unique requirements, such as accuracy, compliance with regulations, explainability, or nuance in specific domains (e.g., financial accounting standards) [03:19:00].
  2. Data Set [03:48:00] This component serves as the benchmark against which the agent is evaluated [03:53:00]. A common pitfall is relying on a few handwritten test cases that do not cover all use cases or edge cases [03:57:00]. An effective data set must include:
    • Inputs: Queries and requests that mirror real-world production scenarios [04:12:00].
    • Ideal Outputs: Examples of what good and bad responses should look like [04:19:00].
    • Edge Cases: Scenarios where the system might fail [04:26:00]. These examples should be written by domain experts who understand the business context and can define quality requirements [04:34:00].
  3. Evaluators [04:52:00] Evaluators are methods used to measure quality [04:55:00].
    • Human Evaluators: Subject matter experts review outputs and provide scores and feedback [04:57:00]. While effective, this method is slow and expensive [05:05:00].
    • Code-based Evaluators: Suitable for objective metrics like response time or latency [05:09:00].
    • LM Evaluators: Promise a balance of nuanced reasoning (like humans) and the speed/scalability of automated systems [05:21:00].

These three components are dynamic and must evolve as the agent improves, the data set expands with more challenging cases, and evaluation criteria become more sophisticated [05:57:00].

Rise of LM Evaluators

LM evaluators have gained significant popularity, with many teams switching their entire evaluation stack to rely on “LM as a judge” [06:22:00]. Their compelling advantages include:

  • Speed: Evaluations that took 8-10 hours with human evaluators can now be completed in under an hour [06:43:00]. For example, processing a thousand test cases takes about a full day with Mechanical Turk but only 50-60 minutes with an LM evaluator (assuming sequential execution) [06:54:00].
  • Cost: A traditional human evaluation via Mechanical Turk for a thousand ratings can cost several hundred dollars [07:21:00]. LM evaluators, depending on the model chosen, can cost anywhere from 120 for the same number of ratings, representing a 10x cost reduction [07:28:00].
  • Consistency: LM evaluators show over 80% consistency with human judgments [07:45:00]. This consistency is comparable to the agreement rates observed between different human evaluators, who also do not agree 100% of the time [07:54:00]. Research papers like NLG Eval and Spade have demonstrated strong correlations between human judgments and LLM scores, and major model providers are increasingly using this approach for alignment [08:09:00].

LM Evaluation Challenges

Despite their benefits, LM evaluators face two major challenges that can render evaluations meaningless [08:40:00]:

The Uncomfortable Truth [08:40:00]

LM evaluators have two significant problems: Criteria Drift and Data Set Drift.

Criteria Drift

Criteria drift occurs when an LM evaluator’s definition of “good” no longer aligns with the user’s perception of quality [08:49:00]. While popular frameworks (e.g., Ragas, Promptfoo, LangChain) provide built-in evaluation criteria, these are often designed for generalizability and may not measure what is crucial for a specific use case [09:01:00].

For example, an AI startup built an LM-based recommendation system for e-commerce, and their evaluator checked standard metrics like context relevance and generation relevance [09:29:00]. While results looked good in testing, user complaints surfaced in production because the evaluator overemphasized keyword relevance without understanding the broader product description and user query context [09:48:00]. This led to a complete miss of actual relevance issues [10:12:00]. Additionally, inconsistent grading can occur if the underlying LLM model used by the evaluator changes (e.g., an unstable OpenAI version) [10:30:00]. Research like Shanker and team at Berkeley’s “EvalGen” paper highlights that evaluation criteria need to evolve, balancing true positives and false positives to maximize F1 score against human judgments [10:50:00].

Data Set Drift

Data set drift refers to a lack of test coverage, where the evaluation data set no longer accurately represents real-world user queries [11:19:00]. Teams might spend weeks creating perfect test cases with clear queries and obvious answers [11:27:00]. However, once launched in beta, real users often provide context-dependent, messy inputs that the meticulously crafted test suite fails to account for [11:38:00].

Common usage patterns that lead to data set drift include:

  • Users asking questions broader than anticipated test cases [11:55:00].
  • Queries requiring real-world data like SERP API results [12:02:00].
  • Users combining multiple questions in unforeseen ways [12:09:00].

In such cases, metrics might still appear favorable because the evaluator is happily scoring on the outdated test cases, but these tests no longer reflect reality [12:20:00].

Solutions: Fixing Meaningless Evals

The core insight for effective evaluation is that evaluators and data sets must be iteratively aligned, similar to how an LLM application itself is aligned [12:56:00].

A three-step approach for effective evaluations:

  1. Align Your Evaluators with Domain Experts [13:19:00]
    • Have domain experts regularly grade outputs and critique the evaluator’s results [13:25:00].
    • Use their critiques and few-shot examples to refine the evaluator prompt, ensuring it aligns with a real-world understanding of quality [13:38:00]. This involves significant iteration and “massaging” of the prompt [13:46:00].
  2. Keep Your Data Sets Aligned with Real-World User Queries [14:09:00]
    • Log production underperforming queries and automatically or manually feed them back into the test suite [14:19:00]. Your test bank should be a living, breathing entity [14:16:00].
  3. Measure and Track Alignment Over Time [14:31:00]
    • Use concrete metrics like F1 score (for binary judgments) or correlation coefficients (for Likert scales) to track how well your evaluator matches human judgment with each iteration [14:35:00]. This provides critical information on whether the evaluator is improving or regressing [14:50:00].

Practical Steps

Implementing these solutions requires sustained effort, but it’s far less work than dealing with the consequences of a meaningless evaluation [14:56:00].

  1. Customize the LM Evaluator Prompt: This is the most important step [15:11:00]. Avoid relying on generic, templated metrics [15:17:00]. Carefully tailor evaluation criteria, add few-shot critique examples from domain experts, and decide between binary or Likert scales (binary is highly recommended) [15:22:00]. Ensure that what’s being measured is truly meaningful to your specific use case and business context [15:38:00].
  2. Involve Domain Experts Early: Get experts to evaluate the evaluator itself [15:53:00]. Starting with just 20 examples in a spreadsheet can provide a good sense of whether evaluator judgments align with domain experts, helping inform prompt improvements [15:59:00].
  3. Start with Logging Production Underperformance: Read your logs [16:17:00]. Every time the system underperforms in production, it’s an opportunity to improve the test bank [16:26:00]. These real-world failures are invaluable for identifying where the evaluation system falls short and for continuously adding ground truth labels to the test bank [16:32:00].
  4. Iterate Your LM Evaluator Prompts: Evaluator prompts are not static [16:55:00]. Continuously test new versions against your expanding test bank and make them more specific to your use case [17:01:00].
  5. Invest in an Eval Console: Build or use a tool that allows domain experts to directly iterate on evaluator prompts and assess agreement with critiques and judgments [17:10:00].
  6. Continuous Measurement: You cannot improve what you do not measure [17:23:00]. Set up a dashboard to track alignment scores (F1 score or correlation metrics) over time [17:30:00]. This systematic tracking informs whether evaluator templates are improving, similar to how an LM application’s prompt is tested [17:47:00].

Ultimately, LM evaluations are only as good as their alignment with real-world usage [18:07:00]. Avoid static evaluation approaches; instead, build iterative feedback loops into the development process. This continuous improvement yields significant payoffs in effectively improving evaluations over time [18:24:00].

Graph View

Backlinks