Tubegraph

Building trust in AI systems

5 min read

From: aidotengineer

Building trust in AI systems, especially for mission-critical decisions in fields like healthcare, requires robust evaluation systems that can operate at scale without room for error [00:00:04]. While creating an MVP product with LLMs is becoming easier, scaling to serve customers at large volumes introduces significant challenges [00:00:30].

Challenges at Scale

As request volumes increase, so does the number of previously unseen edge cases [00:00:44]. For instance, in medical prior authorization, an AI might incorrectly interpret medical nuance, leading to a wrong decision that could occur rarely (e.g., every 1,000 or 10,000 cases) but accumulates to many mistakes when processing over 100,000 cases daily [00:00:50]. Such errors in healthcare can lead to legal action against organizations using AI inappropriately [00:02:04].

Human Review: A Necessary But Unscalable Step

A foundational step in identifying and handling failure cases is performing human reviews of AI outputs [00:02:16]. Companies like Anterior build internal clinical teams and specialized tooling, such as their “Scalp” review dashboard, to facilitate efficient human review of medical records and guidelines against AI-generated answers [00:02:21]. Reviewers can critique incorrect answers, which then helps generate “ground truths” (descriptions of the correct answer) for offline evaluations [00:03:00].

However, the primary problem with human reviews is their lack of scalability [00:03:19]. To maintain consistent review percentages (e.g., 50% or even 5%) as the volume of decisions grows from thousands to hundreds of thousands per day, the required number of human reviewers quickly becomes unsustainable, often exceeding the size of an entire company [00:03:46]. This leaves two critical questions: which cases should be reviewed, and how well did the AI perform on unreviewed cases [00:04:28]?

Limitations of Offline Evaluation Data Sets

Offline evaluation datasets, built from ground truths generated by human reviews, are helpful for defining gold standard datasets, segmenting performance by enterprise or medical condition, and iterating AI pipelines [00:04:39]. However, relying solely on them is risky because new edge cases, especially in areas with high data heterogeneity like medical records, will continually emerge [00:05:21]. Waiting for these new cases to be represented in offline datasets means it might be too late to address issues, which is akin to “playing with fire” [00:05:11].

Real-Time Reference-Free Evaluation Systems

The solution to scaling challenges and promptly addressing new edge cases is a real-time, reference-free evaluation system [00:05:40]. “Reference-free” (or “label-free”) means the evaluation occurs before the true outcome or human review is known, enabling immediate response to issues as they arise [00:05:45].

Leveraging LLMs as Judges

A powerful starting point for reference-free evaluation is using an LLM as a judge [00:06:07]. In this approach, the output from the main AI pipeline is fed into a separate LLM judge along with a scoring system [00:06:10]. This scoring system can evaluate various aspects, such as:

Anterior, for example, uses this to grade the confidence of its AI’s binary output (approval or escalation for review) [00:06:36]. This confidence grading, along with predicted correct outputs, can be used in several ways:

  1. Estimate Performance Across All Cases: Predict estimated performance on all cases in real-time, not just those reviewed by humans, enabling immediate response and feedback to customers [00:07:29].
  2. Compute Alignment: Compare reference-free evaluation outputs with human review outputs to gauge system alignment and trust [00:07:50].
  3. Dynamic Prioritization for Human Review: Combine confidence grading with contextual factors (e.g., cost of procedure, risk of bias, previous error rates) to dynamically prioritize the most relevant cases with the highest probability of error for human review [00:08:03].

This creates a “virtuous cycle” where human reviews validate and improve performance, while reference-free evals dynamically prioritize cases, allowing the system to continually improve its ability to detect and respond to edge cases, a process often described as “validating the validator” [00:08:24]. This iterative, data-driven process of processing high volumes of real data builds a system that is difficult for competitors to replicate [00:08:50].

Integrating Evals into the AI Pipeline

Once confidence in the system’s performance is established, the reference-free evaluation can be integrated directly into the AI pipeline [00:09:01]. Based on the evaluation output, the system can either return a confident response to the customer or trigger a “further action” [00:09:14]. This action might involve:

  • Sending the case to another LLM pipeline using more expensive models [00:09:23]
  • Internal human review by an on-call clinician [00:09:28]
  • Surfacing the case to the customer’s own review dashboard [00:09:34]

This mechanism ensures customer trust in the AI’s outputs [00:09:38].

Impact and Key Principles

Implementing such an evaluation system has profound impacts:

  • Reduced Overhead: It eliminates the need for an ever-expanding team of human experts. For example, Anterior reviews tens of thousands of cases with a team of less than 10 clinical experts, compared to a competitor that hired over 800 nurses [00:09:49].
  • Strong Alignment: It achieves high alignment between AI and human reviews, comparable to the alignment seen between human reviewers themselves [00:10:08].
  • Rapid Response: Errors are quickly identified and corrected, ensuring customer SLAs (Service Level Agreements) are met and maintaining confidence in results [00:10:18].
  • Industry-Leading Performance: Leads to provably superior performance, such as an F1 score of nearly 96% in prior authorization, fostering deep customer trust and satisfaction [00:10:35].

Principles for Effective AI Implementation and Trust Building

  1. Build a System, Not Just an Audit: Don’t just use review data to audit performance; use it to build, audit, and improve the auditing and evaluation system itself [00:11:04].
  2. Evaluate on Live Production Data: Avoid relying solely on offline evaluations. Identify and respond to problems immediately using real-time data [00:11:19].
  3. Empower the Best Reviewers: Prioritize the quality of reviews over quantity. Build custom tooling if necessary to help reviewers work faster and more effectively [00:11:28].

This approach enables building an evaluation system that provides real-time performance estimates, scales to meet demand while maintaining low costs, and is powered by a small, focused team of experts. It allows AI products to move from MVP to serving customers and maintaining their trust at scale [00:11:35].

Graph View

Backlinks