From: aidotengineer
Citation faithfulness and hallucination detection are two key metrics within the open rag eval open-source project, designed for quick and scalable RAG (Retrieval-Augmented Generation) evaluation without the need for golden answers [00:01:06]. These metrics are part of a research-backed effort in collaboration with the University of Waterloo [00:00:32].
Citation Faithfulness
Citation faithfulness is a metric that measures the accuracy of citations provided in a RAG system’s response [00:01:55]. Specifically, it determines whether the citations within the response are correct [00:01:57]. It assesses if the cited passage or information is:
- Fully supported by the answer [00:03:37]
- Partially supported by the answer [00:03:39]
- Not supported by the answer [00:03:41]
This metric helps ensure the high fidelity of information presented by the RAG pipeline [00:03:35].
Hallucination Detection
Hallucination detection is a metric used to identify instances where a RAG system generates content that does not align with its retrieved source material [00:02:01]. This detection is powered by Victara’s dedicated hallucination detection model [00:02:01]. The goal of this metric is to verify if the entire response generated by the RAG system aligns accurately with the content it retrieved [00:03:48].