From: aidotengineer
The Open RAG Eval project, which provides tools for quick and scalable RAG evaluation, is grounded in research conducted in collaboration with the University of Waterloo [00:00:32].
Jimmy Lynn Lab Partnership
Specifically, the research backing the Open RAG Eval project was performed with the Jimmy Lynn lab at the University of Waterloo [00:00:36]. This collaboration contributes to the project’s methodology, particularly in the development of its evaluation metrics [00:00:32].
Research-Backed Metrics
One key example of this collaboration’s impact is the Umbrella metric for RAG retrieval evaluation [00:01:44]. Research from the University of Waterloo lab of Jim Lynn indicates that using the Umbrella approach for RAG evaluation without golden answers correlates well with human judgment, providing reliable results even without golden chunks [00:02:32].