When comparing two different retrievers or vector search configurations for RAG, what retrieval evaluation criteria should we look at to determine which one is better?
When evaluating retrievers or vector search configurations for RAG, focus on three key criteria: retrieval accuracy, efficiency, and robustness to query variation. These metrics help determine which configuration better aligns with your application’s needs.
First, prioritize retrieval accuracy. Measure precision at k (how many of the top-k retrieved documents are relevant) and recall at k (how many total relevant documents appear in the top-k results). For example, if your RAG system uses the top 5 documents, precision@5 tells you if those results are trustworthy. Mean Reciprocal Rank (MRR) is also useful—it rewards systems that rank the first relevant document higher (e.g., a correct answer in position 1 is better than position 3). Use labeled test queries with known relevant documents to compute these metrics. If one retriever achieves 80% precision@5 versus another at 60%, the former is likely better for tasks requiring high-quality inputs for the generator.
Second, assess efficiency. Latency (time per query) and throughput (queries processed per second) are critical for real-time applications. For instance, a brute-force exact search might have perfect accuracy but be too slow for user-facing apps, while an approximate method like HNSW sacrifices minimal accuracy for faster results. Resource usage (memory, CPU/GPU) also matters—large indices may not fit in memory or scale cost-effectively. Compare configurations under realistic load: a retriever that takes 50ms with 90% accuracy might be preferable to one taking 200ms with 95% accuracy, depending on your latency budget.
Finally, test robustness. A good retriever should handle diverse query types, ambiguous phrasing, and edge cases. Evaluate performance across query categories (e.g., fact-based, opinion-seeking) and measure consistency via metrics like standard deviation in precision@k across subsets. For example, a configuration might excel on technical queries but fail on colloquial language. Additionally, check if retrieved documents cover diverse aspects of a query (e.g., for “health benefits of exercise,” ensure results address mental, physical, and social impacts). Use clustering or similarity scores between retrieved items to quantify diversity. A robust retriever minimizes “silent failures” where irrelevant results go undetected until the generator produces a flawed answer.
By balancing accuracy, efficiency, and robustness, you can objectively compare configurations and choose the one that best supports your RAG pipeline’s requirements.