What are the best practices for RAG evaluation and measuring retrieval quality?

RAG evaluation operates at two levels: retrieval quality and generation quality. Retrieval metrics: Recall@k (fraction of relevant docs in top-k), Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain (NDCG@k), and Hit Rate. Generation metrics: faithfulness (is the answer grounded in retrieved context?), answer relevance (does it address the question?), and context relevance (are retrieved chunks pertinent?). Frameworks: RAGAS provides automated evaluation using LLM-as-judge for faithfulness, context precision, and answer correctness. Build a golden test set of 200+ question-answer-context triples covering diverse query types. Automated testing pipeline: embed evaluation in CI/CD, track metrics over time, set regression thresholds (flag if faithfulness drops below 0.85). Red-teaming: test adversarial queries, out-of-domain questions, and ambiguous queries. Key insight: measure retrieval and generation independently to diagnose whether failures come from poor retrieval, poor generation, or both. Monitor in production using implicit feedback (user thumbs up/down, follow-up questions, session abandonment).