How do you evaluate and measure RAG system quality?

RAG evaluation covers three dimensions: retrieval quality, generation quality, and end-to-end performance. Retrieval metrics: precision@k (fraction of retrieved chunks that are relevant), recall@k (fraction of relevant chunks that are retrieved), Mean Reciprocal Rank (inverse rank of first relevant result), and NDCG (normalized discounted cumulative gain, accounts for ranking quality). Generation metrics: faithfulness (is the answer supported by retrieved context, not hallucinated), relevancy (does the answer address the user's question), completeness (does the answer cover all aspects of the question), and correctness (is the factual content accurate). Automated evaluation: use LLM-as-judge to score responses on dimensions above (GPT-4 correlates well with human judgment when given clear rubrics), RAGAS framework provides automated metrics (context precision, context recall, faithfulness, answer relevancy), and DeepEval offers additional evaluation metrics. Human evaluation: create a golden test set of 100-200 question-answer pairs with identified relevant source documents, have domain experts rate system outputs, calculate inter-annotator agreement, and track quality over time. Retrieval evaluation: for each test question, annotate which chunks should be retrieved, measure retrieval against these annotations, and analyze failure cases (was the relevant information chunked poorly, embedded poorly, or ranked poorly). End-to-end testing: measure user satisfaction scores, track answer acceptance rate (did the user ask a follow-up indicating the answer was insufficient), and monitor escalation to human support. Continuous monitoring: track metric trends over time, alert on quality degradation, and A/B test system changes before full rollout.