What is the role of reranking in improving RAG quality?
Reranking is a second-stage retrieval step that reorders the initial set of retrieved documents using a more accurate but computationally expensive model. The first stage retrieves a larger candidate set typically 20-50 documents using fast vector search. The reranker then scores each candidate against the query using cross-attention between the query and document, producing more accurate relevance scores than embedding similarity alone. Popular reranking models include Cohere Rerank, BGE-reranker, and cross-encoder models from the sentence-transformers library. The reranker typically reduces the set to the top 3-5 most relevant documents for the LLM context. This two-stage approach combines the speed of vector search with the accuracy of cross-encoder scoring. Reranking consistently improves RAG answer quality by 10-25% on retrieval benchmarks compared to vector search alone.