How do you implement reranking to improve RAG retrieval quality?

Reranking is a second-stage retrieval step that reorders initial results using a more accurate but slower model. Why rerank: initial retrieval (bi-encoder embedding similarity) is fast but approximate; reranking (cross-encoder that jointly processes query and document) is slower but significantly more accurate. Pipeline: retrieve top-k candidates (k=20-50) using fast vector search, then rerank to select top-n (n=3-5) using a cross-encoder model. Cross-encoder models: ms-marco-MiniLM-L-12 (fast, good quality), BGE-reranker-v2 (excellent quality), and Cohere Rerank (API-based, highest quality). Implementation: retrieve candidates from vector store, pass each candidate with the query through the cross-encoder which outputs a relevance score, sort by score, and select top results. Performance: cross-encoder reranking typically improves NDCG@10 by 10-25% over embedding-only retrieval. Latency: reranking 20-50 candidates takes 50-200ms on GPU, which is acceptable for most applications. Optimization: batch reranking for throughput, cache reranking results for frequent queries, and implement early stopping (if top result has very high confidence, skip reranking remaining candidates). Alternatives: ColBERT (late interaction model that is faster than cross-encoders while approaching their quality), listwise reranking (rerank all candidates simultaneously rather than pairwise), and LLM-based reranking (use an LLM to score relevance, highest quality but most expensive). Evaluation: compare retrieval quality with and without reranking on your evaluation set, measure latency impact, and determine if the quality improvement justifies the additional compute cost.