How do you implement reranking in a RAG pipeline and why is it important?

Reranking is a second-stage retrieval step that reorders initial results using a more powerful cross-encoder model. The retriever fetches top-k candidates (typically k=20-50) quickly using bi-encoder embeddings, then the reranker scores each query-document pair individually with full cross-attention. This two-stage approach combines retrieval speed with reranking accuracy. Implementation: use Cohere Rerank, BGE-reranker-v2-m3, or cross-encoder/ms-marco-MiniLM-L-12-v2. Pass the query and each retrieved chunk as a pair to the reranker, sort by relevance score, and take the top-n (typically n=3-5) for the LLM context. Reranking typically improves answer quality by 15-25% on benchmarks because it catches semantic nuances that embedding similarity misses. For latency optimization: run reranking on GPU with batched inference, limit to top-20 candidates, and use a smaller reranker model (6-layer vs 12-layer) for low-latency applications. Cost-benefit: reranking adds 50-200ms latency but significantly reduces hallucination from irrelevant context.