How do you optimize RAG for low-latency applications?

Low-latency RAG optimization targets each pipeline stage. Embedding: use smaller models (all-MiniLM-L6-v2 at 22ms vs BGE-large at 95ms per query), batch queries when possible, and cache frequent query embeddings. Retrieval: use approximate nearest neighbor (ANN) indexes like HNSW with optimized efsearch parameters, implement result caching with semantic similarity-based cache keys, and pre-filter by metadata before vector search. Reranking: use distilled rerankers (ms-marco-MiniLM-L-6-v2 vs L-12), limit candidates to top-10, and run on GPU for 3-5x speedup. Generation: use streaming responses so users see output immediately, select faster LLMs for time-sensitive queries (GPT-3.5-turbo at 60ms TTFT vs GPT-4 at 300ms), reduce context length by selecting only top-3 chunks instead of top-5. Architecture: deploy embedding and reranking models on dedicated GPU instances, colocate vector database with application servers, implement connection pooling, and use gRPC instead of REST for internal communication. Caching strategy: L1 cache for exact query matches, L2 cache for semantically similar queries (embed and compare), L3 cache for common sub-queries.