How do you optimize RAG for low-latency production environments?

Optimizing RAG latency requires addressing each pipeline stage. For embedding, use cached embeddings for common queries, run embedding models on GPU for sub-10ms encoding, and consider smaller models like all-MiniLM at 384 dimensions for faster encoding. For retrieval, use HNSW indexes for sub-millisecond approximate nearest neighbor search, implement connection pooling to the vector database, and use pre-filtering to reduce the search space. For reranking, limit candidates to 10-20 documents and use distilled reranker models. For generation, implement streaming responses so users see results incrementally, use prompt caching for common prefixes, and choose appropriately sized LLMs. Architectural optimizations include running the embedding model, vector database, and LLM on the same infrastructure to minimize network hops, implementing request-level caching for repeated queries, and pre-computing embeddings for known high-traffic queries. Target end-to-end latency under 2 seconds for interactive applications.