How do you scale RAG systems to handle millions of documents?

Scaling RAG to millions of documents requires optimization at every layer. For vector storage, use distributed vector databases like Milvus, Qdrant, or Pinecone that shard data across multiple nodes. Implement HNSW indexes with appropriate M and efConstruction parameters balancing recall against index build time and memory. Use dimension reduction through PCA or Matryoshka embeddings to reduce storage and speed up search. Partition the index by document category, date, or source for filtered queries. For ingestion, implement parallel processing pipelines that chunk, embed, and index documents concurrently. Use batch embedding with GPU acceleration. For retrieval, implement a multi-stage pipeline with fast approximate search followed by precise reranking. Cache frequent queries and their results. Monitor index performance metrics and rebuild indexes when recall degrades. Implement incremental updates rather than full re-indexing when documents change.