What is semantic caching in RAG and how does it reduce latency?

Semantic caching stores the results of previous RAG queries and matches new queries against cached ones based on semantic similarity rather than exact string matching. When a new query is semantically similar to a previously answered query above a confidence threshold, the cached response is returned immediately without running the full RAG pipeline. Implementation involves maintaining an embedding index of previous query-response pairs, computing similarity between new queries and cached queries, and returning cached results when similarity exceeds the threshold. This dramatically reduces latency for common or repeated queries. Cache invalidation strategies include time-based expiry, source document update tracking, and periodic cache refresh. The hit rate depends on query diversity with customer-facing applications often achieving 30-50% cache hit rates. GPTCache and LangChain provide semantic caching implementations. Balance the similarity threshold to avoid returning stale or slightly mismatched results.