How do you optimize KV cache memory for long-context offline inference?
KV cache optimization is critical for long-context offline inference. Grouped-query attention reduces KV heads from the standard multi-head attention, cutting cache size by 4-8x. KV cache quantization stores cache values in INT8 or INT4 instead of FP16, halving or quartering memory usage with minimal quality impact. Sliding window attention like in Mistral limits the cache to a fixed window size regardless of total context. PagedAttention manages cache memory in discrete blocks, eliminating fragmentation. Flash attention computes attention incrementally without materializing the full attention matrix. Ring attention distributes long sequences across multiple devices. Token merging and pruning remove less important past tokens from the cache. Combining these techniques enables processing 100K+ token contexts on consumer hardware.