What is the impact of KV-cache optimization on offline LLM inference?
KV-cache stores computed key-value attention pairs from previous tokens, avoiding redundant computation during autoregressive generation. Without it, generating N tokens requires O(N^2) compute; with it, O(N). For offline LLMs, KV-cache optimization is critical because memory is limited. Techniques include: grouped-query attention (GQA) reduces cache size by sharing KV heads (Llama 2 uses 8 KV heads vs 32 query heads), multi-query attention (MQA) uses a single KV head, paged attention (vLLM) manages cache like virtual memory pages to eliminate fragmentation, and quantized KV-cache (FP8 or INT4 values) halves memory usage. For a 7B parameter model with 4096 context length, KV-cache consumes 1GB in FP16; with INT4 quantization and GQA, this drops to 125MB, enabling longer contexts on edge devices.