What is FlashAttention and how does it improve offline inference performance?
FlashAttention is an algorithm that computes exact attention while being IO-aware, meaning it accounts for the memory hierarchy of modern GPUs. Standard attention computes the full NxN attention matrix, which becomes a memory bottleneck for long sequences. FlashAttention tiles the computation and uses recomputation instead of storing intermediate results, reducing memory reads and writes to and from GPU HBM by 5-20x. FlashAttention-2 further optimizes work partitioning across GPU warps. The result is 2-4x faster attention computation with significantly less memory usage, enabling longer context lengths on the same hardware. For offline deployment, FlashAttention makes the difference between fitting a model with 32K context versus only 4K context on a given GPU. It is now integrated into major inference engines including llama.cpp, vLLM, and Hugging Face Transformers.