What are the different types of attention mechanisms and how do they work in transformers?
Attention mechanisms allow models to focus on relevant parts of the input when producing each output element. Self-attention (core of transformers): for each token, compute similarity scores with all other tokens, normalize via softmax, and produce a weighted sum of value vectors. Mathematically: Attention(Q,K,V) = softmax(QK^T / sqrt(dk)) V, where Q (queries), K (keys), and V (values) are linear projections of the input. Multi-head attention: run multiple attention heads in parallel, each learning different relationship patterns (syntactic, semantic, positional), then concatenate and project results. This enables the model to attend to different aspects simultaneously. Types of attention: (1) Full self-attention: every token attends to every other token, O(n^2) complexity, used in standard transformers. (2) Causal/masked attention: tokens can only attend to previous positions, used in autoregressive generation (GPT-style). (3) Cross-attention: queries from one sequence, keys and values from another, used in encoder-decoder models (translation, summarization). (4) Sparse attention: only attend to a subset of positions (local window + global tokens), O(n sqrt(n)) complexity, used in Longformer, BigBird. (5) Flash Attention: exact attention computed in a memory-efficient way using tiling and recomputation, dramatically reducing GPU memory usage and improving speed without approximation. (6) Grouped Query Attention (GQA): share key-value heads across multiple query heads, reducing KV-cache size while maintaining quality, used in Llama 2/3 and Mistral. (7) Multi-Query Attention (MQA): all query heads share a single key-value head, maximum KV-cache efficiency, used in Falcon and PaLM. Impact: attention mechanisms are the foundation of modern AI, enabling models to process sequences of any length while capturing long-range dependencies.