How do you implement efficient LLM serving and inference optimization?

LLM serving optimization reduces cost and latency for production applications. Techniques: (1) KV-cache optimization: cache key-value attention states from previous tokens to avoid recomputation. PagedAttention (vLLM) manages KV-cache like virtual memory, eliminating fragmentation and enabling efficient memory use. (2) Continuous batching: dynamically batch incoming requests rather than waiting for fixed batch sizes, maximizing GPU utilization. vLLM and TGI implement this natively. (3) Speculative decoding: use a small draft model to generate candidate tokens quickly, then verify multiple tokens at once with the large model, achieving 2-3x speedup for generation. (4) Quantization: reduce model precision (FP16 to INT8 or INT4) for faster inference and lower memory. GPTQ, AWQ, and GGUF formats support various quantization strategies. (5) Flash Attention: memory-efficient attention computation using tiling and recomputation, enabling longer context windows and reduced memory usage. Serving frameworks: vLLM (highest throughput for batched serving), TGI (Hugging Face, good for production with built-in monitoring), TensorRT-LLM (NVIDIA, maximum single-GPU performance), and llama.cpp (CPU/hybrid serving, GGUF models). Infrastructure: implement load balancing across multiple GPU instances, auto-scale based on queue depth and latency, use request routing based on model and priority, and implement request queuing with priority tiers. Cost optimization: implement model routing (use smaller models for simple tasks), cache common responses with semantic similarity matching, implement streaming responses (start returning tokens while still generating), and use spot instances for batch processing workloads. Monitoring: track tokens per second throughput, P50/P95/P99 latency, GPU utilization, queue depth, and cost per token.