How do you implement AI model compression and optimization for deployment?
Model compression reduces model size and inference cost while maintaining acceptable quality. Techniques: (1) Quantization: reduce numerical precision of model weights. Post-training quantization (PTQ) converts a trained FP32 model to INT8 or INT4 without retraining (fastest, some quality loss). Quantization-aware training (QAT) trains with simulated quantization for better quality. GPTQ and AWQ are specialized methods for LLM quantization. Typical results: 4-bit quantization reduces model size by 8x with 1-3% quality degradation. (2) Pruning: remove unnecessary weights or neurons. Unstructured pruning zeros out individual weights (needs sparse computation support for speedup). Structured pruning removes entire neurons, attention heads, or layers (direct speedup on standard hardware). Typically achieves 50-90% sparsity with under 5% quality loss. (3) Knowledge distillation: train a smaller student model to mimic a larger teacher model. The student learns from the teacher's output distribution (soft labels contain more information than hard labels). Can achieve 80-95% of teacher quality at 10-50x smaller size. (4) Architecture optimization: use efficient architectures (MobileNet, EfficientNet for vision; DistilBERT, TinyBERT for NLP), implement attention optimization (grouped query attention, multi-query attention), and use mixture-of-experts (activate only a subset of parameters per input). Implementation: use Hugging Face Optimum for model optimization, ONNX Runtime for cross-platform optimized inference, TensorRT for NVIDIA GPU optimization, and Core ML for Apple device optimization. Evaluation: measure quality degradation on your evaluation set, benchmark inference speed on target hardware, measure memory usage, and calculate cost savings. Production: maintain both full and compressed model versions, A/B test compressed models against full models, and implement quality monitoring that triggers fallback to full model if compressed model underperforms.