How do you optimize offline AI inference performance on edge devices?

Edge AI optimization combines model compression, hardware utilization, and inference pipeline tuning. Model optimization: quantization reduces precision (FP32 to INT8 reduces size 4x with minimal quality loss), pruning removes low-magnitude weights (structured pruning maintains hardware efficiency), knowledge distillation trains a smaller student model to mimic a larger teacher, and architecture search finds efficient model designs for target hardware. Hardware-specific optimization: use ONNX Runtime for cross-platform optimization with hardware-specific execution providers, TensorRT for NVIDIA GPUs (2-5x speedup over PyTorch), OpenVINO for Intel CPUs and VPUs, and Core ML for Apple Silicon (leveraging Neural Engine). Runtime optimizations: operator fusion (combine adjacent operations to reduce memory transfers), memory planning (reuse tensor buffers to minimize allocation), graph optimization (constant folding, dead code elimination), and dynamic batching (group concurrent requests for throughput). Edge-specific techniques: model caching in flash storage for instant loading, thermal management (reduce precision or batch size under thermal throttling), power-aware inference (balance quality vs battery consumption), and progressive generation (stream tokens to user while generating). Benchmarking: measure tokens per second, time to first token, peak memory usage, power consumption, and model load time. Target metrics vary by device: smartphones should achieve 10+ tokens/second for interactive use, embedded devices prioritize latency for real-time applications. Continuous optimization: profile with platform tools (Android Profiler, Instruments, NVIDIA Nsight), identify bottlenecks, and iterate on model and runtime configuration.