How do you optimize transformer models for offline inference on ARM processors?
ARM optimization for transformers involves: use ARM Compute Library (ACL) for NEON SIMD acceleration, leverage ARM Ethos NPU where available (up to 32 TOPS), apply INT8 quantization with ARM-specific calibration, and use dynamic shape inference to avoid padding waste. Specific techniques: convert models to TFLite with ARM delegate or ONNX with XNNPACK execution provider, implement operator fusion optimized for ARM's load-store architecture, use memory-mapped model loading to reduce RAM footprint, and apply channel-last (NHWC) memory layout which ARM processors prefer. For Apple Silicon (M1-M4), use CoreML with ANE (Apple Neural Engine) delegation for up to 15.8 TOPS. Benchmarks show optimized INT8 models on ARM Cortex-A78 achieve 3-5x speedup over naive FP32 execution, making 1-3B parameter models practical on mobile devices.