What are the GPU acceleration options for offline AI inference?

GPU acceleration dramatically improves offline AI inference speed. NVIDIA GPUs: most mature ecosystem via CUDA, cuDNN, and TensorRT. Consumer cards: RTX 3060 (12GB VRAM, runs 7B models at 40+ tok/s), RTX 4090 (24GB, runs 13B at 60+ tok/s, 70B Q4 with partial offload). Data center: A100 (80GB, handles 70B+ full precision), H100 (80GB, 3x A100 throughput). AMD GPUs: ROCm support in llama.cpp and PyTorch, RX 7900 XTX (24GB) provides competitive performance for inference, but ecosystem maturity trails NVIDIA. Intel Arc: OneAPI and SYCL support in llama.cpp, A770 (16GB) offers budget-friendly inference acceleration. Apple Metal: unified memory architecture in M-series chips means all system memory is accessible to GPU, providing effective VRAM of 16-192GB depending on chip configuration. Multi-GPU: distribute model layers across GPUs using tensor parallelism (split each layer across GPUs for reduced latency) or pipeline parallelism (assign different layers to different GPUs for increased throughput). GPU offloading in llama.cpp: specify number of layers to offload to GPU (-ngl parameter), remaining layers run on CPU, allowing models larger than VRAM to benefit from partial GPU acceleration. Optimization: use Flash Attention 2 for reduced memory usage and faster attention computation, implement KV-cache quantization to reduce VRAM consumption, and use continuous batching for serving multiple concurrent requests. Selection criteria: VRAM capacity determines maximum model size, memory bandwidth affects generation speed, and compute capability determines prompt processing speed.