How do you scale offline AI inference across multiple GPUs?

Scaling offline inference across multiple GPUs uses two primary strategies. Tensor parallelism splits individual model layers across GPUs, enabling models too large for a single GPU to fit in combined memory. Pipeline parallelism assigns different layers to different GPUs, processing requests through the layer stages sequentially. For serving multiple users, data parallelism runs model replicas on separate GPUs handling different requests simultaneously. Tools like vLLM, TGI, and llama.cpp support multi-GPU inference natively. NCCL provides high-bandwidth inter-GPU communication for tensor parallelism. NVLink-connected GPUs provide better performance than PCIe-connected ones for tensor parallel workloads. For multi-node scaling, InfiniBand provides low-latency networking between servers. The optimal parallelism strategy depends on model size relative to per-GPU memory, the number of concurrent users, and the latency versus throughput requirements.