What is model sharding and how does it enable large offline AI models?

Model sharding splits a large model across multiple storage locations or compute units, enabling models that exceed single-device capacity. Types: tensor parallelism (splits individual layers across GPUs), pipeline parallelism (assigns different layers to different GPUs), and expert parallelism (for MoE models, routes to different GPU subsets). For offline deployment: disk-based sharding loads model layers on demand from NVMe storage (llama.cpp supports this with --mmap), multi-GPU sharding distributes a 70B model across 2-4 consumer GPUs, and CPU+GPU hybrid sharding offloads less critical layers to CPU RAM. Implementation: use GGUF format which supports memory-mapped access, configure layer assignment based on available VRAM, and implement prefetching to overlap compute with I/O. A 70B model quantized to Q4KM (40GB) can run on a system with 24GB VRAM + 32GB RAM by keeping 30 layers on GPU and 50 layers memory-mapped from disk, achieving 5-8 tokens/second.