What is GGML and how does it enable offline AI inference on consumer hardware?

GGML (Georgi Gerganov Machine Learning) is a tensor library written in C that enables efficient CPU-based inference for large language models. It uses quantization to reduce model size and memory requirements while maintaining acceptable quality. Key features: pure C implementation with no dependencies (portable across platforms), support for multiple quantization formats (Q2K through Q80), SIMD optimizations (AVX, AVX2, AVX-512, ARM NEON), and memory-mapped file loading for fast startup. The GGUF format (successor to GGML format) stores model weights, tokenizer, and metadata in a single file. Quantization levels: Q4KM offers the best balance of quality and size (roughly 4 bits per weight, 4GB for a 7B model), Q5KM provides higher quality at modest size increase, Q80 approaches full precision quality, and Q2K is smallest but with noticeable degradation. Hardware requirements for a 7B Q4KM model: 6GB RAM minimum, any modern x8664 or ARM64 CPU, with generation speeds of 10-30 tokens/second on recent hardware. The llama.cpp project is the primary inference engine using GGML, supporting GPU offloading for hybrid CPU/GPU execution, batch processing, grammar-constrained generation, and a built-in HTTP server compatible with the OpenAI API format.