What are the differences between AWQ, GPTQ, and GGUF quantization methods?
AWQ or Activation-Aware Weight Quantization preserves the most salient weights based on activation patterns, achieving excellent quality at 4-bit with fast GPU inference. GPTQ uses a one-shot weight quantization method based on approximate second-order information, producing highly accurate GPU-optimized quantized models that integrate well with the Hugging Face ecosystem. GGUF is a format rather than a method, packaging models quantized via llama.cpp methods in a self-contained file. GGUF supports flexible bit depths from 2 to 8 bit, runs on both CPU and GPU, and is the standard for llama.cpp-based tools. AWQ and GPTQ require GPU for inference, while GGUF excels at CPU inference. For pure GPU deployments, AWQ often provides the best speed-quality tradeoff. For CPU or mixed inference, GGUF is the standard choice.