How do you benchmark and evaluate offline AI model quality?

Comprehensive offline model evaluation ensures deployment quality meets application requirements. Evaluation dimensions: accuracy (task-specific metrics), latency (time to first token, tokens per second), resource consumption (RAM usage, CPU/GPU utilization), and robustness (performance under varied inputs). LLM evaluation: use established benchmarks (MMLU for knowledge, HumanEval for code, MT-Bench for conversation quality, HellaSwag for reasoning), run evaluations locally using lm-evaluation-harness framework, and compare quantized model scores against the full-precision baseline. Task-specific evaluation: for classification tasks measure F1/accuracy on a held-out test set, for generation tasks use BLEU/ROUGE/BERTScore against reference outputs, for RAG measure retrieval precision/recall and answer faithfulness. Human evaluation: create a test set of 100-200 representative queries, have domain experts rate outputs on a 1-5 scale for accuracy, relevance, and completeness, calculate inter-annotator agreement (Cohen's kappa), and compare multiple models on the same test set using blind evaluation. Quantization quality check: compare outputs of quantized vs full-precision model on 50+ diverse prompts, measure perplexity difference (should be under 0.5 for good quantization), and test edge cases where quantization might cause issues (long context, mathematical reasoning, code generation). Performance benchmarking: use llama-bench for standardized throughput measurement, test with realistic prompt lengths (not just short prompts), measure memory usage under load, and test concurrent request handling. Regression testing: maintain a golden test set that runs automatically after model updates, compare metrics against previous deployment, and flag any degradation exceeding acceptance thresholds.