How do you implement LLM evaluation and benchmarking for model selection?

LLM evaluation determines which model best fits your application through systematic testing. Evaluation types: (1) Benchmark evaluation: test on standard datasets (MMLU for knowledge breadth, HumanEval for coding, MT-Bench for conversation, GSM8K for math). Useful for comparing model capability broadly but may not reflect performance on your specific task. (2) Task-specific evaluation: create a dataset of representative inputs with expected outputs from your application domain, run each candidate model, and score outputs. This is the most important evaluation. (3) Human evaluation: domain experts rate model outputs on criteria like accuracy, relevance, completeness, and tone. Gold standard but expensive and slow. (4) LLM-as-judge: use a powerful model (GPT-4, Claude) to evaluate outputs from candidate models against defined criteria. Correlates well with human judgment when given clear rubrics. Implementation: create an evaluation dataset of 100-500 representative examples covering common cases, edge cases, and failure modes. Define scoring rubrics (1-5 scale per dimension: accuracy, completeness, format compliance, safety). Run all candidate models, score outputs, and analyze results by category. Automated evaluation pipeline: integrate evaluation into CI/CD so every prompt or model change is automatically tested against the evaluation set. Track metrics over time and alert on regressions. Cost-quality trade-off: evaluate models across price points (GPT-4o vs GPT-4o-mini vs Claude Haiku vs open-source), find the cheapest model that meets quality thresholds for each task type, and implement model routing (complex queries to powerful models, simple queries to fast/cheap models). Latency evaluation: measure time-to-first-token and tokens-per-second for each model, evaluate on realistic prompt lengths, and test under concurrent load.