What is model distillation and how does it create smaller models suitable for offline use?
Model distillation is a technique where a large powerful teacher model transfers its knowledge to a smaller student model. The student model learns to mimic the teacher output distributions rather than training from raw data alone, capturing the teacher learned representations in a more compact form. This produces models that are dramatically smaller but retain much of the teacher capability. For example, DistilBERT retains 97% of BERT performance with 40% fewer parameters. The process involves training the student on soft labels from the teacher, often combined with hard labels from the training data. Distilled models are ideal for offline deployment because they require less memory, run faster on limited hardware, and can achieve near state-of-the-art performance for specific tasks on edge devices.