Distillation Scaling Laws (original) (raw)
Abstract:We propose a distillation scaling law that estimates distilled model performance based on a compute budget and its allocation between the student and teacher. Our findings mitigate the risks associated with large-scale distillation by enabling compute-optimal allocation for both the teacher and student to maximize student performance. We provide compute-optimal distillation recipes for two key scenarios: when a teacher already exists, and when a teacher needs training. In settings involving many students or an existing teacher, distillation outperforms supervised learning up to a compute level that scales predictably with student size. Conversely, if only one student is to be distilled and a teacher also requires training, supervised learning is generally preferable. Additionally, our large-scale study of distillation increases our understanding of the process and helps inform experimental design.
Submission history
From: Dan Busbridge [view email]
[v1] Wed, 12 Feb 2025 17:52:47 UTC (15,174 KB)
[v2] Fri, 25 Jul 2025 16:55:43 UTC (15,226 KB)