Distillation

Sat, 18 Jul 2026 09:31:32 +0000

Definition

This process involves transferring knowledge from a complex, high-performance ’teacher’ neural network to a simpler, more efficient ‘student’ network. The student learns not just from hard labels but also from the soft probability distributions output by the teacher, which contain richer information about class relationships. This allows the student to achieve comparable accuracy with significantly fewer parameters, enabling faster inference and lower computational costs, making it ideal for deployment on resource-constrained devices like mobile phones or edge hardware.

Model Efficiency on English AI Terms Dictionary

Distillation

Definition