Compression on English AI Terms Dictionary

Pruning

Sat, 18 Jul 2026 10:12:36 +0000

Definition

Pruning involves identifying and eliminating neurons, connections, or filters in a neural network that contribute minimally to the output accuracy. By removing these redundant elements, the model becomes smaller and faster to execute without significantly compromising performance. This technique is crucial for deploying deep learning models on resource-constrained devices like mobile phones or embedded systems.

Summary

A model compression technique that removes redundant or less significant parameters to reduce size and improve inference speed.

Knowledge Distillation

Sat, 18 Jul 2026 10:03:41 +0000

Definition

Knowledge distillation is a machine learning method used to compress a large, complex neural network (the teacher) into a smaller, more efficient network (the student). The student model is trained to replicate the output probabilities of the teacher model rather than just the ground truth labels. This process allows the student to capture nuanced patterns and relationships learned by the teacher, resulting in a model that maintains high accuracy while requiring fewer computational resources and memory for deployment.

Distillation

Sat, 18 Jul 2026 09:31:32 +0000

Definition

This process involves transferring knowledge from a complex, high-performance ’teacher’ neural network to a simpler, more efficient ‘student’ network. The student learns not just from hard labels but also from the soft probability distributions output by the teacher, which contain richer information about class relationships. This allows the student to achieve comparable accuracy with significantly fewer parameters, enabling faster inference and lower computational costs, making it ideal for deployment on resource-constrained devices like mobile phones or edge hardware.