Representation collapse

Sat, 18 Jul 2026 10:14:07 +0000

Definition

Representation collapse occurs when a neural network, particularly in self-supervised contrastive learning frameworks, learns to map all input data points to the same fixed output vector. This trivial solution minimizes the loss function without learning meaningful features. To prevent this, techniques like normalization, momentum encoders, or specific loss formulations are employed to ensure the model preserves distinct information across different inputs.

Summary

A failure mode in self-supervised learning where the model outputs identical representations for all inputs, losing discriminative power.

Overfitting

Sat, 18 Jul 2026 09:41:54 +0000

Definition

Overfitting occurs when a model learns the training data too well, including its random noise and outliers, resulting in excellent performance on training data but poor performance on new, unseen test data. This happens because the model becomes overly complex relative to the amount of training data available. Techniques like regularization, dropout, early stopping, and cross-validation are commonly employed to mitigate overfitting and improve the model’s ability to generalize.

Training Dynamics on English AI Terms Dictionary

Representation collapse

Definition

Summary

Overfitting

Definition