Self Supervised on English AI Terms Dictionary

Representation collapse

Sat, 18 Jul 2026 10:14:07 +0000

Definition

Representation collapse occurs when a neural network, particularly in self-supervised contrastive learning frameworks, learns to map all input data points to the same fixed output vector. This trivial solution minimizes the loss function without learning meaningful features. To prevent this, techniques like normalization, momentum encoders, or specific loss formulations are employed to ensure the model preserves distinct information across different inputs.

Summary

A failure mode in self-supervised learning where the model outputs identical representations for all inputs, losing discriminative power.

Predictive learning

Sat, 18 Jul 2026 10:11:14 +0000

Definition

Predictive learning involves training neural networks to infer unobserved data points from observed inputs without explicit human labels. By solving tasks like next-token prediction in language or masked pixel reconstruction in images, the model learns rich internal representations of structure and semantics. This method leverages vast amounts of unlabeled data, enabling scalable pre-training that captures general patterns useful for downstream tasks through fine-tuning.

Summary

A self-supervised approach where models learn representations by predicting missing parts of input data.

Contrastive Learning

Sat, 18 Jul 2026 09:51:47 +0000

Definition

Contrastive learning is a representation learning method that does not require labeled data. It works by creating augmented views of the same input (positive pairs) and contrasting them with different inputs (negative pairs). The model is trained to minimize the distance between positive pairs in the embedding space while maximizing the distance between negative pairs. This approach has become foundational for achieving state-of-the-art results in computer vision and natural language processing tasks.