Stability on English AI Terms Dictionary

Clip

Sat, 18 Jul 2026 09:49:31 +0000

Definition

In deep learning engineering, clipping is commonly applied to gradients to mitigate the exploding gradient problem, ensuring stable backpropagation. It can also refer to limiting output logits before applying softmax to prevent extreme probability distributions. By capping values within a predefined range, clipping improves model robustness and convergence speed, serving as a critical regularization step in training complex architectures like RNNs and Transformers.

Summary

Clipping is a technique used to limit the magnitude of values, such as gradients or output probabilities, to prevent numerical instability during training.

Divergence

Sat, 18 Jul 2026 09:31:32 +0000

Definition

In the context of optimization, divergence occurs when the parameters of a model update in a way that causes the loss to increase rather than decrease, often leading to NaN values or infinite gradients. This is frequently caused by excessively high learning rates, poor weight initialization, or numerical instability in the computation graph. Detecting divergence early is crucial for debugging training pipelines, as it prevents wasted computational resources and ensures the model can converge to a meaningful solution. Techniques like gradient clipping or reducing the learning rate are common remedies.