Sigmoid

Sat, 18 Jul 2026 10:15:20 +0000

Definition

The sigmoid function, defined as σ(z) = 1 / (1 + e^-z), is widely used in machine learning to model probabilities. It squashes input values into the range (0, 1), making it suitable for binary classification output layers. While historically popular in logistic regression and early neural networks, it suffers from the vanishing gradient problem during backpropagation, which can slow down training in deep networks compared to alternatives like ReLU or Leaky ReLU.

ReLU

Sat, 18 Jul 2026 09:42:48 +0000

Definition

ReLU is widely used in deep learning neural networks due to its computational efficiency and ability to mitigate the vanishing gradient problem. Mathematically defined as f(x) = max(0, x), it introduces non-linearity into the model without saturating neurons for positive inputs. Despite potential issues like dying ReLUs, it remains a standard choice for hidden layers in convolutional and fully connected networks.

Summary

Rectified Linear Unit is an activation function that outputs the input directly if positive, otherwise zero.

Activation Functions on English AI Terms Dictionary

Sigmoid

Definition

ReLU

Definition

Summary