Hyperparameters on English AI Terms Dictionary

Batch Size

Sat, 18 Jul 2026 09:47:51 +0000

Definition

Batch size is a critical hyperparameter that determines how many samples are processed before the model’s internal parameters are updated. A larger batch size provides a more accurate estimate of the gradient, leading to stable convergence but requiring more memory and potentially generalizing poorly. Conversely, smaller batch sizes introduce noise into the gradient estimation, which can help escape local minima but may result in noisier convergence paths and longer training times due to frequent updates.

Bayesian optimization

Sat, 18 Jul 2026 09:47:51 +0000

Definition

Bayesian optimization uses a probabilistic surrogate model, typically a Gaussian Process, to model the objective function. It employs an acquisition function to balance exploration and exploitation, selecting the next evaluation point that maximizes expected improvement. This method is highly efficient for tuning hyperparameters in machine learning models where each training run is computationally costly, requiring fewer evaluations than grid or random search to find near-optimal configurations.

Summary

A sequential design strategy for global optimization of black-box functions that are expensive to evaluate.

Learning Rate

Sat, 18 Jul 2026 09:41:26 +0000

Definition

The learning rate determines how much the model’s weights are updated relative to the calculated gradient during each training iteration. A rate that is too high may cause the model to overshoot optimal solutions, while a rate that is too low leads to slow convergence or getting stuck in local minima. Tuning this parameter is essential for efficient training, often involving schedulers that decay the rate over time to fine-tune the model near the end of the training process.

Rate

Sat, 18 Jul 2026 09:36:45 +0000

Definition

In AI, ‘rate’ most frequently refers to the learning rate, a hyperparameter that controls how much to change the model in response to the estimated error each time the model weights are updated. A rate that is too high may cause the model to converge too quickly to a suboptimal solution, while a rate that is too low may result in excessively long training times. It can also refer to API request rates or token generation throughput.