Multi-Head Attention

Sat, 18 Jul 2026 09:34:16 +0000

Definition

Multi-Head Attention extends the standard attention mechanism by running it multiple times in parallel with different learned linear projections. This enables the model to jointly attend to information from different positional subspaces at different positions. By capturing diverse relationships within the input sequence, such as syntactic and semantic dependencies, it significantly enhances the model’s ability to understand context. It is a foundational component of modern Large Language Models (LLMs) and vision transformers, providing robust feature extraction capabilities.

Transformer on English AI Terms Dictionary

Multi-Head Attention

Definition