Transformers on English AI Terms Dictionary

XLM-RoBERTa

Sat, 18 Jul 2026 10:20:18 +0000

Definition

XLM-RoBERTa (Cross-lingual Language Model RoBERTa) is a large-scale multilingual model developed by Meta AI. It extends the RoBERTa architecture by pre-training on a diverse dataset covering over 100 languages. This allows the model to learn shared representations across languages, enabling strong performance in cross-lingual transfer tasks. It is widely used for machine translation, multilingual classification, and zero-shot cross-lingual information retrieval without needing language-specific fine-tuning.

Summary

A multilingual transformer model based on RoBERTa, pre-trained on massive amounts of text from 100+ languages.

Prefix Tuning

Sat, 18 Jul 2026 10:11:14 +0000

Definition

Prefix Tuning is a parameter-efficient adaptation technique for pre-trained transformers. Instead of updating all model weights, it prepends a sequence of trainable continuous vectors (the prefix) to the input embeddings of each layer. These prefixes act as soft prompts that guide the model’s behavior for specific downstream tasks while keeping the base model frozen. This approach significantly reduces memory and computational costs compared to full fine-tuning, making it suitable for resource-constrained environments.

Long Context

Sat, 18 Jul 2026 10:05:43 +0000

Definition

Long context refers to the capacity of transformer-based models to handle extensive input lengths, often exceeding standard limits like 2k or 4k tokens. This capability allows models to analyze entire documents, codebases, or lengthy conversations in a single pass. Achieving this requires architectural innovations such as efficient attention mechanisms (e.g., FlashAttention) or positional encoding adjustments to maintain coherence and memory over vast distances within the sequence.

Summary

The ability of a language model to process and retain information from input sequences containing thousands or millions of tokens.

Fill Mask

Sat, 18 Jul 2026 09:58:20 +0000

Definition

Fill Mask is a fundamental pre-training objective used in transformer-based models like BERT. The process involves masking random tokens in a text sequence and training the model to predict the original values of those masked words. This self-supervised learning approach helps the model understand bidirectional context and semantic relationships between words, forming the basis for many downstream NLP applications such as question answering and text completion.

Summary

A natural language processing task where a model predicts missing tokens within a sentence based on surrounding context.

ExBERT

Sat, 18 Jul 2026 09:57:25 +0000

Definition

ExBERT provides interpretability for the BERT transformer model by analyzing the importance of individual attention heads across different layers. It uses techniques like gradient-based attribution or ablation studies to determine which parts of the model are responsible for specific token predictions or semantic features. This helps researchers understand how BERT processes linguistic information and debugs model behavior in natural language processing tasks.

Summary

A method for explaining BERT’s predictions by identifying which attention heads and layers contribute most to specific outputs.

Positional Encoding

Sat, 18 Jul 2026 09:42:48 +0000

Definition

Since transformers process all tokens in parallel rather than sequentially like RNNs, they lack inherent knowledge of token order. Positional encoding adds specific vectors to input embeddings to preserve sequence information. Common methods include sinusoidal functions learned during training or learned embeddings. This allows the self-attention mechanism to weigh the importance of different tokens based on their position, enabling the model to understand syntax and context effectively.

Summary

A technique that injects information about the relative or absolute position of tokens in a sequence into transformer models.

Encoder

Sat, 18 Jul 2026 09:40:59 +0000

Definition

Encoders process raw input sequences or data structures and convert them into latent space representations, often called embeddings or codes. They are central to architectures like Transformers and Autoencoders. The encoder’s goal is to capture essential features and contextual information while discarding noise, creating a compact summary that downstream components, such as decoders or classifiers, can utilize effectively for prediction or generation tasks.

Summary

An encoder is a component of a neural network that transforms input data into a compressed, meaningful representation.

Adapter

Sat, 18 Jul 2026 09:39:58 +0000

Definition

Adapters are a parameter-efficient fine-tuning technique used primarily in large language models and transformers. Instead of updating all model weights, which is computationally expensive, adapters introduce small, task-specific neural network layers between existing layers. This allows the model to retain its general knowledge while adapting to new tasks with minimal additional parameters. It significantly reduces memory usage and storage requirements, making it feasible to deploy multiple specialized models on top of a single base model without catastrophic forgetting.

Attention

Sat, 18 Jul 2026 09:39:58 +0000

Definition

Attention mechanisms enable models to focus on relevant information when processing inputs, particularly in sequential data like text. By calculating attention scores, the model determines which elements of the input have the most influence on the current prediction. This approach overcomes the limitations of fixed-size context windows in recurrent networks. Self-attention, a core component of Transformers, allows every token to attend to every other token, capturing long-range dependencies and contextual relationships efficiently, thereby significantly improving performance in NLP and computer vision tasks.

Self-Attention

Sat, 18 Jul 2026 09:36:45 +0000

Definition

Self-attention enables models to capture dependencies between all positions in a sequence simultaneously, regardless of distance. By computing attention scores between every pair of tokens, it allows the network to dynamically focus on relevant context, forming the foundational layer of Transformer architectures used in modern natural language processing and computer vision tasks.

Summary

A mechanism allowing a neural network to weigh the importance of different parts of the input sequence relative to each other.