Tokenization on English AI Terms Dictionary

WordPiece

Sat, 18 Jul 2026 10:20:04 +0000

Definition

WordPiece is a tokenization method widely used in natural language processing models like BERT and ALBERT. It breaks down words into smaller subword units to manage morphological richness and reduce vocabulary size. The algorithm starts with a base vocabulary and iteratively adds the most frequent character pairs until a target size is reached. This allows the model to represent rare or unseen words by combining known subwords, improving generalization and handling of linguistic variations effectively.

SentencePiece

Sat, 18 Jul 2026 10:15:05 +0000

Definition

SentencePiece is a popular open-source library for text normalization and tokenization, widely used in modern NLP pipelines. It performs unsupervised learning of a joint word-piece and subword vocabulary, allowing it to handle out-of-vocabulary words and multiple languages effectively. By breaking text into subword units, it reduces vocabulary size while maintaining coverage. It supports various languages and scripts, making it a standard choice for pre-processing inputs for models like T5, BART, and others.

BPE

Sat, 18 Jul 2026 09:40:12 +0000

Definition

Byte Pair Encoding (BPE) is a data compression technique adapted for natural language processing to handle out-of-vocabulary words. It starts with a vocabulary of individual characters and iteratively merges the most frequent adjacent pairs of symbols. This process creates a hierarchy of subword units, allowing models to balance between character-level flexibility and word-level efficiency. It is widely used in transformer-based models like GPT-2 and BERT to manage vocabulary size while preserving semantic meaning across diverse languages.