Tokenization on 中文AI术语词典https://terms-en.ai-term-hub.com/zh/tags/tokenization/Recent content in Tokenization on 中文AI术语词典Hugozh-cnSat, 18 Jul 2026 11:44:45 +0000WordPiecehttps://terms-en.ai-term-hub.com/zh/terms/wordpiece/Sat, 18 Jul 2026 11:38:14 +0000https://terms-en.ai-term-hub.com/zh/terms/wordpiece/<h2 id="definition">Definition</h2> <p>WordPiece 是一种广泛应用于 BERT 和 ALBERT 等自然语言处理模型的分词方法。它将单词分解为更小的子词单元，以应对形态学丰富性并减少词汇表大小，从而更好地处理未见过的单词。</p> <h3 id="summary">Summary</h3> <p>一种子词分词算法，通过递归合并最频繁出现的字符对来处理未登录词（OOV）。</p> <h2 id="key-concepts">Key Concepts</h2> <ul> <li>子词分词</li> <li>词汇扩展</li> <li>未登录词处理</li> <li>形态分析</li> </ul> <h2 id="use-cases">Use Cases</h2> <ul> <li>为 BERT 模型预处理文本</li> <li>处理低资源语言</li> <li>减小嵌入矩阵大小</li> </ul> <h2 id="code-example">Code Example</h2> <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">from</span> transformers <span style="color:#f92672">import</span> BertTokenizer </span></span><span style="display:flex;"><span>tokenizer <span style="color:#f92672">=</span> BertTokenizer<span style="color:#f92672">.</span>from_pretrained(<span style="color:#e6db74">&#39;bert-base-uncased&#39;</span>) </span></span><span style="display:flex;"><span>tokens <span style="color:#f92672">=</span> tokenizer<span style="color:#f92672">.</span>tokenize(<span style="color:#e6db74">&#39;unhappiness&#39;</span>) </span></span><span style="display:flex;"><span>print(tokens) </span></span></code></pre></div><h2 id="related-terms">Related Terms</h2> <ul> <li><a href="https://terms-en.ai-term-hub.com/en/terms/%E5%AD%97%E8%8A%82%E5%AF%B9%E7%BC%96%E7%A0%81-byte-pair-encoding/">字节对编码 (Byte-Pair Encoding)</a></li> <li><a href="https://terms-en.ai-term-hub.com/en/terms/sentencepiece/">SentencePiece</a></li> <li><a href="https://terms-en.ai-term-hub.com/en/terms/%E5%88%86%E8%AF%8D-tokenization/">分词 (Tokenization)</a></li> <li><a href="https://terms-en.ai-term-hub.com/en/terms/nlp-%E9%A2%84%E5%A4%84%E7%90%86-nlp-preprocessing/">NLP 预处理 (NLP preprocessing)</a></li> </ul>SentencePiecehttps://terms-en.ai-term-hub.com/zh/terms/sentencepiece/Sat, 18 Jul 2026 11:33:07 +0000https://terms-en.ai-term-hub.com/zh/terms/sentencepiece/<h2 id="definition">Definition</h2> <p>SentencePiece 是一个流行的开源文本归一化和分词库，广泛用于现代自然语言处理管道中。它执行联合词块和子词词汇表的无监督学习。</p> <h3 id="summary">Summary</h3> <p>一种无监督文本分词器和去分词器库，将原始文本视为子词序列用于自然语言处理预处理。</p> <h2 id="key-concepts">Key Concepts</h2> <ul> <li>子词分词</li> <li>词汇表学习</li> <li>去分词</li> <li>语言无关</li> </ul> <h2 id="use-cases">Use Cases</h2> <ul> <li>为变换器模型预处理数据</li> <li>处理多语言文本语料库</li> <li>减少语言模型的词汇量大小</li> </ul> <h2 id="related-terms">Related Terms</h2> <ul> <li><a href="https://terms-en.ai-term-hub.com/en/terms/tokenizer-%E5%88%86%E8%AF%8D%E5%99%A8/">Tokenizer (分词器)</a></li> <li><a href="https://terms-en.ai-term-hub.com/en/terms/bpe-%E5%AD%97%E8%8A%82%E5%AF%B9%E7%BC%96%E7%A0%81/">BPE (字节对编码)</a></li> <li><a href="https://terms-en.ai-term-hub.com/en/terms/byte-pair-encoding-%E5%AD%97%E8%8A%82%E5%AF%B9%E7%BC%96%E7%A0%81/">Byte-Pair Encoding (字节对编码)</a></li> <li><a href="https://terms-en.ai-term-hub.com/en/terms/nlp-preprocessing-%E8%87%AA%E7%84%B6%E8%AF%AD%E8%A8%80%E5%A4%84%E7%90%86%E9%A2%84%E5%A4%84%E7%90%86/">NLP Preprocessing (自然语言处理预处理)</a></li> </ul>字节对编码 (BPE)https://terms-en.ai-term-hub.com/zh/terms/bpe/Sat, 18 Jul 2026 10:59:27 +0000https://terms-en.ai-term-hub.com/zh/terms/bpe/<h2 id="definition">Definition</h2> <p>字节对编码（BPE）是一种数据压缩技术，经过调整后应用于自然语言处理中，以处理未登录词（Out-of-Vocabulary）。它从单个字符的词汇表开始，并迭代地合并最频繁出现的字符对，直到达到预定的词汇表大小或收敛。这种方法允许模型将罕见词分解为更常见的子词单元，从而提高对未知词汇的处理能力。</p> <h3 id="summary">Summary</h3> <p>字节对编码是一种用于子词分词的算法，它通过迭代合并出现频率最高的字符对来构建词汇表。</p> <h2 id="key-concepts">Key Concepts</h2> <ul> <li>子词分词</li> <li>词汇表合并</li> <li>频率分析</li> <li>未登录词处理</li> </ul> <h2 id="use-cases">Use Cases</h2> <ul> <li>为大语言模型预处理文本</li> <li>处理形态丰富的语言</li> <li>减少神经网络中的词汇表规模</li> </ul> <h2 id="code-example">Code Example</h2> <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">import</span> tiktoken </span></span><span style="display:flex;"><span>enc <span style="color:#f92672">=</span> tiktoken<span style="color:#f92672">.</span>get_encoding(<span style="color:#e6db74">&#34;cl100k_base&#34;</span>) </span></span><span style="display:flex;"><span>tokens <span style="color:#f92672">=</span> enc<span style="color:#f92672">.</span>encode(<span style="color:#e6db74">&#34;unhappiness&#34;</span>) </span></span><span style="display:flex;"><span>print(tokens) </span></span></code></pre></div><h2 id="related-terms">Related Terms</h2> <ul> <li><a href="https://terms-en.ai-term-hub.com/en/terms/wordpiece-wordpiece%E5%88%86%E8%AF%8D%E7%AE%97%E6%B3%95/">WordPiece (WordPiece分词算法)</a></li> <li><a href="https://terms-en.ai-term-hub.com/en/terms/sentencepiece-sentencepiece%E5%88%86%E8%AF%8D%E5%B7%A5%E5%85%B7%E5%BA%93/">SentencePiece (SentencePiece分词工具库)</a></li> <li><a href="https://terms-en.ai-term-hub.com/en/terms/tokenization-%E5%88%86%E8%AF%8D/">Tokenization (分词)</a></li> <li><a href="https://terms-en.ai-term-hub.com/en/terms/subword-units-%E5%AD%90%E8%AF%8D%E5%8D%95%E5%85%83/">Subword Units (子词单元)</a></li> </ul>