https://terms-en.ai-term-hub.com/zh/tags/bert/Recent content in BERT on 中文AI术语词典Hugozh-cnSat, 18 Jul 2026 11:44:45 +0000https://terms-en.ai-term-hub.com/zh/terms/wordpiece/Sat, 18 Jul 2026 11:38:14 +0000https://terms-en.ai-term-hub.com/zh/terms/wordpiece/<h2 id="definition">Definition</h2> <p>WordPiece 是一种广泛应用于 BERT 和 ALBERT 等自然语言处理模型的分词方法。它将单词分解为更小的子词单元，以应对形态学丰富性并减少词汇表大小，从而更好地处理未见过的单词。</p> <h3 id="summary">Summary</h3> <p>一种子词分词算法，通过递归合并最频繁出现的字符对来处理未登录词（OOV）。</p> <h2 id="key-concepts">Key Concepts</h2> <ul> <li>子词分词</li> <li>词汇扩展</li> <li>未登录词处理</li> <li>形态分析</li> </ul> <h2 id="use-cases">Use Cases</h2> <ul> <li>为 BERT 模型预处理文本</li> <li>处理低资源语言</li> <li>减小嵌入矩阵大小</li> </ul> <h2 id="code-example">Code Example</h2> <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">from</span> transformers <span style="color:#f92672">import</span> BertTokenizer </span></span><span style="display:flex;"><span>tokenizer <span style="color:#f92672">=</span> BertTokenizer<span style="color:#f92672">.</span>from_pretrained(<span style="color:#e6db74">&#39;bert-base-uncased&#39;</span>) </span></span><span style="display:flex;"><span>tokens <span style="color:#f92672">=</span> tokenizer<span style="color:#f92672">.</span>tokenize(<span style="color:#e6db74">&#39;unhappiness&#39;</span>) </span></span><span style="display:flex;"><span>print(tokens) </span></span></code></pre></div><h2 id="related-terms">Related Terms</h2> <ul> <li><a href="https://terms-en.ai-term-hub.com/en/terms/%E5%AD%97%E8%8A%82%E5%AF%B9%E7%BC%96%E7%A0%81-byte-pair-encoding/">字节对编码 (Byte-Pair Encoding)</a></li> <li><a href="https://terms-en.ai-term-hub.com/en/terms/sentencepiece/">SentencePiece</a></li> <li><a href="https://terms-en.ai-term-hub.com/en/terms/%E5%88%86%E8%AF%8D-tokenization/">分词 (Tokenization)</a></li> <li><a href="https://terms-en.ai-term-hub.com/en/terms/nlp-%E9%A2%84%E5%A4%84%E7%90%86-nlp-preprocessing/">NLP 预处理 (NLP preprocessing)</a></li> </ul>