代碼地址：https://github.com/taishan1994/sentencepiece_chinese_bpe

Part1前言

目前，大語(yǔ)言模型呈爆發(fā)式的增長(zhǎng)，其中，基于llama家族的模型占據(jù)了半壁江山。而原始的llama模型對(duì)中文的支持不太友好，接下來(lái)本文將講解如何去擴(kuò)充vocab里面的詞以對(duì)中文進(jìn)行token化。

Part2數(shù)據(jù)預(yù)處理

對(duì)斗破蒼穹語(yǔ)料進(jìn)行預(yù)處理，每一行為一句或多句話。

with?open("data/《斗破蒼穹》.txt",?"r",?encoding="utf-8")?as?fp:
????data?=?fp.read().strip().split("\n")
sentences?=?[]

for?d?in?data:
????d?=?d.strip()
????if?"==="?in?d?or?len(d)?==?0?or?d?==?"《斗破蒼穹》來(lái)自:":
????????continue
????sentences.append(d)

with?open("data/corpus.txt",?"w",?encoding="utf-8")?as?fp:
????fp.write("\n".join(sentences))

最終得到corpus.txt。

Part3sentencepiece

首先，我們需要去構(gòu)建中文的詞庫(kù)。一般的，目前比較主流的是使用sentencepiece訓(xùn)練中文詞庫(kù)。安裝指令也很簡(jiǎn)單：pip install sentencepiece。然后，我們準(zhǔn)備好語(yǔ)料，這里我們使用的語(yǔ)料是斗破蒼穹小說(shuō)。

直接看代碼：

import?sentencepiece?as?spm
spm.SentencePieceTrainer.train(
????input='data/corpus.txt',
????model_prefix='tokenizer',
????vocab_size=50000,
????user_defined_symbols=['foo',?'bar'],
????character_coverage=1.0,
????model_type="bpe",
)

這里講下每個(gè)參數(shù)的作用：

• input：指定輸入文本文件的路徑或者是一個(gè)目錄，可以指定多個(gè)輸入文件或目錄。其中每一行可以是一句話或者多句話。
• tokenizer：保存的模型的名稱前綴。
• vocab_size：設(shè)置的詞表大小。
• user_defined_symbols：用于指定用戶自定義的符號(hào)。這些符號(hào)將會(huì)被視為單獨(dú)的 Token，不會(huì)被拆分成子詞。這個(gè)參數(shù)的作用是將一些用戶定義的特殊符號(hào)作為一個(gè)整體加入到生成的詞表中，以便于后續(xù)的模型使用。這里我們簡(jiǎn)單進(jìn)行了測(cè)試。
• model_type: 指定模型的類(lèi)型，有三種可選參數(shù)：unigram, bpe, char. word。
• character_coverage指定覆蓋字符的數(shù)量，可以理解為限制字符集的大小。默認(rèn)值為 1.0，即覆蓋全部字符。
• unk_id: 指定未登錄詞的 ID 號(hào)，即在詞表中為未登錄詞分配一個(gè)整數(shù) ID。默認(rèn)值為 0。
• bos_id: 指定句子開(kāi)頭符號(hào)的 ID 號(hào)，即在詞表中為句子開(kāi)頭符號(hào)分配一個(gè)整數(shù) ID。默認(rèn)值為 1。
• eos_id: 指定句子結(jié)束符號(hào)的 ID 號(hào)，即在詞表中為句子結(jié)束符號(hào)分配一個(gè)整數(shù) ID。默認(rèn)值為 2。
• pad_id: 指定填充符號(hào)的 ID 號(hào)，即在詞表中為填充符號(hào)分配一個(gè)整數(shù) ID。默認(rèn)值為 -1，即不使用填充符號(hào)。

運(yùn)行后會(huì)得到tokenizer.model和tokenizer.vocab兩個(gè)文件。

我們來(lái)看看tokenizer.vocab里面是什么：

<unk>?0
<s>?0
</s>?0
foo?0
bar?0
蕭炎?-0
..?-1
▁“?-2
也是?-3
便是?-4
了一?-5
。”?-6

除了一些特殊符號(hào)外，還有我們自定義的foo和bar，其余的一些詞是BPE訓(xùn)練得到，具體什么是BPE算法這里不作展開(kāi)了。

Part4怎么使用transformers庫(kù)加載sentencepiece模型

直接看代碼：

import?os

os.environ["PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION"]?=?"python"
from?transformers?import?LlamaTokenizer
from?sentencepiece?import?sentencepiece_model_pb2?as?sp_pb2_model
import?sentencepiece?as?spm
from?tokenization?import?ChineseTokenizer

chinese_sp_model_file?=?"sentencepisece_tokenizer/tokenizer.model"

#?load
chinese_sp_model?=?spm.SentencePieceProcessor()
chinese_sp_model.Load(chinese_sp_model_file)

chinese_spm?=?sp_pb2_model.ModelProto()
chinese_spm.ParseFromString(chinese_sp_model.serialized_model_proto())

##?Save
output_dir?=?'./transformers_tokenizer/chinese/'
os.makedirs(output_dir,?exist_ok=True)
with?open(output_dir?+?'chinese.model',?'wb')?as?f:
????f.write(chinese_spm.SerializeToString())
tokenizer?=?ChineseTokenizer(vocab_file=output_dir?+?'chinese.model')

tokenizer.save_pretrained(output_dir)
print(f"Chinese?tokenizer?has?been?saved?to?{output_dir}")

#?Test
chinese_tokenizer?=?ChineseTokenizer.from_pretrained(output_dir)
print(tokenizer.all_special_tokens)
print(tokenizer.all_special_ids)
print(tokenizer.special_tokens_map)
text?=?'''白日依山盡，黃河入海流。欲窮千里目，更上一層樓。
The?primary?use?of?LLaMA?is?research?on?large?language?models,?including'''
print("Test?text:\n",?text)
print(f"Tokenized?by?Chinese-LLaMA?tokenizer:{chinese_tokenizer.tokenize(text)}")

結(jié)果：

Chinese?tokenizer?has?been?saved?to?./transformers_tokenizer/chinese/
['<s>',?'</s>',?'<unk>']
[1,?2,?0]
{'bos_token':?'<s>',?'eos_token':?'</s>',?'unk_token':?'<unk>'}
Test?text:
?白日依山盡，黃河入海流。欲窮千里目，更上一層樓。
The?primary?use?of?LLaMA?is?research?on?large?language?models,?including
Tokenized?by?Chinese-LLaMA?tokenizer:['▁',?'白日',?'依',?'山',?'盡',?',',?'黃',?'河',?'入',?'海',?'流',?'。',?'欲',?'窮',?'千里',?'目',?',',?'更',?'上一層',?'樓',?'。',?'▁',?'T',?'h',?'e',?'▁',?'p',?'r',?'i',?'m',?'a',?'r',?'y',?'▁',?'u',?'s',?'e',?'▁',?'o',?'f',?'▁',?'LL',?'a',?'MA',?'▁i',?'s',?'▁',?'r',?'e',?'s',?'e',?'a',?'r',?'ch',?'▁',?'o',?'n',?'▁',?'l',?'a',?'r',?'g',?'e',?'▁',?'l',?'an',?'g',?'u',?'a',?'g',?'e',?'▁',?'m',?'o',?'d',?'e',?'l',?'s',?',',?'▁i',?'n',?'c',?'lu',?'d',?'i',?'ng']

其中ChineseTokenizer這里參考了llama模型里面使用的方法，并稍微做些修改：

#?coding=utf-8
#?Copyright?2022?EleutherAI?and?the?HuggingFace?Inc.?team.?All?rights?reserved.
#
#?This?code?is?based?on?EleutherAI's?GPT-NeoX?library?and?the?GPT-NeoX
#?and?OPT?implementations?in?this?library.?It?has?been?modified?from?its
#?original?forms?to?accommodate?minor?architectural?differences?compared
#?to?GPT-NeoX?and?OPT?used?by?the?Meta?AI?team?that?trained?the?model.
#
#?Licensed?under?the?Apache?License,?Version?2.0?(the?"License");
#?you?may?not?use?this?file?except?in?compliance?with?the?License.
#?You?may?obtain?a?copy?of?the?License?at
#
#?????http://www.apache.org/licenses/LICENSE-2.0
#
#?Unless?required?by?applicable?law?or?agreed?to?in?writing,?software
#?distributed?under?the?License?is?distributed?on?an?"AS?IS"?BASIS,
#?WITHOUT?WARRANTIES?OR?CONDITIONS?OF?ANY?KIND,?either?express?or?implied.
#?See?the?License?for?the?specific?language?governing?permissions?and
#?limitations?under?the?License.

"""Tokenization?classes?for?LLaMA."""
import?os
from?shutil?import?copyfile
from?typing?import?Any,?Dict,?List,?Optional,?Tuple

import?sentencepiece?as?spm

from?transformers.tokenization_utils?import?AddedToken,?PreTrainedTokenizer
from?transformers.utils?import?logging


logger?=?logging.get_logger(__name__)

VOCAB_FILES_NAMES?=?{"vocab_file":?"tokenizer.model"}

#?PRETRAINED_VOCAB_FILES_MAP?=?{
#?????"vocab_file":?{
#?????????"hf-internal-testing/llama-tokenizer":?"https://huggingface.co/hf-internal-testing/llama-tokenizer/resolve/main/tokenizer.model",
#?????},
#?????"tokenizer_file":?{
#?????????"hf-internal-testing/llama-tokenizer":?"https://huggingface.co/hf-internal-testing/llama-tokenizer/resolve/main/tokenizer_config.json",
#?????},
#?}
#?PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES?=?{
#?????"hf-internal-testing/llama-tokenizer":?2048,
#?}


class?ChineseTokenizer(PreTrainedTokenizer):
????"""
????Construct?a?Llama?tokenizer.?Based?on?byte-level?Byte-Pair-Encoding.

????Args:
????????vocab_file?(`str`):
????????????Path?to?the?vocabulary?file.
????"""

????vocab_files_names?=?VOCAB_FILES_NAMES
????#?pretrained_vocab_files_map?=?PRETRAINED_VOCAB_FILES_MAP
????#?max_model_input_sizes?=?PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
????model_input_names?=?["input_ids",?"attention_mask"]

????def?__init__(
????????self,
????????vocab_file,
????????unk_token="<unk>",
????????bos_token="<s>",
????????eos_token="</s>",
????????pad_token=None,
????????sp_model_kwargs:?Optional[Dict[str,?Any]]?=?None,
????????add_bos_token=True,
????????add_eos_token=False,
????????clean_up_tokenization_spaces=False,
????????**kwargs,
????):
????????self.sp_model_kwargs?=?{}?if?sp_model_kwargs?is?None?else?sp_model_kwargs
????????bos_token?=?AddedToken(bos_token,?lstrip=False,?rstrip=False)?if?isinstance(bos_token,?str)?else?bos_token
????????eos_token?=?AddedToken(eos_token,?lstrip=False,?rstrip=False)?if?isinstance(eos_token,?str)?else?eos_token
????????unk_token?=?AddedToken(unk_token,?lstrip=False,?rstrip=False)?if?isinstance(unk_token,?str)?else?unk_token
????????pad_token?=?AddedToken(pad_token,?lstrip=False,?rstrip=False)?if?isinstance(pad_token,?str)?else?pad_token
????????super().__init__(
????????????bos_token=bos_token,
????????????eos_token=eos_token,
????????????unk_token=unk_token,
????????????pad_token=pad_token,
????????????add_bos_token=add_bos_token,
????????????add_eos_token=add_eos_token,
????????????sp_model_kwargs=self.sp_model_kwargs,
????????????clean_up_tokenization_spaces=clean_up_tokenization_spaces,
????????????**kwargs,
????????)
????????self.vocab_file?=?vocab_file
????????self.add_bos_token?=?add_bos_token
????????self.add_eos_token?=?add_eos_token
????????self.sp_model?=?spm.SentencePieceProcessor(**self.sp_model_kwargs)
????????self.sp_model.Load(vocab_file)

????def?__getstate__(self):
????????state?=?self.__dict__.copy()
????????state["sp_model"]?=?None
????????return?state

????def?__setstate__(self,?d):
????????self.__dict__?=?d
????????self.sp_model?=?spm.SentencePieceProcessor(**self.sp_model_kwargs)
????????self.sp_model.Load(self.vocab_file)

????@property
????def?vocab_size(self):
????????"""Returns?vocab?size"""
????????return?self.sp_model.get_piece_size()

????def?get_vocab(self):
????????"""Returns?vocab?as?a?dict"""
????????vocab?=?{self.convert_ids_to_tokens(i):?i?for?i?in?range(self.vocab_size)}
????????vocab.update(self.added_tokens_encoder)
????????return?vocab

????def?_tokenize(self,?text):
????????"""Returns?a?tokenized?string."""
????????return?self.sp_model.encode(text,?out_type=str)

????def?_convert_token_to_id(self,?token):
????????"""Converts?a?token?(str)?in?an?id?using?the?vocab."""
????????return?self.sp_model.piece_to_id(token)

????def?_convert_id_to_token(self,?index):
????????"""Converts?an?index?(integer)?in?a?token?(str)?using?the?vocab."""
????????token?=?self.sp_model.IdToPiece(index)
????????return?token

????def?convert_tokens_to_string(self,?tokens):
????????"""Converts?a?sequence?of?tokens?(string)?in?a?single?string."""
????????current_sub_tokens?=?[]
????????out_string?=?""
????????prev_is_special?=?False
????????for?i,?token?in?enumerate(tokens):
????????????#?make?sure?that?special?tokens?are?not?decoded?using?sentencepiece?model
????????????if?token?in?self.all_special_tokens:
????????????????if?not?prev_is_special?and?i?!=?0:
????????????????????out_string?+=?"?"
????????????????out_string?+=?self.sp_model.decode(current_sub_tokens)?+?token
????????????????prev_is_special?=?True
????????????????current_sub_tokens?=?[]
????????????else:
????????????????current_sub_tokens.append(token)
????????????????prev_is_special?=?False
????????out_string?+=?self.sp_model.decode(current_sub_tokens)
????????return?out_string

????def?save_vocabulary(self,?save_directory,?filename_prefix:?Optional[str]?=?None)?->?Tuple[str]:
????????"""
????????Save?the?vocabulary?and?special?tokens?file?to?a?directory.

????????Args:
????????????save_directory?(`str`):
????????????????The?directory?in?which?to?save?the?vocabulary.

????????Returns:
????????????`Tuple(str)`:?Paths?to?the?files?saved.
????????"""
????????if?not?os.path.isdir(save_directory):
????????????logger.error(f"Vocabulary?path?({save_directory})?should?be?a?directory")
????????????return
????????out_vocab_file?=?os.path.join(
????????????save_directory,?(filename_prefix?+?"-"?if?filename_prefix?else?"")?+?VOCAB_FILES_NAMES["vocab_file"]
????????)

????????if?os.path.abspath(self.vocab_file)?!=?os.path.abspath(out_vocab_file)?and?os.path.isfile(self.vocab_file):
????????????copyfile(self.vocab_file,?out_vocab_file)
????????elif?not?os.path.isfile(self.vocab_file):
????????????with?open(out_vocab_file,?"wb")?as?fi:
????????????????content_spiece_model?=?self.sp_model.serialized_model_proto()
????????????????fi.write(content_spiece_model)

????????return?(out_vocab_file,)

????def?build_inputs_with_special_tokens(self,?token_ids_0,?token_ids_1=None):
????????bos_token_id?=?[self.bos_token_id]?if?self.add_bos_token?else?[]
????????eos_token_id?=?[self.eos_token_id]?if?self.add_eos_token?else?[]

????????output?=?bos_token_id?+?token_ids_0?+?eos_token_id

????????if?token_ids_1?is?not?None:
????????????output?=?output?+?bos_token_id?+?token_ids_1?+?eos_token_id

????????return?output

????def?get_special_tokens_mask(
????????self,?token_ids_0:?List[int],?token_ids_1:?Optional[List[int]]?=?None,?already_has_special_tokens:?bool?=?False
????)?->?List[int]:
????????"""
????????Retrieve?sequence?ids?from?a?token?list?that?has?no?special?tokens?added.?This?method?is?called?when?adding
????????special?tokens?using?the?tokenizer?`prepare_for_model`?method.

????????Args:
????????????token_ids_0?(`List[int]`):
????????????????List?of?IDs.
????????????token_ids_1?(`List[int]`,?*optional*):
????????????????Optional?second?list?of?IDs?for?sequence?pairs.
????????????already_has_special_tokens?(`bool`,?*optional*,?defaults?to?`False`):
????????????????Whether?or?not?the?token?list?is?already?formatted?with?special?tokens?for?the?model.

????????Returns:
????????????`List[int]`:?A?list?of?integers?in?the?range?[0,?1]:?1?for?a?special?token,?0?for?a?sequence?token.
????????"""
????????if?already_has_special_tokens:
????????????return?super().get_special_tokens_mask(
????????????????token_ids_0=token_ids_0,?token_ids_1=token_ids_1,?already_has_special_tokens=True
????????????)

????????bos_token_id?=?[1]?if?self.add_bos_token?else?[]
????????eos_token_id?=?[1]?if?self.add_eos_token?else?[]

????????if?token_ids_1?is?None:
????????????return?bos_token_id?+?([0]?*?len(token_ids_0))?+?eos_token_id
????????return?(
????????????bos_token_id
????????????+?([0]?*?len(token_ids_0))
????????????+?eos_token_id
????????????+?bos_token_id
????????????+?([0]?*?len(token_ids_1))
????????????+?eos_token_id
????????)

????def?create_token_type_ids_from_sequences(
????????self,?token_ids_0:?List[int],?token_ids_1:?Optional[List[int]]?=?None
????)?->?List[int]:
????????"""
????????Creates?a?mask?from?the?two?sequences?passed?to?be?used?in?a?sequence-pair?classification?task.?An?ALBERT
????????sequence?pair?mask?has?the?following?format:

????????```
????????0?0?0?0?0?0?0?0?0?0?0?1?1?1?1?1?1?1?1?1
????????|?first?sequence????|?second?sequence?|
????????```

????????if?token_ids_1?is?None,?only?returns?the?first?portion?of?the?mask?(0s).

????????Args:
????????????token_ids_0?(`List[int]`):
????????????????List?of?ids.
????????????token_ids_1?(`List[int]`,?*optional*):
????????????????Optional?second?list?of?IDs?for?sequence?pairs.

????????Returns:
????????????`List[int]`:?List?of?[token?type?IDs](../glossary#token-type-ids)?according?to?the?given?sequence(s).
????????"""
????????bos_token_id?=?[self.bos_token_id]?if?self.add_bos_token?else?[]
????????eos_token_id?=?[self.eos_token_id]?if?self.add_eos_token?else?[]

????????output?=?[0]?*?len(bos_token_id?+?token_ids_0?+?eos_token_id)

????????if?token_ids_1?is?not?None:
????????????output?+=?[1]?*?len(bos_token_id?+?token_ids_1?+?eos_token_id)

????????return?output

不難發(fā)現(xiàn)其實(shí)里面使用了一些sentencepiece里面的函數(shù)。

Part5怎么合并英文詞表和中文詞表？

直接看代碼：

import?os

os.environ["PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION"]?=?"python"
from?transformers?import?LlamaTokenizer
from?sentencepiece?import?sentencepiece_model_pb2?as?sp_pb2_model
import?sentencepiece?as?spm

llama_tokenizer_dir?=?"transformers_tokenizer/llama/tokenizer.model"
chinese_sp_model_file?=?"sentencepisece_tokenizer/tokenizer.model"

#?load
llama_tokenizer?=?LlamaTokenizer.from_pretrained(llama_tokenizer_dir)
chinese_sp_model?=?spm.SentencePieceProcessor()
chinese_sp_model.Load(chinese_sp_model_file)

llama_spm?=?sp_pb2_model.ModelProto()
llama_spm.ParseFromString(llama_tokenizer.sp_model.serialized_model_proto())
chinese_spm?=?sp_pb2_model.ModelProto()
chinese_spm.ParseFromString(chinese_sp_model.serialized_model_proto())

#?print?number?of?tokens
print(len(llama_tokenizer),?len(chinese_sp_model))
print(llama_tokenizer.all_special_tokens)
print(llama_tokenizer.all_special_ids)
print(llama_tokenizer.special_tokens_map)

##?Add?Chinese?tokens?to?LLaMA?tokenizer
llama_spm_tokens_set?=?set(p.piece?for?p?in?llama_spm.pieces)
print(len(llama_spm_tokens_set))
print(f"Before:{len(llama_spm_tokens_set)}")
for?p?in?chinese_spm.pieces:
????piece?=?p.piece
????if?piece?not?in?llama_spm_tokens_set:
????????new_p?=?sp_pb2_model.ModelProto().SentencePiece()
????????new_p.piece?=?piece
????????new_p.score?=?0
????????llama_spm.pieces.append(new_p)
print(f"New?model?pieces:?{len(llama_spm.pieces)}")

##?Save
output_sp_dir?=?'transformers_tokenizer/llama_chinese'
output_hf_dir?=?'transformers_tokenizer/llama_chinese'??#?the?path?to?save?Chinese-LLaMA?tokenizer
os.makedirs(output_sp_dir,?exist_ok=True)
with?open(output_sp_dir?+?'/chinese_llama.model',?'wb')?as?f:
????f.write(llama_spm.SerializeToString())
tokenizer?=?LlamaTokenizer(vocab_file=output_sp_dir?+?'/chinese_llama.model')

tokenizer.save_pretrained(output_hf_dir)
print(f"Chinese-LLaMA?tokenizer?has?been?saved?to?{output_hf_dir}")

#?Test
llama_tokenizer?=?LlamaTokenizer.from_pretrained(llama_tokenizer_dir)
chinese_llama_tokenizer?=?LlamaTokenizer.from_pretrained(output_hf_dir)
print(tokenizer.all_special_tokens)
print(tokenizer.all_special_ids)
print(tokenizer.special_tokens_map)
text?=?'''白日依山盡，黃河入海流。欲窮千里目，更上一層樓。
The?primary?use?of?LLaMA?is?research?on?large?language?models,?including'''
print("Test?text:\n",?text)
print(f"Tokenized?by?LLaMA?tokenizer:{llama_tokenizer.tokenize(text)}")
print(f"Tokenized?by?Chinese-LLaMA?tokenizer:{chinese_llama_tokenizer.tokenize(text)}")

核心部分是這一塊：

for?p?in?chinese_spm.pieces:
????piece?=?p.piece
????if?piece?not?in?llama_spm_tokens_set:
????????new_p?=?sp_pb2_model.ModelProto().SentencePiece()
????????new_p.piece?=?piece
????????new_p.score?=?0
????????llama_spm.pieces.append(new_p)

也就是將原始詞表中沒(méi)有的新加入進(jìn)去。

最后看一下結(jié)果：

32000?50000
['<s>',?'</s>',?'<unk>']
[1,?2,?0]
{'bos_token':?'<s>',?'eos_token':?'</s>',?'unk_token':?'<unk>'}
32000
Before:32000
New?model?pieces:?81163
Chinese-LLaMA?tokenizer?has?been?saved?to?transformers_tokenizer/llama_chinese
['<s>',?'</s>',?'<unk>']
[1,?2,?0]
{'bos_token':?'<s>',?'eos_token':?'</s>',?'unk_token':?'<unk>'}
Test?text:
?白日依山盡，黃河入海流。欲窮千里目，更上一層樓。
The?primary?use?of?LLaMA?is?research?on?large?language?models,?including
Tokenized?by?LLaMA?tokenizer:['▁',?'白',?'日',?'<0xE4>',?'<0xBE>',?'<0x9D>',?'山',?'<0xE5>',?'<0xB0>',?'<0xBD>',?'，',?'黃',?'河',?'入',?'海',?'流',?'。',?'<0xE6>',?'<0xAC>',?'<0xB2>',?'<0xE7>',?'<0xA9>',?'<0xB7>',?'千',?'里',?'目',?'，',?'更',?'上',?'一',?'<0xE5>',?'<0xB1>',?'<0x82>',?'<0xE6>',?'<0xA5>',?'<0xBC>',?'。',?'<0x0A>',?'The',?'▁primary',?'▁use',?'▁of',?'▁L',?'La',?'MA',?'▁is',?'▁research',?'▁on',?'▁large',?'▁language',?'▁models',?',',?'▁including']
Tokenized?by?Chinese-LLaMA?tokenizer:['▁白',?'日',?'依',?'山',?'盡',?'，',?'黃',?'河',?'入',?'海',?'流',?'。',?'欲',?'窮',?'千里',?'目',?'，',?'更',?'上一層',?'樓',?'。',?'<0x0A>',?'The',?'▁primary',?'▁use',?'▁of',?'▁L',?'La',?'MA',?'▁is',?'▁research',?'▁on',?'▁large',?'▁language',?'▁models',?',',?'▁including']

會(huì)發(fā)現(xiàn)再加入了我們定義的詞表后確實(shí)能夠?qū)χ形倪M(jìn)行分詞了。

Part6怎么使用修改后的詞表？

如果我們重新從頭開(kāi)始訓(xùn)練，那么其實(shí)使用起來(lái)很簡(jiǎn)單：

config?=?AutoConfig.from_pretrained(...)
tokenizer?=?LlamaTokenizer.from_pretrained(...)
model?=?LlamaForCausalLM.from_pretrained(...,?config=config)
model_vocab_size?=?model.get_output_embeddings().weight.size(0)
model.resize_token_embeddings(len(tokenizer))

但是如果我們想要保留原始模型embedding的參數(shù)，那么我們可以這么做：

• 1、找到新詞表和舊詞表id之間的映射關(guān)系。
• 2、將模型里面新詞表里面包含的舊詞表用原始模型的embedding替換。
• 3、如果新詞在舊詞表里面沒(méi)有出現(xiàn)就進(jìn)行相應(yīng)的初始化再進(jìn)行賦值。比如transformers庫(kù)中的llama是這么進(jìn)行初始化的：

?def?_init_weights(self,?module):
????????std?=?self.config.initializer_range
????????if?isinstance(module,?nn.Linear):
????????????module.weight.data.normal_(mean=0.0,?std=std)
????????????if?module.bias?is?not?None:
????????????????module.bias.data.zero_()
????????elif?isinstance(module,?nn.Embedding):
????????????module.weight.data.normal_(mean=0.0,?std=std)
????????????if?module.padding_idx?is?not?None:
????????????????module.weight.data[module.padding_idx].zero_()

具體怎么做可以參考一下這個(gè)：https://github.com/yangjianxin1/LLMPruner

Part7總結(jié)

到這里為止，我們已經(jīng)學(xué)會(huì)了：

• 1、使用sentencepiece訓(xùn)練一個(gè)中文的詞表。
• 2、使用transformers加載sentencepiece模型。
• 3、怎么合并中英文的詞表，并使用transformers使用合并后的詞表。
• 4、在模型中怎么使用新詞表。

Part8參考

https://github.com/ymcui/Chinese-LLaMA-Alpaca

https://github.com/yangjianxin1/LLMPruner

https://github.com/huggingface/transformers文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-497876.html