Step 1: Load the original Llama tokenizer

First, we load the original Llama tokenizer and its vocabulary.

python
from transformers import LlamaTokenizer
​
llama_tokenizer = LlamaTokenizer.from_pretrained('meta-llama/Llama-2-7b-chat-hf')

Step 2: Load the new language tokenizer

Next, we load the new language tokenizer--in this case, Hindi--and its vocabulary.

python
new_tokenizer = LlamaTokenizer.from_pretrained('./hi_tokenizer')

Step 3: Merge the tokenizers

We then merge the two tokenizers by adding the new tokens to the original tokenizer.

python
python extend_tokenizer.py --new_tokenizer_path=./hi_tokenizer --extended_tokenizer_save_path=./extended_tokenizer

The extend_tokenizer.py script appends the new tokens to the original tokenizer's vocabulary and saves the extended tokenizer.

Step 4: Verify the merged tokenizer

We verify that the merged tokenizer works as expected by testing it on a text sample.

python
text = "मैं एक अच्छा हाथी हूँ"
our_tokenizer = LlamaTokenizer.from_pretrained('./extended_tokenizer')
print(our_tokenizer.tokenize(text))

Output:

python
['▁मैं', '▁एक', '▁अच', '्', 'छा', '▁हाथी', '▁हूँ']