To merge the tokenizers, we need to combine the original Llama tokenizer with the new language tokenizer. This involves adding new tokens to the original tokenizer without disturbing its original vocabulary.

Step 1: Load the original Llama tokenizer

First, we load the original Llama tokenizer and its vocabulary.

python
from transformers import LlamaTokenizer
​
llama_tokenizer = LlamaTokenizer.from_pretrained('meta-llama/Llama-2-7b-chat-hf')

Step 2: Load the new language tokenizer

Next, we load the new language tokenizer--in this case, Hindi--and its vocabulary.

python
new_tokenizer = LlamaTokenizer.from_pretrained('./hi_tokenizer')

Step 3: Merge the tokenizers

We then merge the two tokenizers by adding the new tokens to the original tokenizer.

python
python extend_tokenizer.py --new_tokenizer_path=./hi_tokenizer --extended_tokenizer_save_path=./extended_tokenizer

The extend_tokenizer.py script appends the new tokens to the original tokenizer's vocabulary and saves the extended tokenizer.

Step 4: Verify the merged tokenizer

We verify that the merged tokenizer works as expected by testing it on a text sample.

python
text = "मैं एक अच्छा हाथी हूँ"
our_tokenizer = LlamaTokenizer.from_pretrained('./extended_tokenizer')
print(our_tokenizer.tokenize(text))

Output:

python
['▁मैं', '▁एक', '▁अच', '्', 'छा', '▁हाथी', '▁हूँ']