CommunityMultilingualFine-tuning
Extend Llama to a New Language: A Technical GuideLlama is a powerful language model that has shown impressive results in various natural-language processing tasks.Read moreTo merge the tokenizers, we need to combine the original Llama tokenizer with the new language tokenizer. This involves adding new tokens to the original tokenizer without disturbing its original vocabulary.
First, we load the original Llama tokenizer and its vocabulary.
python
Next, we load the new language tokenizer--in this case, Hindi--and its vocabulary.
python
We then merge the two tokenizers by adding the new tokens to the original tokenizer.
python
The extend_tokenizer.py script appends the new tokens to the original tokenizer's vocabulary and saves the extended tokenizer.
We verify that the merged tokenizer works as expected by testing it on a text sample.
python
Output:
python
RAFT is a technique that combines fine-tuning with Retrieval Augmented Generation (RAG) to better utilize custom domain text data. RAFT teaches the model to differentiate between oracle documents (D*) that contain the answer to a question and distractor documents (Di) that do not.
RAFT architecture
To fine-tune Llama for the new language, we need to prepare a dataset with questions, oracle documents, and distractor documents.
We prepare a dataset with the following format:
We fine-tune Llama using the RAFT technique, which involves training the model to differentiate between oracle documents and distractor documents.
