Running Llama on Windows

For our script, open the editor of your choice, and create a Python script. We’ll first add the imports that we need for our example:

py
import transformers
import torch
from transformers import AutoTokenizer

Let's define the model we’d like to use. In our demo, we will use the 8B instruct model which is fine tuned for chat:

py
model = "meta-llama/Llama-3.1-8B-Instruct"

We will also instantiate the tokenizer which can be derived from AutoTokenizer, based on the model we’ve chosen, using the from_pretrained method of AutoTokenizer. This will download and cache the pre-trained tokenizer and return an instance of the appropriate tokenizer class.

py
tokenizer = AutoTokenizer.from_pretrained(model)

To use our model for inference:

py
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

Hugging Face pipelines allow us to specify which type of task the pipeline needs to run (text-generation in this case), the model that the pipeline should use to make predictions (specified by model), the precision to use with this model (torch.float16), the device on which the pipeline should run (device_map), and various other options. We’ll also set the device_map argument to auto, which means the pipeline will automatically use a GPU if one is available.

Next, let's provide some text prompts as inputs to our pipeline for it to use when it runs to generate responses. Let’s define this as the variable, sequences:

py
sequences = pipeline(
    'I have tomatoes, basil and cheese at home. What can I cook for dinner?\n',
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    truncation = True,
    max_length=400,
)

The pipeline sets do_sample to True, which allows us to specify the decoding strategy we’d like to use to select the next token from the probability distribution over the entire vocabulary. In our example, we are using top_k sampling.

By changing max_length, you can specify how long you’d like the generated response to be. Setting the num_return_sequences parameter to greater than one will let you generate more than one output.

Finally, we add the following to provide input, and information on how to run the pipeline:

py
for seq in sequences:
    print(f"Result: {seq['generated_text']}")

Save your script and head back to the terminal. We will save it as llama3-hf-demo.py. Before we run the script, let’s make sure we can access and interact with Hugging Face directly from the terminal. To do that, make sure you have the Hugging Face CLI installed:

bash
pip install -U "huggingface_hub[cli]"

followed by

bash
huggingface-cli login

Here, it will ask us for our access token which we can get from our HF account under Settings. Copy it and provide it in the command line. We are now all set to run our script.

bash
python llama3-hf-demo.py

Running Llama-3.1-8B-Instruct locally

To check out the full example and run it on your local machine, see the detailed sample notebook that you can refer to in the llama-cookbook GitHub repo. Here you will find an example of how to run Llama 3 models using already converted Hugging Face weights, as well as an example that goes over how you can convert the original weights into Hugging Face format and run using those.

We’ve also created various other demos and examples to provide you with guidance and as references to help you get started with Llama models and to make it easier for you to integrate them into your own use cases. To try these examples, check out our llama-cookbook GitHub repo or install llama-cookbook from PyPI. You’ll find complete walkthroughs for how to get started with Llama models, including examples of inference, fine tuning, and training on custom data sets. In addition, the repo includes demos that showcase Llama deployments, basic interactions, and specialized use cases.