Model Cards & Prompt formats

Llama 3.1

Introduction

This page describes the prompt format for Llama 3.1 with an emphasis on new features in that release.

With the subsequent release of Llama 3.2, we have introduced new lightweight models in 1B and 3B and also multimodal models in 11B and 90B. The first few sections of this page--Prompt Template, Base Model Prompt, and Instruct Model Prompt--are applicable across all the models released in both Llama 3.1 and Llama 3.2.

Please leverage this guidance in order to take full advantage of the new Llama models. Although prompts designed for Llama 3 should work unchanged in Llama 3.1 and Llama 3.2, we recommend that you update your prompts to the new format to obtain the best results.

Note: We have introduced Llama 3.3 70B, an instruction-turned model with the latest advancements in post-training techniques; see the model card for detailed performance information. This new model supersedes the instruction-tuned Llama 3.1 70B model. The new model name is Llama-3.3-70B-Instruct; developers should install and use the new model wherever they would otherwise have used instruction-tuned Llama 3.1 70B.

Download the new Llama 3.3 model.

Model Card

For comprehensive technical information about the Llama 3.1 collection of large-language models, please see the official model card, located on GitHub.

Download Llama 3.1 models.

Prompt Template

Special Tokens

Tokens

Description

Supported Roles

There are 4 different roles that are supported by Llama text models:

system: Sets the context in which to interact with the AI model. It typically includes rules, guidelines, or necessary information that help the model respond effectively.
user: Represents the human interacting with the model. It includes the inputs, commands, and questions to the model.
ipython: A new role introduced in Llama 3.1. Semantically, this role means "tool". This role is used to mark messages with the output of a tool call when sent back to the model from the executor.
assistant: Represents the response generated by the AI model based on the context provided in the system, ipython and user prompts.

[system, assistant, user, ipython]

Pretrained Model Prompt

This is the simplest way to prompt llama-based models, starting with text and allowing the model to generate new content based on the user-provided input.

xml
<|begin_of_text|>{{ user_message}}

Instruct Model Prompt

The format for a regular multi-turn conversation between a user and the model has not changed between Llama 3 and Llama 3.1. Here is a simple example of how it's formatted.

xml
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Cutting Knowledge Date: December 2023
Today Date: 23 July 2024
You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>
What is the capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

For more examples of the prompt template, please refer to text_prompt_format.md in the meta-llama GitHub repository.

Download the new Llama 3.3 model.

Code Interpreter

The code interpreter feature allows the model to generate executable Python code in response to the user query. The code interpreter is triggered when Environment: ipython is included in the system prompt.

Notes

The model’s response is wrapped in a <|python_tag|> and terminated with an <|eom_id|> tag.
The <|eom_id|> indicates a continued multi-step reasoning. That is, the model is expecting a continuation message with the output of the tool call.

Sample Input:

xml
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Environment: ipython<|eot_id|><|start_header_id|>user<|end_header_id|>
Write code to check if number is prime, use that to see if the number 7 is prime<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Response:

xml
<|python_tag|>def is_prime(n):
    if n <= 1:
        return False
    for i in range(2, int(n**0.5) + 1):
        if n % i == 0:
            return False
    return True
print(is_prime(7))  # Output: True<|eom_id|>

Note that the model itself does not execute the code output; you need to use a code interpreter framework (such as those available in the llama-cookbook) to execute the generated code and pass results back to the model for further processing.

Tool Calling (8B/70B/405B)

Tool-calling with the larger models can be done in three ways:

Built-in Tools (Brave, Wolfram)
JSON-based Tool Calling
User-defined Custom Tool Calling

Note: We recommend using Llama 70B-instruct or Llama 405B-instruct for applications that combine conversation and tool calling. Llama 8B-Instruct can not reliably maintain a conversation alongside tool calling definitions. It can be used for zero-shot tool calling, but tool instructions should be removed for regular conversations between the model and the user.

Defining multiple tools in the system and/or user prompt can cause the model to generate multiple tool calls as a result of the user query. Developers must iterate on the prompts required to get the desired result and parse the result accordingly.

To better illustrate the flows below, this diagram shows the flow of information described:

User and assistant conversation

The format for a regular multi-turn conversation between a user and the model has not changed between Llama 3 and Llama 3.1. Here is a simple example of how it's formatted.

xml
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Cutting Knowledge Date: December 2023
Today Date: 23 July 2024
You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>
What is the capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Built in Python based tool calling

The models are trained to use two built-in tools:

Brave Search: Tool call to perform web searches.
Wolfram Alpha: Tool call to perform complex mathematical calculations.

These can be turned on using the system prompt:

xml
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Environment: ipython
Tools: brave_search, wolfram_alpha
Cutting Knowledge Date: December 2023
Today Date: 23 July 2024
You are a helpful assistant<|eot_id|>
<|start_header_id|>user<|end_header_id|>
What is the current weather in Menlo Park, California?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

A few things to note:

Just including Environment: ipython turns on code interpreter; therefore, you don’t need to specify code interpretation on the Tools: line. The model can generate python code which is interpreted by the executor, with the result provided back to the model.
The message body of the assistant response starts with a special tag <|python_tag|>
As alluded to above, in such an environment, the model can generate <|eom_id|> instead of just the standard <|eot_id|> . The latter indicates the turn is finished, while the former indicates continued multi-step reasoning. That is, the model is expecting a continuation message with the output of the tool call.

The built-in tool is called using python syntax, and not JSON format as for zero-shot tools. The python call that the model generates for each of the tools is of the form:

xml
# for Search
<|python_tag|>
brave_search.call(query="...")
<|eom_id|>
# for Wolfram
<|python_tag|>
wolfram_alpha.call(query="...")
<|eom_id|>

To create custom tool calls using the zero-shot learning capability of the model, you can provide instructions for the tool definition in the system prompt along with the format you want the model to produce for the call. A working example is provided in the GitHub repository. You can check it out here.

Step - 1 User Prompt & System prompts

xml
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Environment: ipython
Tools: brave_search, wolfram_alpha
Cutting Knowledge Date: December 2023
Today Date: 23 July 2024
You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
Can you help me solve this equation: x^3 - 4x^2 + 6x - 24 = 0<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Step - 2 Model determining which tool to call

xml
<|python_tag|>wolfram_alpha.call(query="solve x^3 - 4x^2 + 6x - 24 = 0")<|eom_id|>

Step - 3 Response generated by tool, that is, Wolfram Alpha.

javascript
{
  "queryresult": {
    "success": true,
    "inputstring": "solve x^3 - 4x^2 + 6x - 24 = 0",
    "pods": [
      {
        "title": "Input interpretation",
        "subpods": [
          {
            "title": "",
            "plaintext": "solve x^3 - 4 x^2 + 6 x - 24 = 0"
          }
        ]
      },
      {
        "title": "Results",
        "primary": true,
        "subpods": [
          {
            "title": "",
            "plaintext": "x = 4"
          },
          {
            "title": "",
            "plaintext": "x = ± (i sqrt(6))"
          }
        ]
      },
      ...
    ]
  }
}

Step - 4 Reprompt Model with tool response

xml
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Environment: ipython
Tools: brave_search, wolfram_alpha
Cutting Knowledge Date: December 2023
Today Date: 23 July 2024
You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
Can you help me solve this equation: x^3 - 4x^2 + 6x - 24 = 0<|eot_id|><|start_header_id|>assistant<|end_header_id|>
<|python_tag|>wolfram_alpha.call(query="solve x^3 - 4x^2 + 6x - 24 = 0")<|eom_id|><|start_header_id|>ipython<|end_header_id|>
{"queryresult": {"success": true, "inputstring": "solve x^3 - 4x^2 + 6x - 24 = 0", "pods": [{"title": "Input interpretation", "subpods": [{"title": "", "plaintext": "solve x^3 - 4 x^2 + 6 x - 24 = 0"}]}, {"title": "Results", "primary": true, "subpods": [{"title": "", "plaintext": "x = 4"}, {"title": "", "plaintext": "x = \u00b1 (i sqrt(6))"}]}, ... ]}}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Step - 5 Response from Agent to User

xml
The solutions to the equation x^3 - 4x^2 + 6x - 24 = 0 are x = 4 and x = ±(i√6).<|eot_id|>

JSON based tool calling

Meta Llama models can now output custom tool calls from a single message to allow easier tool calling. The following prompts provide an example of how custom tools can be called from the output of the model. It's important to note that the model itself does not execute the calls; it provides structured output to facilitate calling by an executor. An example executor can be found in the llama-agentic-system. The tool format is similar to the OpenAI definition and can be adjusted to better suit your needs, as noted in the following section.

Notes

With custom tool calling, for the model to output the eom_id, the following instruction needs to be added to the system prompt: Environment: ipython. Else, it should output eot_id.
The system prompt needs to be adjusted to inform the model how to deal with tool call outputs
The tool definition is provided in the user prompt, as that is how the model was trained for the built in JSON tool calling. However, it's possible to provide the tool definition in the system prompt as well—and get similar results. Developers must test which way works best for their use case.

Step - 1 User prompt with custom tool details

xml
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Cutting Knowledge Date: December 2023
Today Date: 23 July 2024
When you receive a tool call response, use the output to format an answer to the orginal user question.
You are a helpful assistant with tool calling capabilities.<|eot_id|><|start_header_id|>user<|end_header_id|>
Given the following functions, please respond with a JSON for a function call with its proper arguments that best answers the given prompt.
Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. Do not use variables.
{
    "type": "function",
    "function": {
    "name": "get_current_conditions",
    "description": "Get the current weather conditions for a specific location",
    "parameters": {
        "type": "object",
        "properties": {
        "location": {
            "type": "string",
            "description": "The city and state, e.g., San Francisco, CA"
        },
        "unit": {
            "type": "string",
            "enum": ["Celsius", "Fahrenheit"],
            "description": "The temperature unit to use. Infer this from the user's location."
        }
        },
        "required": ["location", "unit"]
    }
    }
}
Question: what is the weather like in San Fransisco?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Step - 2 Model determining which tool to call

Note:

As we are not using the Environment: ipython instruction in the system prompt, the model ends generation with the eot_id token, not the eom_id token

xml
{"name": "get_current_conditions", "parameters": {"location": "San Francisco, CA", "unit": "Fahrenheit"}}<eot_id>

Step - 3 Result from calling the tool is passed back to the model

When the result of the tool call is sent back to the model, we use the new role, ipython, in the header.

xml
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant with tool calling capabilities. When you receive a tool call response, use the output to format an answer to the orginal use question.<|eot_id|><|start_header_id|>user<|end_header_id|>
Given the following functions, please respond with a JSON for a function call with its proper arguments that best answers the given prompt.
Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. Do not use variables.
{
    "type": "function",
    "function": {
    "name": "get_current_conditions",
    "description": "Get the current weather conditions for a specific location",
    "parameters": {
        "type": "object",
        "properties": {
        "location": {
            "type": "string",
            "description": "The city and state, e.g., San Francisco, CA"
        },
        "unit": {
            "type": "string",
            "enum": ["Celsius", "Fahrenheit"],
            "description": "The temperature unit to use. Infer this from the user's location."
        }
        },
        "required": ["location", "unit"]
    }
    }
}
Question: what is the weather like in San Fransisco?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
{"name": "get_current_conditions", "parameters": {"location": "San Francisco, CA", "unit": "Fahrenheit"}}<|eot_id|><|start_header_id|>ipython<|end_header_id|>
{"output": "Clouds giving way to sun Hi: 76° Tonight: Mainly clear early, then areas of low clouds forming Lo: 56°"}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Step - 4 The model generates the final response the user

xml
The weather in Menlo Park is currently cloudy with a high of 76° and a low of 56°, with clear skies expected tonight.<eot_id>

User-defined Custom tool calling

The following prompts provide an example of how to define your own format in the system prompt and make use of it while interacting with the model. This format will use a new and custom <function> tag to enclose a JSON object with the function parameters.

Notes

The model uses the eom_id token because of the Environment: ipython instruction.
The system prompt defines the custom format as well as the tools the model has available.

Step - 1 System prompt with custom tool details

xml
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Environment: ipython
Tools: brave_search, wolfram_alpha
Cutting Knowledge Date: December 2023
Today Date: 23 July 2024
# Tool Instructions
- Always execute python code in messages that you share.
- When looking for real time information use relevant functions if available else fallback to brave_search
You have access to the following functions:
Use the function 'spotify_trending_songs' to: Get top trending songs on Spotify
{
  "name": "spotify_trending_songs",
  "description": "Get top trending songs on Spotify",
  "parameters": {
    "n": {
      "param_type": "int",
      "description": "Number of trending songs to get",
      "required": true
    }
  }
}
If a you choose to call a function ONLY reply in the following format:
<{start_tag}={function_name}>{parameters}{end_tag}
where
start_tag => `<function`
parameters => a JSON dict with the function argument name as key and function argument value as value.
end_tag => `</function>`
Here is an example,
<function=example_function_name>{"example_name": "example_value"}</function>
Reminder:
- Function calls MUST follow the specified format
- Required parameters MUST be specified
- Only call one function at a time
- Put the entire function call reply on one line
- Always add your sources when using search results to answer the user query
You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
Can you check the top 5 trending songs on spotify?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Step - 2 Model determining which tool to call

Now the model responds in accordance with the format defined in the system prompt, with the function name in the start tag, and the parameters, as a JSON object, in between the start and end tags.

Note:

As we are using the Environment: ipython instruction in the system prompt, the model ends generation with the eom_id token.

xml
<function=spotify_trending_songs>{"n": "5"}</function><eom_id>

Step - 3 Result from calling the tool is passed back to the model

When the result of the tool call is sent back to the model, we use the new role, ipython, in the header.

xml
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Environment: ipython
Tools: brave_search, wolfram_alpha
Cutting Knowledge Date: December 2023
Today Date: 23 July 2024
# Tool Instructions
- Always execute python code in messages that you share.
- When looking for real time information use relevant functions if available else fallback to brave_search
You have access to the following functions:
Use the function 'spotify_trending_songs' to: Get top trending songs on Spotify
{"name": "spotify_trending_songs", "description": "Get top trending songs on Spotify", "parameters": {"n": {"param_type": "int", "description": "Number of trending songs to get", "required": true}}}
If a you choose to call a function ONLY reply in the following format:
<{start_tag}={function_name}>{parameters}{end_tag}
where
start_tag => `<function`
parameters => a JSON dict with the function argument name as key and function argument value as value.
end_tag => `</function>`
Here is an example,
<function=example_function_name>{"example_name": "example_value"}</function>
Reminder:
- Function calls MUST follow the specified format
- Required parameters MUST be specified
- Only call one function at a time
- Put the entire function call reply on one line
- Always add your sources when using search results to answer the user query
You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
Can you check the top 5 trending songs on spotify?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
<function=spotify_trending_songs>{"n": "5"}</function><|eom_id|><|start_header_id|>ipython<|end_header_id|>
["1. BIRDS OF A FEATHER by Billie Eilish", "2. Espresso by Sabrina Carpenter", "3. Please Please Please by Sabrina Carpenter", "4. Not Like Us by Kendrick Lamar", "5. Gata Only by FloyyMenor, Cris Mj"]<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Step - 4 The model generates the final response the user

xml
The top 5 trending songs on Spotify are:
1. BIRDS OF A FEATHER by Billie Eilish
2. Espresso by Sabrina Carpenter
3. Please Please Please by Sabrina Carpenter
4. Not Like Us by Kendrick Lamar
5. Gata Only by FloyyMenor, Cris Mj<|eot_id|>

Please confirm your identity

Please confirm your identity

Llama 3.1

Introduction

Model Card

Prompt Template

Special Tokens

Supported Roles

Pretrained Model Prompt

Instruct Model Prompt

Code Interpreter

Tool Calling (8B/70B/405B)

User and assistant conversation

Built in Python based tool calling

JSON based tool calling

User-defined Custom tool calling