Prompt caching and token counting

Know how many input tokens a request carries, and pay less for the ones you repeat. Meta Model API caches repeated prompt prefixes automatically, and counts a request's input tokens before you send it.

Prompt caching

Reuse the same system prompt, examples, and history without reprocessing them on every turn. Meta Model API caches the stable prefix automatically — no flag or key to manage — so repeated prefixes cut time-to-first-token and input cost.

How it works

Model API uses prefix caching. For each request it compares the start of your tokenized prompt to recently cached key-value (KV) state. Where the leading tokens match, that prefix is served from cache and only the tokens after the first difference are computed from scratch.

It runs on every request with no action from you. You do not pass a cache key, set a flag, or mark breakpoints. Send your normal request and you get the benefit when a recent request shared the same prefix. Your main control is order: put stable content first, such as system prompt, instructions, and examples, and put variable content last, such as the user message. That keeps the reusable prefix as long as possible.

See what was cached

Every response tells you how much of the input came from cache. Check the endpoint-specific usage object:

json
{
  "usage": {
    "input_tokens": 1847,
    "output_tokens": 98,
    "total_tokens": 1945,
    "input_tokens_details": {
      "cached_tokens": 1792
    },
    "output_tokens_details": {
      "reasoning_tokens": 0
    }
  }
}

json
{
  "usage": {
    "prompt_tokens": 1847,
    "completion_tokens": 98,
    "total_tokens": 1945,
    "prompt_tokens_details": {
      "cached_tokens": 1792
    }
  }
}

Chat Completions reports it at usage.prompt_tokens_details.cached_tokens; Responses reports it at usage.input_tokens_details.cached_tokens. In both cases cached_tokens counts input tokens served from cache. A value above zero means the server reused a cached prefix and skipped that prefill. Zero is expected on the first request with a new prefix or after eviction.

Structure your prompt for cache hits

The cache matches from the start of the prompt forward, so order determines how much can be reused:

System prompts: a long, stable system prompt is ideal for caching. If every request starts with the same 2,000-token system prompt, those tokens are served from cache after the first request.
Few-shot examples: examples that stay constant across requests are cached as part of the prefix.
Conversation history: in a multi-turn conversation the accumulated history forms a growing prefix. Each new turn only needs to compute KV state for the new user message and the latest response.

Tokens that differ from the cached prefix end the match, and everything after the first difference is computed fresh. Putting volatile content like a timestamp or the current user message early shortens the reusable prefix.

Group similar requests with a cache key

You get caching with no key at all. At high volume across many backends, you can optionally send prompt_cache_key to increase hit rate. Requests that share a key route together, so a request is more likely to land on a backend that already holds its prefix. Pick a stable string that identifies the shared prefix, such as an application name or use case, not a per-user or per-request value.

prompt_cache_key is accepted on both Responses and Chat Completions, and replaces the deprecated user field.

python
import os
from openai import OpenAI
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
response = client.chat.completions.create(
    model="muse-spark-1.3",
    prompt_cache_key="my-app-system-prompt",
    messages=[
        {
            "role": "system",
            "content": "You are a senior technical support engineer for Acme Corp. You have deep knowledge of all Acme products, APIs, and billing systems. Always provide step-by-step instructions. Be concise and professional.",
        },
        {
            "role": "user",
            "content": "How do I rotate my API keys without downtime?",
        },
    ],
)
print(response.model_dump_json(indent=2))

typescript
import OpenAI from 'openai';
const apiKey = process.env.MODEL_API_KEY;
if (!apiKey) {
  throw new Error('MODEL_API_KEY is not set');
}
const client = new OpenAI({
  baseURL: 'https://api.meta.ai/v1',
  apiKey,
});
const response = await client.chat.completions.create({
  model: 'muse-spark-1.3',
  prompt_cache_key: 'my-app-system-prompt',
  messages: [
    {
      role: 'system',
      content: 'You are a senior technical support engineer for Acme Corp. You have deep knowledge of all Acme products, APIs, and billing systems. Always provide step-by-step instructions. Be concise and professional.',
    },
    {
      role: 'user',
      content: 'How do I rotate my API keys without downtime?',
    },
  ],
});
console.log(JSON.stringify(response, null, 2));

python
import json
import os
import requests
response = requests.post(
    "https://api.meta.ai/v1/chat/completions",
    headers={
        "Authorization": f"Bearer {os.environ['MODEL_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "model": "muse-spark-1.3",
        "prompt_cache_key": "my-app-system-prompt",
        "messages": [
            {
                "role": "system",
                "content": "You are a senior technical support engineer for Acme Corp. You have deep knowledge of all Acme products, APIs, and billing systems. Always provide step-by-step instructions. Be concise and professional.",
            },
            {
                "role": "user",
                "content": "How do I rotate my API keys without downtime?",
            },
        ],
    },
)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))

shell
curl -X POST "https://api.meta.ai/v1/chat/completions" \
  -H "Authorization: Bearer $MODEL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "muse-spark-1.3",
  "prompt_cache_key": "my-app-system-prompt",
  "messages": [
    {
      "role": "system",
      "content": "You are a senior technical support engineer for Acme Corp. You have deep knowledge of all Acme products, APIs, and billing systems. Always provide step-by-step instructions. Be concise and professional."
    },
    {
      "role": "user",
      "content": "How do I rotate my API keys without downtime?"
    }
  ]
}'

Use one stable key per shared prefix: a good key names your app or use case, such as "customer-support-agent" or "code-review-v2".
Don't over-partition: unique keys per user or per session lower hit rates because each key routes independently. Share a key across requests that truly share a prefix.

Extend retention on Responses

By default a cached prefix lives in memory and is evicted under pressure or after inactivity. On the Responses API, prompt_cache_retention optionally requests longer retention:

Value	Behavior
`"in_memory"`	Keep the cache in memory (the default behavior). Fastest access; evicted under memory pressure.
`"24h"`	Request extended retention, keeping the cached prefix active for longer, up to 24 hours.

json
{
  "model": "muse-spark-1.3",
  "prompt_cache_retention": "24h",
  "input": "Summarize the incident report from this quarter."
}

Check cache performance

Read the cached-token usage field on each response to confirm caching is helping:

cached_tokens is most of input_tokens: cache is working well. Only the new tokens, such as the latest user message, needed computation.
cached_tokens is zero: no cache hit. Expected on the first request with a new prefix, or after eviction.
cached_tokens is lower than expected: the prefix likely changed partway through, such as an edited system prompt, ending the match early.

Counts reflect the request sent

These usage counts reflect the tokens in the request you actually send. On the Responses API with previous_response_id, the server reconstructs prior turns from stored history, so a follow-up reports only the new turn's input_tokens and cached_tokens, not the full reconstructed prefix. To see accounting across the entire prefix, manage the input array yourself instead of chaining with previous_response_id. To measure the full context, stored history included, before you send, use token counting.

Best practices

Put stable content first: this is the main lever. The cache matches from the beginning, so system prompts, instructions, and few-shot examples should come before dynamic content. Volatile content early shortens the reusable prefix.
Monitor cached_tokens: track the ratio of cached_tokens to input tokens over time to confirm your prompt structure caches well.
Add a prompt_cache_key at scale: optional, but it raises hit rates for high-volume workloads by keeping requests that share a prefix on the same backend.
Set "24h" retention for bursty workloads: on Responses, if traffic arrives in bursts with idle gaps, extended retention avoids rebuilding the cache between bursts.

Token counting

Get an exact token count before you run inference. You stay inside the context window and size batches safely. Meta Model API renders the request and runs the selected model's tokenizer, so the count includes the context the model would receive.

Count input tokens before you send

POST /v1/responses/input_tokens returns the number of tokens in the fully rendered input for the model call represented by your request. Send the supported fields from the body you would send to /v1/responses, at minimum model and input, then read input_tokens from the result. The server resolves previous_response_id history and applies the current request's instructions, tool definitions, media, and prompt scaffolding before tokenization.

python
import json
import os
import requests
response = requests.post(
    "https://api.meta.ai/v1/responses/input_tokens",
    headers={
        "Authorization": f"Bearer {os.environ['MODEL_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "model": "muse-spark-1.3",
        "input": "What is the capital of France?",
    },
)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))

shell
curl -X POST "https://api.meta.ai/v1/responses/input_tokens" \
  -H "Authorization: Bearer $MODEL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "muse-spark-1.3",
  "input": "What is the capital of France?"
}'

The response contains only the count:

text
{
  "object": "response.input_tokens",
  "input_tokens": <integer>
}

Validation: The endpoint applies the same initial request and replay-structure validation as /v1/responses for supported fields, including function_call.arguments JSON validity, call_id matching, tool definitions, and function-name constraints. Conversation resources are not supported; use previous_response_id for multi-turn history. If a check fails, the endpoint returns HTTP 400 with the same error shape as a generation request. A successful count validates and tokenizes the initial rendered context; it does not predict context growth or validation after future hosted-tool results.

The endpoint tokenizes only. It does not generate output, execute hosted tools, create or store a response, or record generation usage. Use the result to check context-window fit, not to estimate billing. For the full endpoint reference, see Count input tokens.

Token counts are model-specific

A count only applies to the model that produced it. Tokenizers differ per model, so the same text can map to different totals on different models. Always count against the model you will actually call, and recount when you switch models.

This is why you count through the API rather than a local tokenizer library: the input_tokens endpoint applies the target model's prompt template and tokenizer to the reconstructed request.

Compare rendered context with response usage

Every prompt includes Meta-injected steering context: a hidden system prompt, a default developer message, and tool scaffolding for built-in tools such as web search. The count endpoint includes these tokens because they occupy the model's context.

Generated response usage is separate accounting data. Requests that execute hosted tools report usage.input_tokens cumulatively across model iterations. Each iteration contributes its reported input-token count after applying accounting adjustments, including injected-token discounts. Cumulative usage can therefore exceed both the preflight count and the final model context size.

Do not use usage.input_tokens as a final-context occupancy value. Use POST /v1/responses/input_tokens on the reconstructed request when you need a context-size measurement.

Count multimodal input

Images and files contribute to the input total. Their token contribution depends on the selected model and version, prompt template, media preprocessing, and image resolution. Fixed per-image or prompt-overhead estimates are not portable across deployments.

Measure representative payloads again after a model, tokenizer, prompt-template, or media-processing deployment. Use input_tokens instead of estimating from image count or dimensions. See supported image formats and limits.

Read token usage after a response

Every non-streaming response reports accounting data in a usage object. Field names vary by endpoint:

Responses: { input_tokens, output_tokens, total_tokens, input_tokens_details: { cached_tokens }, output_tokens_details: { reasoning_tokens } }.
Chat Completions: { prompt_tokens, completion_tokens, total_tokens, prompt_tokens_details: { cached_tokens }, completion_tokens_details: { reasoning_tokens } }.

reasoning_tokens counts chain-of-thought tokens, a subset of the output total; cached_tokens counts prompt tokens served from the prompt cache. For a Responses request with hosted tools, input_tokens adds the reported input-token count from every model iteration after per-iteration accounting adjustments. When you stream with Chat Completions, request a final usage chunk with stream_options: {"include_usage": true}; on the Responses API, read usage from the final response object.

Count tokens for Anthropic-format requests

If you call the Anthropic-compatible Messages API, count tokens with POST /v1/messages/count_tokens. It accepts the same Anthropic-shaped body as /v1/messages and returns { "input_tokens": <integer> }. See Count tokens for the reference.

Next steps

Stay inside the window: check Models for the context limit, then compact history as conversations grow.
Wire Count input tokens into preflight checks before large or variable requests.
Use the Responses API to carry context across turns, with stateless reasoning replay or previous_response_id.
Add tool calling to a cached system prompt so tool definitions are computed once and reused across turns.
Combine with structured output for low-latency extraction pipelines that reuse the same schema instructions.