Deploy with vLLM

Serve Muse Glimmer with vLLM(opens in new tab) for high-throughput, low-latency inference with an OpenAI-compatible API endpoint. vLLM handles continuous batching, PagedAttention, and tensor parallelism out of the box.

The vLLM recipe is source of truth

The official vLLM Muse Glimmer recipe(opens in new tab) is the source of truth for the image tag, flags, and hardware requirements. vLLM moves faster than this page: check the recipe first when a command here doesn't match what you see.

Prerequisites

  • Docker with the NVIDIA container runtime
  • An NVIDIA GPU with drivers for CUDA 13.0, which is what the image is pinned to
  • 72 GB of VRAM to serve, on one card or across several

Install

Docker is the supported path:

bash
docker pull vllm/vllm-openai:muse-glimmer

pip install vllm doesn't serve this model. Muse Glimmer support in vLLM is an open, unmerged pull request (vllm-project/vllm#51655(opens in new tab)), so the model code and the muse_glimmer parsers are absent from every released wheel. The recipe sets pip: false for that reason, and the image is how you get the unreleased code.

The image is roughly 10.5 GB and publishes linux/amd64 and linux/arm64. There's no published ROCm image; on ROCm, build from the pull request.

What the image gives you:

CapabilityHow you get it
Text model (MuseGlimmerForCausalLM)Automatic, with no trust_remote_code
Tool-call parser--tool-call-parser muse_glimmer
Reasoning parser--reasoning-parser muse_glimmer
Chat templateShips with the checkpoint as chat_template.jinja. Don't pass --chat-template.
Model configAutomatic (muse-glimmer, muse_glimmer_text, muse_glimmer_vision)

Parser names use underscores. vLLM matches muse_glimmer literally, so the hyphenated spelling fails. --served-model-name muse-glimmer is a label you choose, and it stays hyphenated.

Start the server

The image sets ENTRYPOINT ["vllm", "serve"], so everything after the image name is appended to that. Pass the model and flags directly rather than retyping vllm serve or calling python -m vllm.entrypoints.openai.api_server.

bash
docker run --rm --gpus all --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:muse-glimmer \
  meta-models/Muse-Glimmer-30B \
  --served-model-name muse-glimmer \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer \
  --generation-config auto

The flags from --served-model-name down are the recipe's argument list for a single card. The docker run wrapper around them is standard vLLM boilerplate, so adjust it to your host. Mounting ~/.cache/huggingface is what stops the container re-downloading roughly 60 GB of weights on every start.

To serve weights you've already downloaded, mount them and pass the path instead of a Hub id:

bash
docker run --rm --gpus all --ipc=host \
  -p 8000:8000 \
  -v /path/to/Muse-Glimmer-30B:/model \
  vllm/vllm-openai:muse-glimmer \
  /model \
  --served-model-name muse-glimmer \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer \
  --generation-config auto

To serve across multiple GPUs, raise the tensor-parallel size:

bash
docker run --rm --gpus all --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:muse-glimmer \
  meta-models/Muse-Glimmer-30B \
  --served-model-name muse-glimmer \
  --tensor-parallel-size 2 \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer \
  --generation-config auto

For tighter memory, at a cost in throughput and context:

bash
docker run --rm --gpus all --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:muse-glimmer \
  meta-models/Muse-Glimmer-30B \
  --served-model-name muse-glimmer \
  --gpu-memory-utilization 0.9 --max-model-len 8300 --enforce-eager \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer \
  --generation-config auto
Budget 72 GB of VRAM

That is what the recipe asks for: the bf16 weights are roughly 60 GB, and the rest goes to KV cache, activations, and CUDA overhead. Sizing to the weights alone is the usual way to run out of memory shortly after startup.

The flags that carry Muse Glimmer-specific behavior:

FlagWhy
--served-model-name muse-glimmerThe name the API answers to. Every example here sends "model": "muse-glimmer".
--enable-auto-tool-choiceLets the model decide when to call a tool.
--tool-call-parser muse_glimmerConverts native ATEM output into standard OpenAI tool_calls.
--reasoning-parser muse_glimmerRoutes the thinking channel to message.reasoning instead of leaking it into content.
--generation-config autoPicks up the checkpoint's published sampling settings and stop tokens.
--tensor-parallel-size NNumber of GPUs for tensor parallelism.
--max-model-len NCaps the sequence length to limit KV-cache memory.
Stream reasoning workloads

Muse Glimmer is a reasoning model that produces long chain-of-thought. Request streaming ("stream": true) for reasoning workloads so long generations don't hit request timeouts, and give max_tokens enough headroom for the reasoning trace plus the final answer.

Sampling

Serve with --generation-config auto, as every command above does, and vLLM picks up the checkpoint's published sampling settings: temperature 1.0, top_p 0.95, and top_k 64.

Send a request

The server exposes an OpenAI-compatible chat completions endpoint:

curl
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "muse-glimmer",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Write a Python function to compute Fibonacci numbers."}
    ],
    "max_tokens": 2048,
    "stream": true
  }'
Python
from openai import OpenAI
​
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
​
stream = client.chat.completions.create(
    model="muse-glimmer",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a Python function to compute Fibonacci numbers."},
    ],
    max_tokens=2048,
    stream=True,
)
​
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Neither request sets sampling parameters, so the server applies the published settings from --generation-config auto.

Verify tool calling

curl
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "muse-glimmer",
    "messages": [
      {"role": "user", "content": "What is the weather in Paris in celsius? Use the tool."}
    ],
    "tools": [{"type": "function", "function": {
      "name": "get_weather",
      "description": "Get current weather for a city.",
      "parameters": {"type": "object", "properties": {
        "city": {"type": "string"},
        "units": {"type": "string", "enum": ["celsius", "fahrenheit"]}},
        "required": ["city"]}}}],
    "tool_choice": "auto"
  }'

You get back get_weather(city="Paris", units="celsius"), with the reasoning under message.reasoning, or delta.reasoning when streaming. Parsing happens server-side, so an OpenAI-compatible harness needs no ATEM-specific handling.

One tool call per message

Several calls arrive as consecutive assistant messages rather than as several entries in one tool_calls array. A harness that reads only the first element of the first message will silently drop work.

Stop tokens

Muse Glimmer needs eos_token_id = [200001, 200008], which is <|end_of_text|> and <|eot|>. Serving with --generation-config auto picks these up from the checkpoint.

These token IDs come from the converted checkpoint. Confirm them against your checkpoint's tokenizer_config.json, since older exports can differ. See special tokens for the full chat-format token set.

Troubleshooting

SymptomCauseFix
ValueError: model architectures ... are not supportedRunning a released vLLM wheel rather than the imageThe model code isn't in any wheel. Use vllm/vllm-openai:muse-glimmer.
invalid tool call parserHyphenated parser nameUse underscores: muse_glimmer.
Tool calls returned as plain textParser not enabledPass --tool-call-parser muse_glimmer. The chat template comes from the checkpoint.
Only the first tool call runsHarness expects one tool_calls arrayOne call per message. Read consecutive assistant messages.
Generation never stopsWrong stop tokensSee Stop tokens. Never stop on the end-of-message token.
Reasoning leaking into contentNo reasoning parserPass --reasoning-parser muse_glimmer, which routes reasoning to message.reasoning.
Flat, repetitive answersRunning greedyDrop "temperature": 0 and use the published settings.
Out of memory shortly after startupSized to the 60 GB weights rather than the 72 GB serving footprintLower --max-model-len, raise --tensor-parallel-size, or serve a quantized checkpoint.

Next steps

Add speculative decoding to reduce per-token latency. For CPU or mixed inference, see llama.cpp; for other runtimes, see Run inference.