Run inference

Run Muse Glimmer on your own infrastructure or call it through a hosted inference provider. The self-hosted runtimes load the released checkpoints, or a quantized version of them, and use the Muse Glimmer chat format described in the prompting guide.

To call the model without provisioning hardware, use a hosted provider. For partner-verified deployments on specific hardware, see Hardware platforms.

Muse Glimmer is a dense 30B model. The bf16 weights are around 60 GB and need about 72 GB of VRAM to serve; the published quantized builds are 17 to 20 GB. Get the model lists every artifact.

Self-host

Each tab is a quick start for one runtime. Each runtime's section below covers installation, checkpoint variants, starting a server, and sending requests.

bash
docker run --rm --gpus all --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:muse-glimmer \
  meta-models/Muse-Glimmer-30B \
  --served-model-name muse-glimmer \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer \
  --generation-config auto

bash
sglang serve \
  --model-path meta-models/Muse-Glimmer-30B \
  --served-model-name muse-glimmer \
  --reasoning-parser muse \
  --tool-call-parser muse \
  --mem-fraction-static 0.85 \
  --host 0.0.0.0 --port 30000

bash
llama serve -hf meta-models/Muse-Glimmer-30B-GGUF \
  --hf-file Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
  -a muse-glimmer \
  -ngl 99 -c 131072 -np 1 \
  --host 127.0.0.1 --port 8080 --api-key <your-key> \
  --chat-template-kwargs '{"reasoning_strength":"low"}'

bash
python -m executorch.examples.models.muse_glimmer.serving.serve \
  --model-path exports/solo/model.pte \
  --data-path exports/solo/aoti_cuda_blob.ptd \
  --tokenizer-path assets/hf/tokenizer.json \
  --hf-tokenizer assets/hf \
  --worker-bin cmake-out/examples/models/muse-glimmer/muse_glimmer_worker \
  --model-id muse-glimmer-30B \
  --tool-parser atem \
  --host 127.0.0.1 --port 8000

bash
pip install "transformers>=5.15" accelerate torch

bash
ollama run muse-glimmer

bash
lms server start

bash
unsloth studio -p 8888

The vLLM and SGLang tabs serve the bf16 checkpoint, and the llama.cpp tab serves the 17 GB GGUF with the vision projector. Each runtime's section below has the other variants.

vLLM

Serve Muse Glimmer with vLLM(opens in new tab) for high-throughput, low-latency inference with an OpenAI-compatible API endpoint. vLLM provides continuous batching, PagedAttention, and tensor parallelism.

Prerequisites

• Docker with the NVIDIA container runtime
• An NVIDIA GPU with drivers for CUDA 13.0, which is what the image is pinned to
• 72 GB of VRAM to serve, on one card or across several

Install

Docker is the supported path:

bash
docker pull vllm/vllm-openai:muse-glimmer

The image is roughly 10.5 GB and publishes linux/amd64 and linux/arm64. There's no published ROCm image.

What the image gives you:

Parser names use underscores. vLLM matches muse_glimmer literally, so the hyphenated spelling fails. --served-model-name muse-glimmer is a label you choose, and it stays hyphenated.

Start the server

The image sets ENTRYPOINT ["vllm", "serve"], so everything after the image name is appended to that. Pass the model and flags directly rather than retyping vllm serve or calling python -m vllm.entrypoints.openai.api_server.

bash
docker run --rm --gpus all --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:muse-glimmer \
  meta-models/Muse-Glimmer-30B \
  --served-model-name muse-glimmer \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer \
  --generation-config auto

The flags from --served-model-name down are the recipe's argument list for a single card. The docker run wrapper around them is standard vLLM boilerplate, so adjust it to your host. Mounting ~/.cache/huggingface is what stops the container re-downloading roughly 60 GB of weights on every start.

To serve weights you've already downloaded, mount them and pass the path instead of a Hub id:

bash
docker run --rm --gpus all --ipc=host \
  -p 8000:8000 \
  -v /path/to/Muse-Glimmer-30B:/model \
  vllm/vllm-openai:muse-glimmer \
  /model \
  --served-model-name muse-glimmer \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer \
  --generation-config auto

To serve across multiple GPUs, raise the tensor-parallel size:

bash
docker run --rm --gpus all --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:muse-glimmer \
  meta-models/Muse-Glimmer-30B \
  --served-model-name muse-glimmer \
  --tensor-parallel-size 2 \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer \
  --generation-config auto

For tighter memory, at a cost in throughput and context:

bash
docker run --rm --gpus all --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:muse-glimmer \
  meta-models/Muse-Glimmer-30B \
  --served-model-name muse-glimmer \
  --gpu-memory-utilization 0.9 --max-model-len 8300 --enforce-eager \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer \
  --generation-config auto

The flags that carry Muse Glimmer-specific behavior:

Sampling

Serve with --generation-config auto, as every command above does, and vLLM picks up the checkpoint's published sampling settings: temperature 1.0, top_p 0.95, and top_k 64.

Send a request

The server exposes an OpenAI-compatible chat completions endpoint:

curl
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "muse-glimmer",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Write a Python function to compute Fibonacci numbers."}
    ],
    "max_tokens": 2048,
    "stream": true
  }'

Python
from openai import OpenAI
​
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
​
stream = client.chat.completions.create(
    model="muse-glimmer",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a Python function to compute Fibonacci numbers."},
    ],
    max_tokens=2048,
    stream=True,
)
​
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Neither request sets sampling parameters, so the server applies the published settings from --generation-config auto.

Verify tool calling

curl
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "muse-glimmer",
    "messages": [
      {"role": "user", "content": "What is the weather in Paris in celsius? Use the tool."}
    ],
    "tools": [{"type": "function", "function": {
      "name": "get_weather",
      "description": "Get current weather for a city.",
      "parameters": {"type": "object", "properties": {
        "city": {"type": "string"},
        "units": {"type": "string", "enum": ["celsius", "fahrenheit"]}},
        "required": ["city"]}}}],
    "tool_choice": "auto"
  }'

You get back get_weather(city="Paris", units="celsius"), with the reasoning under message.reasoning, or delta.reasoning when streaming. Parsing happens server-side, so an OpenAI-compatible harness needs no ATEM-specific handling.

Stop tokens

Muse Glimmer needs eos_token_id = [200001, 200008], which is <|end_of_text|> and <|eot|>. Serving with --generation-config auto picks these up from the checkpoint.

These token IDs come from the converted checkpoint. Confirm them against your checkpoint's tokenizer_config.json, since older exports can differ. See special tokens for the full chat-format token set.

Troubleshooting

SGLang

Serve Muse Glimmer with SGLang(opens in new tab) for high-throughput inference behind an OpenAI-compatible endpoint.

Install

Install SGLang from source:

bash
pip install --upgrade pip
pip install uv
git clone https://github.com/sgl-project/sglang.git
cd sglang
uv pip install -e python

A prebuilt image is the alternative:

bash
docker pull lmsysorg/sglang:dev-muse-glimmer

The tag is multi-arch and resolves to amd64 or arm64, so it also covers aarch64 machines such as DGX Spark. dev-muse-glimmer-amd64 and dev-muse-glimmer-arm64 pin one architecture. See the upstream install guide(opens in new tab) for platform variations.

Choose a checkpoint

SGLang serves four checkpoint formats. Your GPU decides which one you use:

The bf16 checkpoint is the only one that takes images. NVFP4 serves with no conversion step.

SGLang's team verified these platforms, each on a single accelerator:

SGLang publishes throughput and interactivity figures for each platform in the official guide(opens in new tab). Measure your own with:

bash
python3 -m sglang.bench_serving \
  --backend sglang --host localhost --port 30000 \
  --model meta-models/Muse-Glimmer-30B \
  --dataset-name random --random-input-len 1024 --random-output-len 512 \
  --num-prompts 128 --max-concurrency 64 --flush-cache

Start the server

bash
sglang serve \
  --model-path meta-models/Muse-Glimmer-30B \
  --served-model-name muse-glimmer \
  --reasoning-parser muse \
  --tool-call-parser muse \
  --mem-fraction-static 0.85 \
  --host 0.0.0.0 --port 30000

bash
sglang serve \
  --model-path meta-models/Muse-Glimmer-30B-GGUF/Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
  --served-model-name muse-glimmer \
  --reasoning-parser muse \
  --tool-call-parser muse \
  --mem-fraction-static 0.85 \
  --host 0.0.0.0 --port 30000

bash
sglang serve \
  --model-path RadixArk/Muse-Glimmer-NVFP4 \
  --served-model-name muse-glimmer \
  --reasoning-parser muse \
  --tool-call-parser muse \
  --language-model-only \
  --kv-cache-dtype fp8_e4m3 \
  --mem-fraction-static 0.85 \
  --host 0.0.0.0 --port 30000

bash
SGLANG_USE_MLX=1 SGLANG_MLX_CACHE_LIMIT_GB=8 \
sglang serve \
  --model-path RadixArk/Muse-Glimmer-q4km-gs128-MLX \
  --served-model-name muse-glimmer \
  --trust-remote-code \
  --reasoning-parser muse \
  --tool-call-parser muse \
  --disable-radix-cache \
  --mem-fraction-static 0.85 \
  --host 0.0.0.0 --port 30000

python -m sglang.launch_server with the same flags is the equivalent long form.

GGUF. --model-path points at the file, not the repository. It's the same 17 GB file that llama.cpp serves. The path is text only, because SGLang has no mmproj support, and SGLang warns at startup that the path isn't optimized. Don't add --language-model-only: the GGUF config is already text-only, and the serve is rejected.

NVFP4. --language-model-only is required. The NVFP4 checkpoint ships no vision weights, and without the flag the loader tries to build them and the engine fails to start. On 32 GB cards, --kv-cache-dtype fp8_e4m3 roughly doubles the KV pool: SGLang measured 57K to 115K tokens on an RTX 5090. SGLang doesn't list NVFP4 for H200.

MLX. Apple silicon uses MLX repacks, because the MLX backend has no GGUF path. The two extra settings are requirements: the sliding-window layers need the radix cache off, and SGLANG_MLX_CACHE_LIMIT_GB=8 stops the MLX buffer cache from growing the memory footprint under concurrent load. The three MLX artifacts are drop-in --model-path swaps. q4km-gs128 carries the same Q4_K_M codes as the GGUF and is SGLang's default, q4 is the fastest, and q4k-dynamic has the highest fidelity. The MLX backend is text only and has no speculative decoding.

Pass both parser flags explicitly. SGLang's parser name for both is muse.

Memory and parallelism

SGLang sets --mem-fraction-static per machine:

Lower it if the server runs out of memory during startup, and raise it for more KV cache. DFlash speculative decoding needs room for the draft model, so SGLang lowers the fraction further on DGX Spark: 0.65 for bf16 and 0.38 for NVFP4.

Every combination SGLang verified runs on a single accelerator with no tensor parallelism. Muse Glimmer fits on one card in each of these formats, so treat multi-GPU serving as untested.

Speculative decoding

The DFlash draft model meta-models/Muse-Glimmer-30B-assistant serves as published. Add:

bash
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path meta-models/Muse-Glimmer-30B-assistant \
  --speculative-dflash-block-size 5

Two targets need one more flag each:

• GGUF target: add --speculative-draft-load-format auto. Without it, the draft inherits the target's gguf load format and the loader rejects the draft directory.
• NVFP4 target: add --speculative-draft-model-quantization fp8.

SGLang also verified the GGUF draft, dflash-Muse-Glimmer-30B-Q4_K_M.gguf, on B300. Pass --speculative-draft-model-quantization gguf with it. Otherwise the draft is built unquantized from random weights: nothing errors, speculation runs, and the accept length stays at 1.0 with no speedup.

See Speculative decoding for how DFlash works.

Verify the server

SGLang exposes an OpenAI-compatible endpoint on port 30000:

curl
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "muse-glimmer",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "max_tokens": 512
  }'

Reasoning comes back under message.reasoning_content. Check tool calling with:

curl
curl -s http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"muse-glimmer","messages":[{"role":"user","content":"What is the weather in Paris in celsius? Use the tool."}],
       "tools":[{"type":"function","function":{
         "name":"get_weather","description":"Get current weather for a city.",
         "parameters":{"type":"object","properties":{
           "city":{"type":"string"},"units":{"type":"string","enum":["celsius","fahrenheit"]}},
           "required":["city"]}}}],
       "tool_choice":"auto","temperature":1}'

A working parser returns a tool_calls array with get_weather(city="Paris", units="celsius") and the thinking under reasoning_content. Parsing happens server-side, so an OpenAI-compatible harness needs no ATEM-specific handling.

Stop tokens

Muse Glimmer needs eos_token_id = [<|end_of_text|>, <|eot|>]. Never stop on <|eom|>, which ends a message while the turn continues. See special tokens.

There's no serve flag for this. SGLang builds its stop set at load time by unioning eos_token_id from the checkpoint's config.json and its generation_config.json, so the checkpoint metadata is where you set it. Because it's a union, a request can add IDs through stop_token_ids but can't remove them.

Troubleshooting

llama.cpp

Run Muse Glimmer on your machine with llama.cpp(opens in new tab), a C/C++ inference engine that supports CPU, mixed CPU/GPU, and full GPU execution. It's portable across CPU, Metal, CUDA, ROCm, and Vulkan, and it serves the published GGUF checkpoints with no conversion or quantization step.

Get llama.cpp

Install the Llama app(opens in new tab) from the llama.cpp team on macOS or Windows. On Linux, install the command-line version:

bash
curl -LsSf https://llama.app/install.sh | sh

Muse Glimmer needs llama.cpp b10353 or newer. Older builds refuse to load the checkpoints with unknown model architecture: 'muse-glimmer'. To build from source instead, follow upstream's build guide(opens in new tab).

Choose a checkpoint

llama serve -hf downloads and caches the checkpoint from meta-models/Muse-Glimmer-30B-GGUF(opens in new tab). Name the text model with --hf-file:

Full-precision weights aren't published as GGUF. For bf16, use the safetensors checkpoint at meta-models/Muse-Glimmer-30B(opens in new tab) with vLLM.

Start the server

Muse Glimmer supports a native context of 131072:

bash
llama serve -hf meta-models/Muse-Glimmer-30B-GGUF \
  --hf-file Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
  -a muse-glimmer \
  -ngl 99 -c 131072 -np 1 \
  --host 127.0.0.1 --port 8080 --api-key <your-key> \
  --chat-template-kwargs '{"reasoning_strength":"low"}'

The server prints the context size it allocated:

srv load_model: initializing, n_slots = 1, n_ctx_slot = 131072, kv_unified = 'false'

If it instead reports exceeds the training context ... - capping, the GGUF's context_length metadata is stale and the server clamps the slot to it. No serve flag overrides this. Fix the metadata with the script in the llama.cpp repository, where muse-glimmer is the architecture name llama.cpp registers and so is the key prefix:

bash
python gguf-py/gguf/scripts/gguf_set_metadata.py <model>.gguf muse-glimmer.context_length 131072

Thinking lands in message.reasoning_content and message.content stays clean. That's the server default, with no flag needed. Pass --reasoning-format none to keep thinking inline in message.content instead.

The vision projector downloads with the text model, so image input needs no extra flag. Pass --no-mmproj for text-only. For speculative decoding, download dflash-Muse-Glimmer-30B-Q4_K_M.gguf from the same repository and add -md dflash-Muse-Glimmer-30B-Q4_K_M.gguf --spec-type draft-dflash -ngld 99 --spec-draft-n-max 4. A [spec] failed to measure draft model memory warning at startup is harmless, and the draft loads and serves normally after it.

Smoke test, where --noproxy bypasses a proxy that would intercept loopback:

curl
curl -s --noproxy '*' http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" -H "Authorization: Bearer <your-key>" \
  -d '{"model":"muse-glimmer","messages":[{"role":"user","content":"What is 17 * 23?"}]}'

Context per slot

n_ctx_slot in the startup log, not -c, is what bounds a single generation. Getting this wrong fails quietly: a generation that runs out of room produces no answer and no error, so a batch job or an eval reports a worse number and gives you nothing to debug.

Muse Glimmer reasons at length: a single generation can approach a 32768-token slot on its own. To keep concurrency without shrinking the slot, scale -c with -np:

bash
  -c 524288 -np 4        # 131072 per slot, still 4-way concurrent

KV cache stays affordable at that size: GQA with 2 KV heads, and sliding-window attention on three of every four layers.

Control reasoning length

The chat template reads reasoning_strength and defaults to high:

jinja
{%- set rs = reasoning_strength if reasoning_strength is defined and reasoning_strength else 'high' -%}

Set it server-wide with --chat-template-kwargs, or per request:

JSON
{"model":"muse-glimmer","messages":[],"chat_template_kwargs":{"reasoning_strength":"low"}}

The model card documents four levels: low, medium, high, and xhigh. Lower isn't automatically worse: where the environment validates the answer or the caller can retry cheaply, extra reasoning adds little. Reasoning tokens count against max_tokens, and a request that hits the ceiling mid-thought returns empty content with finish_reason: "length".

Reasoning can't be turned off here. The template opens the thinking channel unconditionally, so --reasoning off and "reasoning_effort": "none" both leave the output unchanged. reasoning_strength: low is how you spend fewer thinking tokens, and --reasoning-budget N is how you hard-cap them.

Vision

Send a remote URL, a data:image/...;base64,... URI, or a local path:

curl
curl -s --noproxy '*' http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" -H "Authorization: Bearer <your-key>" \
  -d '{"model":"muse-glimmer","messages":[{"role":"user","content":[
        {"type":"text","text":"What is in this image?"},
        {"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}
      ]}]}'

Images count as prompt tokens, scaling with resolution.

Command line, without the server

llama cli runs the model in the terminal:

bash
llama cli -hf meta-models/Muse-Glimmer-30B-GGUF \
  --hf-file Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
  -ngl 99 -c 32768 -st

-st (--single-turn) answers once and exits. Without it, llama cli stays interactive and waits on stdin, which reads as a hang.

llama cli prints the thinking trace inline with the answer. --reasoning-format only applies to the server's JSON response, so use llama serve when you want content and reasoning_content apart.

Verify tool calling

curl
curl -s --noproxy '*' http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" -H "Authorization: Bearer <your-key>" \
  -d '{"model":"muse-glimmer","messages":[{"role":"user","content":"What is the weather in Paris in celsius? Use the tool."}],
       "tools":[{"type":"function","function":{
         "name":"get_weather","description":"Get current weather for a city.",
         "parameters":{"type":"object","properties":{
           "city":{"type":"string"},"units":{"type":"string","enum":["celsius","fahrenheit"]}},
           "required":["city"]}}}],
       "tool_choice":"auto"}'

Returns finish_reason: "tool_calls" and a tool_calls array with get_weather(city="Paris", units="celsius"), with reasoning under reasoning_content. Parsing is server-side, so any OpenAI-compatible harness needs no ATEM-specific handling.

Stop tokens

Muse Glimmer needs eos_token_id = [<|end_of_text|>, <|eot|>]. Never stop on <|eom|>. The template embedded in the GGUF handles this. See special tokens.

ExecuTorch

Serve Muse Glimmer on your own CUDA GPU or Apple silicon with ExecuTorch(opens in new tab), behind an OpenAI-compatible HTTP endpoint that supports vision, tool calling, and DFlash speculative decoding. ExecuTorch is ahead-of-time: you export the model once into a .pte program for a specific backend, then serve from that program.

With ExecuTorch, you write the model and its decoding strategy once in PyTorch, including Muse Glimmer's multimodal input and block-diffusion speculative decoding. torch.export then lowers the whole graph ahead of time: to Triton on CUDA, and to MLX-native or custom Metal kernels on Apple silicon.

Prebuilt exports are published as well, under different filenames, so exporting a 30B model yourself is optional.

Prerequisites

• An ExecuTorch checkout built from source per the upstream guide(opens in new tab), including examples/models/muse-glimmer
• The server dependencies: pip install -r examples/llm_server/python/requirements.txt
• The Hugging Face CLI (pip install huggingface_hub)
• A CUDA GPU for the CUDA backend, or Apple silicon for the MLX backend

Run every command below from the ExecuTorch repository root.

Download the model assets

Exports lower directly from the quantized GGUF checkpoints, the same files llama.cpp uses. The serving path also needs the tokenizer metadata from the main model repository:

bash
hf download meta-models/Muse-Glimmer-30B-GGUF --local-dir assets/quant \
  --include 'Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf' \
  --include 'dflash-Muse-Glimmer-30B-Q4_K_M.gguf' \
  --include 'mmproj-Muse-Glimmer-30B-Q4_K_M.gguf'
​
hf download meta-models/Muse-Glimmer-30B \
  tokenizer.json chat_template.jinja config.json processor_config.json \
  --local-dir assets/hf

Keep chat_template.jinja beside the rest of the tokenizer metadata: the serving path renders prompts and tool definitions with it.

Set the paths used by the commands that follow:

bash
TARGET=assets/quant/Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf
DRAFT=assets/quant/dflash-Muse-Glimmer-30B-Q4_K_M.gguf
MMPROJ=assets/quant/mmproj-Muse-Glimmer-30B-Q4_K_M.gguf
BACKEND=cuda   # mlx on macOS

Export the model

$BACKEND selects the target: cuda or mlx. A CUDA export autotunes Triton kernels against the GPU it runs on, so export on the same architecture you intend to serve from.

Target-only export

For text inference:

bash
python -m executorch.examples.models.muse_glimmer.export.export_solo \
  --gguf "$TARGET" \
  --backend "$BACKEND" \
  --output-dir exports/solo

DFlash export

DFlash speculative decoding lowers the target and the draft together, into their own export directory:

bash
python -m executorch.examples.models.muse_glimmer.export.export_dflash \
  --target-gguf "$TARGET" \
  --draft-gguf "$DRAFT" \
  --backend "$BACKEND" \
  --output-dir exports/dflash

Both land in one .pte: the draft shares the target's token embeddings and output head rather than carrying copies. The block dimension is exported dynamically, so you select block length at serve time. The exported range is backend-specific: [2, 16] on MLX and [2, 4] on CUDA, where the draft count is also capped at 3.

Vision exports

Add --mmproj "$MMPROJ" to either export command for text and image inference. A vision export also writes pos_embed.bin beside model.pte.

Download a prebuilt export

meta-models/Muse-Glimmer-30B-ExecuTorch-PTE(opens in new tab) publishes 16 ready-made exports, which skips the export step. You still build the runner: a .pte is a model program, not a runtime.

Directories are named muse-glimmer-<quantization>-128K-<modality>-<decoding>-<backend>, and all 16 combinations of the four axes exist:

• Quantization: k-quant-17G targets 24 GB of VRAM. k-quant-dynamic targets 32 GB with less degradation, and the model card(opens in new tab) quantifies the tradeoff.
• Modality: text, or text-image for vision.
• Decoding: solo, or dflash for speculative decoding.
• Backend: metal for Apple silicon, sm80+ptx for CUDA on SM80 and newer.

Context length is 128K for every variant. Sizes, by directory:

Each directory holds <directory-name>.pte. The sm80+ptx variants add <directory-name>.ptd, and the text-image variants add pos_embed.bin. On CUDA the weights live in the .ptd and the .pte is only tens of megabytes, so both files are required. The repository root carries the tokenizer metadata (tokenizer.json, tokenizer_config.json, and chat_template.jinja), so this one repository covers everything the server needs. The assets/hf download above is for the export path.

bash
EXPORT_DIR=muse-glimmer-k-quant-17G-128K-text-solo-sm80+ptx
​
hf download meta-models/Muse-Glimmer-30B-ExecuTorch-PTE \
  --include "$EXPORT_DIR/*" \
  --include tokenizer.json --include tokenizer_config.json --include chat_template.jinja \
  --local-dir exports

Build the runner

bash
(cd examples/models/muse-glimmer && cmake --workflow --preset muse-glimmer-cuda)

Use the muse-glimmer-mlx preset on Apple silicon. Binaries land in cmake-out/examples/models/muse-glimmer/; the server needs muse_glimmer_worker.

Serve the model

bash
python -m executorch.examples.models.muse_glimmer.serving.serve \
  --model-path exports/solo/model.pte \
  --data-path exports/solo/aoti_cuda_blob.ptd \
  --tokenizer-path assets/hf/tokenizer.json \
  --hf-tokenizer assets/hf \
  --worker-bin cmake-out/examples/models/muse-glimmer/muse_glimmer_worker \
  --model-id muse-glimmer-30B \
  --tool-parser atem \
  --host 127.0.0.1 --port 8000

Adjust the command for your export:

• MLX: drop --data-path.
• DFlash: replace exports/solo with exports/dflash in both artifact paths. The server detects the exported method contract.
• Vision: add --pos-embed-path <export-dir>/pos_embed.bin.

From a prebuilt export, substitute the artifact and tokenizer flags. The rest of the command is unchanged:

bash
  --model-path "exports/$EXPORT_DIR/$EXPORT_DIR.pte" \
  --data-path "exports/$EXPORT_DIR/$EXPORT_DIR.ptd" \
  --tokenizer-path exports/tokenizer.json \
  --hf-tokenizer exports \

Drop --data-path for a -metal variant, which has no .ptd. Add --pos-embed-path "exports/$EXPORT_DIR/pos_embed.bin" for a text-image variant. A -dflash variant needs no extra flag.

Send a request to the running server:

curl
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "muse-glimmer-30B",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "max_tokens": 32
  }'

The server implements /health, /v1/models, and /v1/chat/completions (streaming and non-streaming). --max-context bounds the context window, and prompts over that bound are rejected with a 400.

Verify tool calling

--tool-parser atem is what makes tool calling work end to end. The Hugging Face chat template renders your tool definitions, and the server converts Muse Glimmer's native ATEM output into an OpenAI-compatible tool_calls array, so your harness needs no Muse Glimmer-specific handling.

Two OpenAI parameters are rejected with a structured 400 rather than ignored:

• reasoning_effort: control reasoning with the chat template's reasoning_strength argument instead.
• tool_choice="required": none, auto, and unset are accepted.

Stop tokens

Muse Glimmer needs eos_token_id = [<|end_of_text|>, <|eot|>]. Never stop on <|eom|>, which ends a message while the turn continues. See special tokens.

Runtime limits

These bound the runtime. The model card(opens in new tab) covers the model itself.

• No video input: text and images only, one image per request.
• No continuous batching: one request runs at a time. --num-runners must be 1, the exported methods are batch-1, and execution is serialized. Concurrent sessions are isolated from each other rather than served in parallel.
• No cross-session prefix sharing and no checkpointing: every session holds its own KV cache, nothing is reused across sessions, and session state is discarded rather than saved.

For concurrent throughput from one machine, serve with vLLM.

Troubleshooting

Hugging Face Transformers

Run Muse Glimmer in plain Python with Hugging Face Transformers(opens in new tab). There's no server: use this path to learn the generation loop before you build an agent on top of it.

Install

bash
pip install "transformers>=5.15" accelerate torch

Use transformers 5.15 or later. Earlier versions lack the native Muse Glimmer path and prompt for trust_remote_code.

Run the model

Python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
​
model_id = "meta-models/Muse-Glimmer-30B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
​
messages = [{"role": "user", "content": "In one sentence, what is Muse Glimmer good at?"}]
enc = tok.apply_chat_template(messages, add_generation_prompt=True,
                              return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**enc, max_new_tokens=128, do_sample=True, temperature=1.0, top_p=0.95,
                     eos_token_id=[tok.convert_tokens_to_ids("<|eot|>"),
                                   tok.convert_tokens_to_ids("<|end_of_text|>")])
print(tok.decode(out[0][enc["input_ids"].shape[-1]:], skip_special_tokens=True))

The bf16 weights are around 60 GB, so device_map="auto" spreads them across the GPUs you have.

Stop tokens

Pass eos_token_id as <|eot|> and <|end_of_text|>, as the example does. Never add <|eom|>, which ends a message while the turn continues. See special tokens.

Ollama

Run Muse Glimmer locally with one command through Ollama(opens in new tab), then connect it to a coding agent or call it through Ollama's API.

Install

Install Ollama from ollama.com/download(opens in new tab). Muse Glimmer requires Ollama 0.32.7 or later:

bash
ollama --version

Run the model

bash
ollama run muse-glimmer

On Apple silicon, use Ollama's MLX engine instead:

bash
ollama run muse-glimmer:30b-mlx

The Ollama library page(opens in new tab) lists every published tag.

Connect a coding agent

ollama launch configures a coding agent or another long-running agent to run on Muse Glimmer:

bash
ollama launch claude --model muse-glimmer
ollama launch pi --model muse-glimmer
ollama launch opencode --model muse-glimmer
ollama launch hermes --model muse-glimmer
ollama launch openclaw --model muse-glimmer

Send a request

Ollama listens on http://localhost:11434. The native chat API needs no API key:

curl
curl http://localhost:11434/api/chat -d '{
  "model": "muse-glimmer",
  "stream": false,
  "messages": [
    {"role": "user", "content": "In one sentence, what is Muse Glimmer good at?"}
  ]
}'

OpenAI clients use the compatible chat completions endpoint. ollama is a placeholder key for clients that require one:

curl
curl http://localhost:11434/v1/chat/completions \
  -H 'Authorization: Bearer ollama' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "muse-glimmer",
    "messages": [
      {"role": "user", "content": "In one sentence, what is Muse Glimmer good at?"}
    ]
  }'

LM Studio

Run Muse Glimmer locally with LM Studio Bionic(opens in new tab), LM Studio's agent app for open models. Use it for agentic coding, research, and document tasks on your own hardware.

Install

Download LM Studio Bionic from lmstudio.ai(opens in new tab).

Use the agent

1. Under Settings > Explore, select Muse Glimmer and download the recommended model file for your hardware.
2. Create a new project. To let Bionic run commands in your local folders, turn on Allow Coding.
3. Select Muse Glimmer from the model picker.
4. Prompt Bionic to complete a task, such as editing attached documents or researching a topic.

Confirm the model works by running an agentic task. For coding, ask Muse Glimmer to inspect a file in a real repository, explain the relevant code path, and propose a diff. For other tasks, attach a document and ask Muse Glimmer to extract facts with file citations.

Start the server

LM Studio ships an OpenAI-compatible local API server. From a terminal:

bash
lms server start

The server starts on port 1234. Change it with --port.

To start it from the app instead, turn on the local API server under Settings > Local Model API, then copy the base URL into your scripts or OpenAI-compatible client.

Verify tool calling

In Bionic, run a task that needs a tool. Tool calls appear in the chat log as the agent works.

Through the API, send a request that offers one or more tools and asks the model to use one. Confirm that the response requests the expected tool with valid arguments.

Troubleshooting

More

• Bionic docs(opens in new tab)
• LM Studio's Muse Glimmer blog post(opens in new tab)

Unsloth

Run and serve Muse Glimmer locally with Unsloth(opens in new tab), an open-source desktop app. It serves OpenAI- and Anthropic-compatible endpoints, connects coding agents, and supports tool calling, web search, remote access, and model export.

Links

• Unsloth Muse Glimmer GGUFs(opens in new tab)
• Download Unsloth(opens in new tab)
• Unsloth docs(opens in new tab)

Install

Download Unsloth(opens in new tab) for macOS, Windows, Linux, or WSL, or install it from a terminal:

bash
curl -fsSL https://unsloth.ai/install.sh | sh

powershell
irm https://unsloth.ai/install.ps1 | iex

Launch it from the installed app shortcut, or run:

bash
unsloth studio -p 8888

See the platform requirements(opens in new tab).

Use Unsloth Desktop

1. Open Select model or Model Hub, search for Muse Glimmer, and download GGUF, NVFP4, or another format. Select the official Meta checkpoint or the Unsloth build. For GGUF, choose a quantization that fits your VRAM, unified memory, and system RAM while leaving room for KV cache.
2. Chat with the model once it downloads.
3. To serve the model, create an API key under Settings > API. Unsloth exposes OpenAI-compatible /v1/chat/completions and /v1/responses endpoints, plus an Anthropic-compatible /v1/messages.

Keep the localhost binding unless you need remote access. Keep API keys private, and review enabled tools before you expose the server.

For LAN access:

bash
unsloth studio -H 0.0.0.0 -p 8888

For HTTPS tunnel access:

bash
unsloth studio --secure -p 8888

Connect a coding agent

Load Muse Glimmer in Unsloth, then run:

bash
unsloth start claude

Replace claude with any of the supported agents(opens in new tab).

Unsloth publishes Muse Glimmer GGUFs at unsloth/Muse-Glimmer-30B-GGUF(opens in new tab): Dynamic quants from UD-IQ2_XXS through UD-Q8_K_XL, plus bf16 and the mmproj vision projectors. Pick one when you launch an agent, where the : suffix selects the quant:

bash
unsloth start claude \
  --model unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL \
  --context-length 131072

--gguf-variant UD-Q4_K_XL does the same and overrides the suffix. If Unsloth isn't already running, --model starts a temporary server that stops when the agent exits.

Verify tool calling

The endpoint takes OpenAI-style tools. Replace default with the id from /v1/models:

curl
curl http://localhost:8888/v1/chat/completions \
  -H "Authorization: Bearer $UNSLOTH_STUDIO_AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model":"default",
    "messages":[{"role":"user","content":"What is the weather in Paris in celsius? Use the tool."}],
    "tools":[{"type":"function","function":{
      "name":"get_weather","description":"Get current weather for a city.",
      "parameters":{"type":"object","properties":{
        "city":{"type":"string"},"units":{"type":"string","enum":["celsius","fahrenheit"]}},
        "required":["city"]}}}],
    "tool_choice":"auto"}'

A working setup returns a tool_calls array, not the call rendered as plain text.

Stop tokens

Muse Glimmer needs eos_token_id = [<|end_of_text|>, <|eot|>]. Never stop on <|eom|>. See special tokens.

Unsloth sets inference parameters from the model on load, and the chat template and other per-model settings are editable in the app. Check which stop tokens Unsloth applies before you run long agentic turns, because stopping on <|eom|> truncates tool calling.

Hosted providers

Call Muse Glimmer through a hosted provider when you'd rather not run the hardware yourself. These providers serve the open weights:

Together AI

Together AI serves Muse Glimmer as a managed API. There are no weights to download, no runtime to install, and no GPU to own. You send requests to https://api.together.xyz/v1/chat/completions with a model string and an API key.

Together AI requires an API key. To run without one, see Self-host.

The figures under "What Together publishes" come from Together's Muse Glimmer model page(opens in new tab).

Set the model string

Plaintext
meta-models/Muse-Glimmer-30B

Together lists this string as the endpoint on its model page and uses it in all three of its published examples.

What Together publishes

Tool calling is supported through the model's chat template, which makes this endpoint usable for the agent loops described in the tool-calling guide.

Get a key

bash
export TOGETHER_API_KEY=...   # https://api.together.xyz/settings/api-keys

Check that it works and that the model is visible to your account:

bash
curl -s https://api.together.xyz/v1/models \
  -H "Authorization: Bearer $TOGETHER_API_KEY" | grep -o 'meta-models/Muse-Glimmer-30B'

Send a request

Together publishes three clients. The curl form shows the wire format:

curl
curl -X POST "https://api.together.xyz/v1/chat/completions" \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-models/Muse-Glimmer-30B",
    "messages": [
      {"role": "user", "content": "What are some fun things to do in New York?"}
    ]
  }'

Python, using Together's official SDK (pip install together; it reads TOGETHER_API_KEY from the environment):

Python
from together import Together
​
client = Together()
response = client.chat.completions.create(
    model="meta-models/Muse-Glimmer-30B",
    messages=[{"role": "user", "content": "What are some fun things to do in New York?"}],
)
print(response.choices[0].message.content)

TypeScript, using together-ai:

TypeScript
import Together from 'together-ai';
​
const together = new Together();
const completion = await together.chat.completions.create({
  model: 'meta-models/Muse-Glimmer-30B',
  messages: [{ role: 'user', content: 'What are some fun things to do in New York?' }],
});
console.log(completion.choices[0].message.content);

The path, bearer header, and {model, messages} body use the OpenAI chat-completions wire format. An HTTP client pointed at https://api.together.xyz/v1 can call it.

Verify tool calling

A response with text only proves the endpoint is up. A tool-calling round trip verifies the model integration: expect tool_calls in the response, not prose.

curl
curl -s https://api.together.xyz/v1/chat/completions \
  -H "Authorization: Bearer $TOGETHER_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "model": "meta-models/Muse-Glimmer-30B",
    "messages": [{"role": "user", "content": "What's the weather in Paris?"}],
    "tools": [{"type": "function", "function": {"name": "get_weather",
      "description": "Get current weather for a city",
      "parameters": {"type": "object", "properties": {"city": {"type": "string"}},
                     "required": ["city"]}}}]
  }'

A working call returns tool_calls, with reasoning available separately on the response object.

You can also drive the same prompt from Together's Muse Glimmer playground(opens in new tab).

Set sampling and stop tokens

Sampling. Defaults come from the checkpoint: temperature 1.0, top_p 0.95. Greedy decoding loops on this model, so don't set temperature to 0.

Weights. Nothing here downloads them. For the model card or to run the same model locally, use meta-models/Muse-Glimmer-30B(opens in new tab) on Hugging Face.

What Together measured

These metrics are not on the model page. They describe Together's serving stack rather than anything you configure.

Together measured a dedicated 2xH100-80GB tensor-parallel replica under a sustained 0.4 QPS long-context replay: mean input 58,290 tokens (p99 approximately 127K), mean output 244 tokens, and 1–2 requests in flight. Time to first token (TTFT) is substantially lower at 1K-token inputs.

Together also reports serving the model quantized to FP8: MLP-only blockwise e4m3, 128x128, with dynamic activations. Attention, embeddings, and the vision tower stay bf16. Together chose this configuration after checking quality against a bf16 reference on the same hardware: GPQA-Diamond 83.7 +/- 0.9 over six runs, against 83.5 on the model card. Precision is not something you select on this endpoint. The detail helps when you compare a hosted result with a local bf16 result.

Troubleshooting

Support

• Model page(opens in new tab)
• Docs(opens in new tab)
• Pricing(opens in new tab)
• Support(opens in new tab)

Next steps

Once your server is up, frame your requests with the prompting guide, shrink the model with quantization, or speed up self-hosted generation with speculative decoding. To fine-tune the model, see Post-training.