Deploy with llama.cpp

Run Muse Glimmer on your machine with llama.cpp(opens in new tab), a C/C++ inference engine that supports CPU, mixed CPU/GPU, and full GPU execution. It's portable across CPU, Metal, CUDA, ROCm, and Vulkan, and it serves the published GGUF checkpoints with no conversion or quantization step.

Download the checkpoints

GGUF builds live in meta-models/Muse-Glimmer-30B-GGUF(opens in new tab). Fetch only the two files you need: the text model and, for image input, the vision projector.

bash
pip install -U huggingface_hub
hf download meta-models/Muse-Glimmer-30B-GGUF --local-dir ./muse-glimmer \
  --include "Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf" \
  --include "mmproj-Muse-Glimmer-30B-Q4_K_M.gguf"

Full-precision weights aren't published as GGUF. For bf16, use the safetensors checkpoint at meta-models/Muse-Glimmer-30B(opens in new tab) with vLLM.

Get llama.cpp

Muse Glimmer support first shipped in release b10353(opens in new tab). Use b10353 or newer. A prebuilt binary and a source build both work.

llama_model_load: error loading model: unknown model architecture: 'muse-glimmer'

The releases page(opens in new tab) publishes prebuilt binaries for macOS arm64 (Metal), Linux (CPU, CUDA, ROCm, Vulkan, SYCL), and Windows. If you take one, unpack it and skip to Start the server, reading ./build/bin/ in the commands below as the directory you unpacked.

To build from source instead, master is well past the floor:

bash
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

Add --branch b10353 to pin to the floor, or any later tag to pin to a known build.

Pick your backend:

bash
# Apple silicon (Metal)
cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF
​
# NVIDIA (CUDA >= 12.4; set the arch for your card, 90 = Hopper, 120 = Blackwell)
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF \
  -DCMAKE_CUDA_ARCHITECTURES=90 \
  -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF
​
# CPU only
cmake -B build -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF

bash
cmake --build build -j"$(sysctl -n hw.ncpu)" --target llama-server llama-cli llama-mtmd-cli   # macOS
cmake --build build -j"$(nproc)"             --target llama-server llama-cli llama-mtmd-cli   # Linux

If ccache errors during the build, add -DGGML_CCACHE=OFF.

Confirm the checkout registers the architecture before you go looking for other reasons a load failed:

bash
grep -c LLM_ARCH_MUSE_GLIMMER src/llama-arch.cpp   # expect >= 1

A 0 means the checkout predates Muse Glimmer support and will refuse these files.

Run the commands below from the llama.cpp clone (or the unpacked release directory), with the muse-glimmer/ download directory inside it, so both ./build/bin/ and ./muse-glimmer/ resolve. Otherwise pass absolute paths to -m and --mmproj.

Start the server

Muse Glimmer supports a native context of 131072:

bash
./build/bin/llama-server \
  -m ./muse-glimmer/Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
  --mmproj ./muse-glimmer/mmproj-Muse-Glimmer-30B-Q4_K_M.gguf \
  -a muse-glimmer \
  -ngl 99 -c 131072 -np 1 \
  --host 127.0.0.1 --port 8080 --api-key <your-key> \
  --jinja \
  --chat-template-kwargs '{"reasoning_strength":"low"}'

The server prints the context you actually got:

srv load_model: initializing, n_slots = 1, n_ctx_slot = 131072, kv_unified = 'false'

If it instead reports exceeds the training context ... - capping, the GGUF's context_length metadata is stale and the server clamps the slot to it. No serve flag overrides this. Fix the metadata with the script in the llama.cpp clone, where muse-glimmer is the architecture name llama.cpp registers and so is the key prefix:

bash
python gguf-py/gguf/scripts/gguf_set_metadata.py <model>.gguf muse-glimmer.context_length 131072

Thinking lands in message.reasoning_content and message.content stays clean. That's the server default, with no flag needed. Pass --reasoning-format none to keep thinking inline in message.content instead.

Drop --mmproj for text-only. For speculative decoding add -md <draft>.gguf --spec-type draft-dflash -ngld 99 --spec-draft-n-max 4. A [spec] failed to measure draft model memory warning at startup is harmless, and the draft loads and serves normally after it.

Smoke test, where --noproxy bypasses a proxy that would intercept loopback:

curl
curl -s --noproxy '*' http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" -H "Authorization: Bearer <your-key>" \
  -d '{"model":"muse-glimmer","messages":[{"role":"user","content":"What is 17 * 23?"}],"temperature":0}'

Context per slot

n_ctx_slot in the startup log, not -c, is what bounds a single generation. Getting this wrong fails quietly: a generation that runs out of room produces no answer and no error, so a batch job or an eval reports a worse number and gives you nothing to debug.

Muse Glimmer reasons at length, which makes the margin thinner than it looks. A single generation can approach a 32768-token slot on its own. To keep concurrency without shrinking the slot, scale -c with -np:

bash
  -c 524288 -np 4        # 131072 per slot, still 4-way concurrent

KV cache stays affordable at that size: GQA with 2 KV heads, and sliding-window attention on three of every four layers.

Controlling reasoning length

The chat template reads reasoning_strength and defaults to high:

jinja
{%- set rs = reasoning_strength if reasoning_strength is defined and reasoning_strength else 'high' -%}

Set it server-wide with --chat-template-kwargs, or per request:

JSON
{"model":"muse-glimmer","messages":[],"chat_template_kwargs":{"reasoning_strength":"low"}}

The model card documents four levels: low, medium, high, and xhigh. Lower isn't automatically worse: where the environment validates the answer or the caller can retry cheaply, extra reasoning buys little. Reasoning tokens count against max_tokens, and a request that hits the ceiling mid-thought returns empty content with finish_reason: "length".

Reasoning can't be turned off here. The template opens the thinking channel unconditionally, so --reasoning off and "reasoning_effort": "none" both leave the output unchanged. reasoning_strength: low is how you spend fewer thinking tokens, and --reasoning-budget N is how you hard-cap them.

Vision

Send a remote URL, a data:image/...;base64,... URI, or a local path:

curl
curl -s --noproxy '*' http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" -H "Authorization: Bearer <your-key>" \
  -d '{"model":"muse-glimmer","messages":[{"role":"user","content":[
        {"type":"text","text":"What is in this image?"},
        {"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}
      ]}],"temperature":0}'

Images are billed as prompt tokens, scaling with resolution.

Command line, without the server

The build produces two CLI binaries alongside llama-server. Text goes through llama-cli:

bash
./build/bin/llama-cli \
  -m ./muse-glimmer/Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
  -ngl 99 -c 32768 --jinja -st

-st (--single-turn) answers once and exits. Without it, llama-cli stays interactive and waits on stdin, which reads as a hang.

For images, use llama-mtmd-cli:

bash
./build/bin/llama-mtmd-cli \
  -m       ./muse-glimmer/Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
  --mmproj ./muse-glimmer/mmproj-Muse-Glimmer-30B-Q4_K_M.gguf \
  -ngl 99 -c 32768 --jinja \
  --image photo.png -p "Describe this image."

--jinja is required here too. Without it, llama-mtmd-cli aborts with this custom template is not supported, try using --jinja.

Both CLIs print the thinking trace inline with the answer, and neither separates the two. --reasoning-format only applies to the server's JSON response, so use llama-server when you want content and reasoning_content apart.

Verify tool calling

curl
curl -s --noproxy '*' http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" -H "Authorization: Bearer <your-key>" \
  -d '{"model":"muse-glimmer","messages":[{"role":"user","content":"What is the weather in Paris in celsius? Use the tool."}],
       "tools":[{"type":"function","function":{
         "name":"get_weather","description":"Get current weather for a city.",
         "parameters":{"type":"object","properties":{
           "city":{"type":"string"},"units":{"type":"string","enum":["celsius","fahrenheit"]}},
           "required":["city"]}}}],
       "tool_choice":"auto","temperature":0}'

Returns finish_reason: "tool_calls" and a tool_calls array with get_weather(city="Paris", units="celsius"), with reasoning under reasoning_content. Parsing is server-side, so any OpenAI-compatible harness needs no ATEM-specific handling.

Stop tokens

Muse Glimmer needs eos_token_id = [<|end_of_text|>, <|eot|>]. Never stop on <|eom|>. The template handles this, and --jinja is what wires it up. See special tokens.

Next steps

For faster generation, add speculative decoding with a smaller draft model. To quantize the model yourself, see quantization; for other runtimes, see Run inference.