Deploy with SGLang

Serve Muse Glimmer with SGLang(opens in new tab) for high-throughput inference behind an OpenAI-compatible endpoint.

Install

Muse Glimmer support lives on the muse-glimmer branch (PR #34262(opens in new tab)) rather than in a released version, so install from source:

bash
pip install --upgrade pip
pip install uv
git clone -b muse-glimmer https://github.com/sgl-project/sglang.git
cd sglang
uv pip install -e "python[all]"

A prebuilt image is the alternative:

bash
docker pull lmsysorg/sglang:dev-muse-glimmer

Start the server

Serve the BF16 checkpoint:

bash
sglang serve \
  --model-path meta-models/Muse-Glimmer-30B \
  --served-model-name muse-glimmer \
  --reasoning-parser muse \
  --tool-call-parser muse \
  --mem-fraction-static 0.85 \
  --host 0.0.0.0 --port 30000

--reasoning-parser muse splits the thinking channel into message.reasoning_content, and --tool-call-parser muse emits structured message.tool_calls. SGLang's parser name for both flags is muse. Pass both flags explicitly.

--served-model-name muse-glimmer sets the name the API answers to. Without it, SGLang serves under the full --model-path value and the requests below won't match.

Lower --mem-fraction-static if the server runs out of memory during startup.

For the GGUF, NVFP4, and Apple silicon MLX checkpoints, and for the hardware SGLang has verified each one on, see the official guide(opens in new tab).

Verify the server

SGLang exposes an OpenAI-compatible endpoint on port 30000:

curl
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "muse-glimmer",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "max_tokens": 512
  }'

Reasoning comes back under message.reasoning_content, and tool calls arrive as a standard tool_calls array. Parsing happens server-side, so an OpenAI-compatible harness needs no ATEM-specific handling.

Stop tokens

Muse Glimmer needs eos_token_id = [<|end_of_text|>, <|eot|>]. Never stop on <|eom|>, which ends a message while the turn continues. See special tokens.

There's no serve flag for this. SGLang builds its stop set at load time by unioning eos_token_id from the checkpoint's config.json and its generation_config.json, so the checkpoint metadata is where you set it. Because it's a union, a request can add IDs through stop_token_ids but can't remove them.

Next steps