Post-training

Adapt Muse Glimmer to your task with an open-source training framework on your own GPUs. There are two training objectives. Use supervised fine-tuning (SFT) when you have examples of the response you want, and reinforcement learning (RL) when you can check success automatically but can't fully demonstrate it.

Option	Objective	Methods
Axolotl	Supervised fine-tuning	Quantized low-rank adaptation (QLoRA), bf16 low-rank adaptation (LoRA) with Fully Sharded Data Parallel 2 (FSDP2), full-parameter bf16 with FSDP2
TRL	Supervised fine-tuning	LoRA, QLoRA
TitanRL	Reinforcement learning	Group Relative Policy Optimization (GRPO) with a verifiable reward
TRL	Reinforcement learning	Direct Preference Optimization (DPO), GRPO

Choose the training signal

Start with SFT when you have target responses. It's the direct route for teaching a new output format, tone, domain pattern, or visual task. Most projects start, and often stop, at SFT.

Use RL when the model can already attempt the task and success is cheap to check, but the behavior is unreliable. RL is a good fit when:

The model can do the task but does it unreliably: the wrong format, giving up after an error, or never producing the required output.
Success is cheap and objective to check: a file exists, a number matches, a test passes. If a human or an LLM has to judge each attempt, expect noise and reward hacking.
You have a measured baseline, so "better" is a number.

RL shapes behavior. It doesn't teach the model facts it lacks. Start with SFT for that, then add RL for the remaining failures that have an objective reward. RL is also more stable from an SFT checkpoint that already produces well-formatted, on-task outputs.

Stage	When to use	Data you need
Supervised fine-tuning	Domain adaptation, style, format, instruction following	Prompt and response pairs in chat format
Reinforcement learning	Preference alignment, tool-use success, agentic behavior	Preference pairs, a reward model, or a verifiable reward

Size the hardware

Post-training a 30B dense model is much heavier than quantized inference. The SFT methods trade memory for how much of the model can change:

SFT method	What stays in memory
QLoRA	4-bit frozen base plus trainable adapters
LoRA	bf16 frozen base plus trainable adapters
Full-parameter with FSDP2	Sharded weights, gradients, and optimizer state

QLoRA is the lowest-memory first experiment. Try LoRA when you want an unquantized frozen base, and use full-parameter training only when adapters are a measured capacity bottleneck.

For full-parameter bf16 training, SFT or RL, with the model sharded N ways, the parameter state per GPU starts at:

	Per GPU at 30B
bf16 weights	About 60 GB / N
bf16 gradients	About 60 GB / N
fp32 Adam state (m and v)	About 240 GB / N

These figures exclude activations, temporary buffers, and the generator that RL keeps loaded.

Supervised fine-tuning

SFT teaches Muse Glimmer from demonstrations: a prompt and the response the model should learn to produce.

Axolotl

Fine-tune Muse Glimmer on text, single-image, and multi-image demonstrations with Axolotl(opens in new tab). The recipe starts with QLoRA, compares it with bf16 LoRA, and uses full-parameter FSDP2 only when adapters don't provide enough capacity. It ships the configs, data preparation script, tests, and measured results.

Precision	bf16 compute; QLoRA loads the frozen base in 4-bit
Trainer	Axolotl 0.19.0(opens in new tab)
Model	`meta-models/Muse-Glimmer-30B` at revision `a4e59da52a7bc87ae7251dd5545c0dd437c44b68`
Hardware verified on	8 × H100 for all methods; QLoRA also on 1 × H100
Max VRAM observed	97,340 MiB per GPU, during the full FSDP2 final save
Experiment tracking	Weights & Biases (W&B)

Install

Clone the cookbook and run these commands from post-training/sft/axolotl/, in a dedicated environment so the CUDA and compiled-extension pins don't leak into another project:

bash
uv venv --python 3.12 .venv
source .venv/bin/activate
export UV_TORCH_BACKEND=cu130
uv pip install "torch==2.12.1" "torchvision==0.27.1"
uv pip install "axolotl==0.19.0"
uv pip install "pytest==9.1.1" "matplotlib==3.11.2"
CUDA_HOME=/usr/local/cuda-13.0 \
CC=/usr/bin/gcc \
CXX=/usr/bin/g++ \
TORCH_CUDA_ARCH_LIST=9.0 \
  uv pip install --no-build-isolation "flash-attn==2.8.3"
CCE_REPOSITORY=https://github.com/axolotl-ai-cloud/ml-cross-entropy.git
CCE_REVISION=4dfa52212d0552805aec5a6a2176c0abbea277d2
CUDA_HOME=/usr/local/cuda-13.0 \
CC=/usr/bin/gcc \
CXX=/usr/bin/g++ \
TORCH_CUDA_ARCH_LIST=9.0 \
  uv pip install --no-build-isolation \
  "cut-cross-entropy[transformers] @ git+${CCE_REPOSITORY}@${CCE_REVISION}"
uv pip check
pytest -q tests

Don't install the deepspeed extra. The configs use FSDP2, and its optional deepspeed-kernels dependency has no compatible Python 3.12 wheel. Keep the CUDA toolkit, PyTorch backend, and H100 architecture settings together when you rebuild either compiled extension.

Authenticate W&B with wandb login rather than putting a credential in YAML or shell history. On a disconnected training node, cache the model and dataset first, set wandb_mode: offline in a copy of the config, and run wandb sync <run-directory> from a connected host afterward.

Prepare your data

Axolotl consumes OpenAI-style messages. Text content can be a string. Multimodal user content is an ordered list of typed image and text parts, with each image placed before the question it belongs to:

JSON
{
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "image", "path": "images/page-01.png"},
        {"type": "image", "path": "images/page-02.png"},
        {"type": "text", "text": "Compare the totals on both pages."}
      ]
    },
    {"role": "assistant", "content": "The second total is 18 percent higher."}
  ]
}

The configs train on the circuit subset of CoSyn-400K(opens in new tab). The preparation script emits one conversation per question and answer pair and writes a manifest with source revisions, content hashes, and data statistics:

bash
python scripts/prepare_dataset.py cosyn-circuit \
  --tokenizer-model meta-models/Muse-Glimmer-30B \
  --tokenizer-revision a4e59da52a7bc87ae7251dd5545c0dd437c44b68 \
  --max-tokens 4096

The same script prepares a text-only set from Tulu 3 SFT Personas(opens in new tab) (tulu) and a multi-image set from Docmatix(opens in new tab) (docmatix). Add --max-examples-per-split 16 --output-root data/smoke for a quick pipeline check.

The configs set skip_prepare_dataset: true, so axolotl preprocess doesn't exercise the multimodal path. Run the processor preflight in the recipe as well as the training smoke run below.

Start training

Three configs share the same prepared data and a global batch of 32 on eight GPUs:

Config	Method	When to use
`muse-glimmer-30b-qlora.yaml`	LoRA adapters over a 4-bit frozen base	The first run and fast iteration
`muse-glimmer-30b-lora-fsdp2.yaml`	LoRA adapters over a bf16 frozen base, sharded with FSDP2	Adapters without base quantization
`muse-glimmer-30b-full-fsdp2.yaml`	Every parameter, with FSDP2 sharding the training state	Only after an evaluation shows adapter capacity is the bottleneck

Both adapter configs target the language projections and keep the vision tower, adapter, and projection frozen.

Start with a bounded run that crosses two optimizer steps, skips validation, and writes a checkpoint at step 2:

bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  axolotl train configs/muse-glimmer-30b-qlora.yaml \
  --max-steps 2 --gradient-accumulation-steps 1 \
  --eval-strategy no --save-strategy steps --save-steps 2 \
  --launcher torchrun -- --nproc_per_node=8 --nnodes=1

Swap in either FSDP2 config for the other methods. The full-parameter smoke run needs about 222 GB of disk for one sharded training checkpoint plus its consolidated model. Once the smoke run passes, drop the overrides to train for the config's one epoch with validation and an epoch checkpoint.

The full-parameter path uses adamw_torch_fused. Avoid a bitsandbytes 8-bit optimizer with FSDP2 unless you have validated that combination for this model and Axolotl release.

Results

The recipe ran five controlled one-epoch runs on eight H100s with seed 42 and a 4,096-token limit. CoSyn and Tulu used 2,048 training and 256 validation examples over 64 steps; Docmatix used 256 training and 36 validation documents over 32 steps. These are bounded slice results, not production benchmarks:

Dataset	Method	Validation loss, step 0 to final	Trainer runtime	Peak MiB per GPU
CoSyn	QLoRA	2.9434 to 0.8655	320.4 s	27,011
CoSyn	bf16 LoRA FSDP2	2.8341 to 0.8968	651.1 s	41,094
CoSyn	Full bf16 FSDP2	2.8341 to 0.8717	1,105 s	97,340
Tulu	QLoRA	1.1078 to 0.7973	327.2 s	29,401
Docmatix	QLoRA	1.0974 to 0.7187	80.81 s	28,052

Full FSDP2 reached 97,340 MiB while consolidating its final checkpoint, which leaves almost no margin on an H100. The recipe's results(opens in new tab) include the W&B runs, held-out base-versus-tuned generation metrics, and plots.

Use the tuned model

Adapter runs gather a final PEFT adapter, and full-parameter training gathers the full state. To reload a QLoRA adapter through Transformers, pass dtype=torch.bfloat16 to from_pretrained and set bnb_4bit_quant_storage=torch.bfloat16 in BitsAndBytesConfig. Without both, the vision path loads in byte dtype.

For bf16 LoRA, keep FSDP2 activation_checkpointing: false. With Axolotl 0.19.0, activation checkpointing leaks wrapper names into the gathered adapter, and PEFT skips those weights on reload.

Serve the result with any runtime on Run inference.

Troubleshooting

Symptom	Cause	Fix
Only the first page reaches the model	Top-level image list	Put images inline and in order.
Image and text mismatch	Packing or content order	Turn packing off and preserve order.
Out of memory after step one	Adam state allocates on the first step	Size for step two, or shorten inputs.
Only one worker starts	Launcher arguments in the wrong place	Put them after `--`.
W&B creates no run	No login, or the network is blocked	Run `wandb login`, or use offline mode.
QLoRA reload has `Byte` vision tensors	Missing load dtype	Use bf16 model dtype and 4-bit storage.
FSDP2 LoRA reload skips trained keys	Activation checkpointing wrappers leaked into names	Set `activation_checkpointing: false`.
FSDP2 save fails	Version or state mismatch	Use the pinned versions and require a reload check.
DeepSpeed install fails	Unneeded kernels	Leave out the DeepSpeed extra.

TRL

Fine-tune Muse Glimmer with TRL(opens in new tab) and PEFT(opens in new tab) on your own labeled examples. Low-rank adaptation (LoRA) is the recommended approach: it trains a small set of adapter weights while keeping the base model frozen, reducing GPU memory requirements by an order of magnitude.

Choose an approach

Method	Relative training cost	When to use
LoRA	Low	Domain adaptation, style transfer, instruction tuning
QLoRA	Low (slower steps, less memory)	Same use cases as LoRA, with a quantized base model
Full fine-tune	High	Maximum quality when compute is not a constraint

Prepare your data

Format your training data as JSONL with the chat format Muse Glimmer expects:

JSON
{"messages": [{"role": "system", "content": "You are a medical assistant."}, {"role": "user", "content": "What are the symptoms of type 2 diabetes?"}, {"role": "assistant", "content": "Common symptoms include increased thirst, frequent urination, fatigue, and blurred vision."}]}
{"messages": [{"role": "system", "content": "You are a medical assistant."}, {"role": "user", "content": "How is hypertension diagnosed?"}, {"role": "assistant", "content": "Hypertension is diagnosed by measuring blood pressure. A reading consistently at or above 130/80 mmHg indicates hypertension."}]}

One conversation per line.
Include a system prompt in every example if you want the model to follow a consistent persona.
Aim for at least a few hundred (roughly 500+) high-quality examples for meaningful adaptation; several thousand (5,000–10,000+) for strong domain performance. Quality and consistency matter more than raw count.

LoRA fine-tuning

Python
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer
model_name = "meta-models/Muse-Glimmer-30B"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
training_args = TrainingArguments(
    output_dir="./muse-glimmer-lora",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
)
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,  # your loaded dataset
    processing_class=tokenizer,
)
trainer.train()
trainer.save_model("./muse-glimmer-lora")

QLoRA fine-tuning

QLoRA loads the base model in 4-bit precision with bitsandbytes, then trains LoRA adapters on top. This cuts GPU memory requirements substantially compared to standard LoRA.

Python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, TrainingArguments
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
import torch
model_name = "meta-models/Muse-Glimmer-30B"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_name, quantization_config=bnb_config, device_map="auto"
)
model = prepare_model_for_kbit_training(model)
tokenizer = AutoTokenizer.from_pretrained(model_name)
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
trainer = SFTTrainer(
    model=model,
    args=TrainingArguments(
        output_dir="./muse-glimmer-qlora",
        num_train_epochs=3,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=16,
        learning_rate=2e-4,
        bf16=True,
        logging_steps=10,
        save_strategy="epoch",
    ),
    train_dataset=train_dataset,
    processing_class=tokenizer,
)
trainer.train()
trainer.save_model("./muse-glimmer-qlora")

Hyperparameter guidance

Parameter	Recommended range	Notes
Learning rate	1e-4 – 3e-4 (LoRA or QLoRA)	Lower for larger datasets; full fine-tunes use ~1e-5
LoRA rank (`r`)	8 – 32	Higher rank = more capacity, more memory
LoRA alpha	2× rank	Standard scaling
Batch size	4 – 16 (effective)	Use gradient accumulation to reach this
Epochs	2 – 5	Watch for overfitting on small datasets
Model context window	128K tokens (131,072)	Model capacity, not a recommended per-example training length
Training sequence length	Dataset- and hardware-dependent	Start with shorter sequences to reduce memory, and increase only after validating quality and hardware headroom

Evaluate after fine-tuning

Compare your fine-tuned model against the base model on a held-out test set before deploying. Track both your task metric and a general-capability benchmark so you catch regressions from over-fitting.

Load the base model with the adapter you trained. Swap ./muse-glimmer-lora for ./muse-glimmer-qlora if you took the QLoRA path, or point from_pretrained at the merged directory once you have run Merge adapters below.

Python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-models/Muse-Glimmer-30B",
    dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, "./muse-glimmer-lora")
tokenizer = AutoTokenizer.from_pretrained("meta-models/Muse-Glimmer-30B")
correct = 0
for example in test_set:  # each has .messages (prompt) and .expected
    prompt = tokenizer.apply_chat_template(example.messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    output = model.generate(**inputs, max_new_tokens=512, do_sample=True, temperature=1.0, top_p=0.95)
    answer = tokenizer.decode(output[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
    correct += int(example.expected in answer)
print(f"accuracy: {correct / len(test_set):.1%}")

If your fine-tune improves the task metric but drops general capability, reduce epochs, lower the learning rate, or add more diverse data.

Merge adapters

Merge LoRA adapters back into the base model for deployment without the PEFT library:

Python
from peft import PeftModel
from transformers import AutoModelForCausalLM
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-models/Muse-Glimmer-30B",
    dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, "./muse-glimmer-lora")
merged = model.merge_and_unload()
merged.save_pretrained("./muse-glimmer-30b-merged")

The merged model can be served with any runtime (vLLM, llama.cpp) without adapter overhead.

Reinforcement learning

Reinforcement learning optimizes Muse Glimmer against a reward signal rather than fixed target outputs. Use it when you can score an output but can't fully demonstrate it: aligning to human preferences, rewarding successful tool calls, or shaping multi-step agentic behavior. Start RL from a supervised fine-tuned checkpoint.

TitanRL

TitanRL(opens in new tab) is TorchTitan's reinforcement-learning stack. It uses the same TorchTitan model definition in the trainer and its vLLM generator, reducing the risk of a training and serving implementation mismatch.

The data-analyst recipe trains Muse Glimmer to compute a number from a CSV or log file and write it to answer.json. It uses GRPO with a deterministic reward. The tasks need no LLM judge, labeled dataset, or external analysis service.

Precision	bf16, full-parameter training
Generator	TitanRL's in-process vLLM engine
Hardware verified on	8 × H100 95 GB, single node
GPU split	Six trainer GPUs (FSDP 3, TP 2), two generator GPUs (TP 2)
Offline	Tasks run offline; setup downloads dependencies and weights. Use W&B offline mode after setup.
Requires	TorchTitan `main` with TitanRL, Monarch, TorchStore, vLLM, and FlashAttention-3

Install

Install uv(opens in new tab) if needed. Start with fresh clones and a named Python environment:

bash
git clone https://github.com/meta-models/meta-oss-cookbook.git
export COOKBOOK="$PWD/meta-oss-cookbook"
git clone https://github.com/pytorch/torchtitan.git
cd torchtitan
GLIMMER_ENV=glimmer-rl-data-analyst-smoke
uv venv --python 3.12 "$GLIMMER_ENV"
source "$GLIMMER_ENV/bin/activate"

If the cookbook is already cloned, set COOKBOOK to its absolute path. Use a fresh environment name for each independent test or training run.

Run the recipe's installer wrapper from the TorchTitan root:

bash
bash "$COOKBOOK/muse-glimmer/post-training/rl/titanrl/data-analyst/scripts/install_env.sh"

The wrapper delegates to TorchTitan's installer and preserves the installed dataset package's dependency bounds during the nightly resolve. The upstream installer selects matching Torch, torchvision, and vLLM nightlies, installs TorchStore and batch-invariant ops, and selects FlashAttention-3 for H100. Make an installed CUDA toolkit's nvcc available on PATH, or use the installer's --nvcc option.

The upstream installer removes TileLang despite vLLM's package declaration. TorchStore's declared Monarch version can also differ from TorchTitan's runtime requirement; upstream installs TorchStore with --no-deps. These packaging mismatches do not establish a clean dependency check. Keep the upstream runtime stack and run the recipe tests rather than downgrading Monarch to satisfy stale metadata.

Copy the recipe and run its CPU tests:

bash
mkdir -p torchtitan/rl/examples/glimmer_data_analyst
cp -R "$COOKBOOK/muse-glimmer/post-training/rl/titanrl/data-analyst/titanrl_files/." \
  torchtitan/rl/examples/glimmer_data_analyst/
export PYTHONPATH="$PWD:${PYTHONPATH:-}"
uv pip install pytest
pytest torchtitan/rl/examples/glimmer_data_analyst/tests/ -v

The tests construct both real configuration factories and check dataset, reward, and command-execution behavior. They do not build GPU actors or replace the training check.

Download the original open-weights checkpoint:

bash
python scripts/download_hf_assets.py \
  --repo_id meta-models/Muse-Glimmer-30B \
  --local_dir torchtitan/rl/example_checkpoint \
  --all

Use the original Hugging Face checkpoint without pre-converting its Q/K layout. TorchTitan's state-dict adapter handles the conversion on load. Leave space for full checkpoints, which include optimizer state and can be several times larger than the model weights. The full configuration retains up to three checkpoints.

W&B is enabled in both configurations. For online tracking, run wandb login. For local tracking without a login or W&B network traffic:

bash
export WANDB_MODE=offline

To disable W&B, set metrics.enable_wandb=False in the Python configuration. There is no generic dotted-field W&B-disable launch flag. Before each run, activate the environment and export PYTHONPATH; Monarch-spawned workers import the checkout from it.

Start training

From the TorchTitan root, run the smoke configuration first. It requests 10 optimizer updates, four prompts with four samples each, and 16 held-out validation prompts:

bash
export PYTHONPATH="$PWD:${PYTHONPATH:-}"
OUT=outputs/rl/glimmer_data_analyst_smoke
mkdir -p "$OUT"
set -o pipefail
python -m torchtitan.rl.train \
  --module torchtitan.rl.examples.glimmer_data_analyst.config_registry \
  --config rl_grpo_muse_glimmer_30b_data_analyst_smoke \
  --output-dir "$OUT" 2>&1 | tee "$OUT/console.log"

The smoke avoids mid-run checkpoints and retains the final full-state save. The selected module must contain the factory itself.

Use another fresh named environment for the full demonstration. It requests 15 optimizer updates, eight prompts with eight samples each, and 64 held-out validation prompts:

bash
OUT=outputs/rl/glimmer_data_analyst
mkdir -p "$OUT"
set -o pipefail
python -m torchtitan.rl.train \
  --module torchtitan.rl.examples.glimmer_data_analyst.config_registry \
  --config rl_grpo_muse_glimmer_30b_data_analyst \
  --output-dir "$OUT" 2>&1 | tee "$OUT/console.log"

Both configurations train every parameter of the 30B model. The full demonstration is bounded because these easy templates can lose reward variation quickly. Check that final weight synchronization, validation, and checkpoint saving finish, and that the process exits with code zero.

Plot the selected run:

bash
uv pip install matplotlib
python "$COOKBOOK/muse-glimmer/post-training/rl/titanrl/data-analyst/scripts/plot_results.py" \
  "$OUT"

The helper reads console.log, online or offline W&B logs, and rollout_samples.jsonl. Use --log /path/to/saved-console.log if the console output is elsewhere. It writes reward.png and outcomes.png.

The recipe records every sampled sibling and terminal error. Plot statistics describe recorded occurrences; resumed runs can contain replayed IDs. Requested tool-call counts do not establish that commands executed, and a completed rollout does not establish that its answer was correct.

Muse Glimmer's two KV heads cap generator tensor parallelism at 2. Retain full activation rematerialization to leave room for Adam state. Longer contexts or larger batches need their own memory check.

How the reward works

Each rollout gets a private task directory, a data.csv or access.log, a question with one numeric answer, and one tool: bash. Four seeded templates cover filtered CSV sums, maximum group averages, log-level counts, and counts of the most common HTTP status. Train and validation use separate index ranges. The loader checkpoints and replays unacknowledged inputs.

The environment grades answer.json after each tool turn. With at least one successful command, the cumulative rewards are:

Outcome	Reward before repeat penalties
Correct number in `answer.json`	1.0
Well-formed file with the wrong number	0.45
Malformed output file	0.20
Successful command, but no output file	0.05
No progress	0.0

A repeat of a previously failed command deducts 0.05 from partial reward, bounded at zero. A correct artifact receives 1.0 without a repeat penalty. Intermediate credits preserve reward differences while the model learns to execute a command and write an answer.

GRPO compares sibling attempts for the same prompt. Equal rewards provide no relative advantage, so the default filter drops zero-variance groups. Repeated groups with no trainable samples trigger an error. They do not trigger graceful final validation and checkpointing; inspect rewards, checkpoints, and the exit status.

Results

The 10-update smoke and 15-update full demonstration completed on eight H100s with final weight synchronization, held-out validation, full-state checkpoint saving, and exit code zero. A separate restart restored the smoke checkpoint and continued optimizer, scheduler, and data-loader state from update 10 to 11. The CPU and helper suite passed 70 tests without skips.

In the full demonstration, displayed mean training rollout reward rose from 0.24 to 0.98. Held-out correctness improved from 26 of 64 synthetic tasks before training to 64 of 64 afterward. This small, recipe-specific check does not measure general analysis quality. W&B was enabled in offline mode.

The recipe's results(opens in new tab) contain the exact checked sources, environment, metrics, and separate failed long-budget run. Verification used isolated local clones and SHA256-verified cached weights; a fresh public-network bootstrap and online W&B synchronization were not certified.

Make it yours

Add your own task: extend data.py with a prompt, input file and contents, and a deterministic golden value.
Train longer: add harder tasks or a curriculum before increasing async_loop.num_training_steps. Keep GRPO's zero-variance filtering enabled.
Change the reward: adjust rubric.py. For strict 0 or 1 rewards, set file_exists_credit, file_parses_credit, and ran_ok_credit to zero.
Tune token budgets together: the trainer context and microbatch budget are 4096 tokens, generation allows 384, and the prompt cap reserves that generation budget. max_rollout_tokens limits the prompt, not prompt plus generation; output character limits are not tokenizer limits.
Resume a checkpoint: use the same factory and output directory with --resume-step=-1. To continue beyond a finished budget, increase the update limit in the Python factory. Arbitrary dotted-field flags are not supported.

Troubleshooting

Symptom	Check
Factory not found	Select the exact `.config_registry` module shown above.
Unknown `--dump-folder` or dotted-field flags	Use `--output-dir`; configure other fields in Python or a registered upstream override.
Missing TorchStore or vLLM, or native import errors	Finish the installer in the active fresh environment; do not mix environments.
`--print-config` raises an unhashable `instancemethod` error	An upstream PyTorch-enum and CPython pretty-printer incompatibility also affects stock recipes. Normal loading works; inspect `config.to_dict()` instead.
First generation is slow	FlashInfer may be compiling kernels; check that `nvcc` is available.
Out of memory after the first update	Size for Adam state and retain full activation rematerialization.
`truncated_prompt_too_long` dominates	Reduce turn and tool-output budgets, or raise context with enough generation headroom.
Consecutive untrainable groups	Inspect rewards and sample validity. For longer training, use harder tasks instead of treating the guard error as success.
Plot helper cannot find metrics	Save `console.log` with `tee`, or pass `--log`.

TRL

Optimize Muse Glimmer against preference pairs or a programmatic reward with TRL(opens in new tab).

Choose a method

Method	Reward source	Cost	When to use
DPO (Direct Preference Optimization)	Preference pairs (chosen vs. rejected)	Low: offline, no reward model, no sampling	You have pairwise preference data and want alignment without standing up a reward model
GRPO (Group Relative Policy Optimization)	A programmatic or model-based reward	Higher: online sampling during training	You have a verifiable reward (unit tests pass, answer matches, tool call succeeds)
PPO	A trained reward model	Highest: reward model + online sampling	Classic RLHF when you have (or train) a scalar reward model

For most teams, DPO is the fastest path: it needs only preference pairs and trains offline like SFT. Use GRPO when your task has a checkable, programmatic reward (common for reasoning and tool-use). PPO is the most general reinforcement learning from human feedback (RLHF) method but needs the most infrastructure.

Preference optimization with DPO

DPO trains directly on preference pairs, with no separate reward model and no online sampling. Format your data as a prompt with a chosen and a rejected completion:

JSON
{"prompt": [{"role": "user", "content": "Explain a mutex to a new engineer."}], "chosen": [{"role": "assistant", "content": "A mutex is a lock that lets only one thread into a critical section at a time..."}], "rejected": [{"role": "assistant", "content": "A mutex is when the CPU mutexes the threads."}]}

Python
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOConfig, DPOTrainer
model_name = "meta-models/Muse-Glimmer-30B"
model = AutoModelForCausalLM.from_pretrained(model_name, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
config = DPOConfig(
    output_dir="./muse-glimmer-dpo",
    beta=0.1,                 # KL penalty strength; higher = stays closer to the reference
    learning_rate=5e-7,       # DPO uses a much lower LR than SFT
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    num_train_epochs=1,
    bf16=True,
    logging_steps=10,
)
trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=preference_dataset,  # your loaded dataset with prompt/chosen/rejected
    processing_class=tokenizer,
)
trainer.train()
trainer.save_model("./muse-glimmer-dpo")

DPO keeps a frozen reference copy of the model and penalizes the policy for drifting too far from it (the beta term). Start with beta=0.1 and a learning rate around 5e-7. RL-family objectives are far more sensitive to learning rate than SFT.

Online RL with a programmatic reward (GRPO)

When your task has a reward you can compute, such as code that passes tests, a math answer that matches, or a tool call that succeeds, GRPO optimizes against it directly. GRPO samples a group of completions per prompt and reinforces the ones that score above the group average, which removes the need for a separate value (critic) network.

Python
from trl import GRPOConfig, GRPOTrainer
def reward_fn(completions, **kwargs):
    # Return one scalar reward per completion. Example: reward a parseable final answer.
    # Replace this example with your task's verifiable reward.
    return [1.0 if "<answer>" in c else 0.0 for c in completions]
config = GRPOConfig(
    output_dir="./muse-glimmer-grpo",
    learning_rate=1e-6,
    per_device_train_batch_size=1,
    num_generations=8,         # group size sampled per prompt
    max_completion_length=4096,  # allow room for Muse Glimmer's long reasoning traces
    bf16=True,
)
trainer = GRPOTrainer(
    model="meta-models/Muse-Glimmer-30B",  # Start from your SFT checkpoint.
    reward_funcs=reward_fn,
    args=config,
    train_dataset=prompt_dataset,  # prompts only; completions are sampled online
)
trainer.train()

Hyperparameter guidance

Parameter	DPO	GRPO and PPO	Notes
Learning rate	1e-7 – 1e-6	1e-6 – 3e-6	Orders of magnitude lower than SFT
KL penalty (`beta`)	0.1 – 0.5	KL coeff 0.01 – 0.1	Higher keeps the policy closer to the reference
Epochs or steps	1 – 2 epochs	Step-based	RL overfits fast; watch reward and a held-out eval
Group size (`num_generations`)	Not used	4 – 16	Larger groups give a lower-variance advantage estimate

Guard against reward hacking

RL optimizes exactly what you reward, including unintended shortcuts. Protect against drift:

Keep a KL penalty to the reference (SFT) model so the policy can't collapse into degenerate outputs that game the reward.
Hold out an eval set the reward never sees, and track general capability on a domain benchmark alongside reward. If reward climbs while eval quality drops, the model is hacking the reward.
Inspect samples, not just scalars. Read generations at each checkpoint; reward hacking is usually obvious to a human before it shows up in aggregate metrics.

Evaluate and deploy

Compare the RL checkpoint against both the base and the SFT checkpoint on a held-out test set before shipping. When it wins, merge any adapters and serve it with any runtime.

Next steps

Serve the tuned model with a runtime from Run inference, using the chat format in the prompting guide. To convert it for local inference, see quantization.

Post-training

Choose the training signal

Size the hardware

Supervised fine-tuning

Axolotl

Links

Install

Prepare your data

Start training

Results

Use the tuned model

Troubleshooting

TRL

Choose an approach

Prepare your data

LoRA fine-tuning

QLoRA fine-tuning

Hyperparameter guidance

Evaluate after fine-tuning

Merge adapters

Reinforcement learning

TitanRL

Links

Install

Start training

How the reward works

Results

Make it yours

Troubleshooting

TRL

Choose a method

Preference optimization with DPO

Online RL with a programmatic reward (GRPO)

Hyperparameter guidance

Guard against reward hacking

Evaluate and deploy

Next steps

Please confirm your identity

Please confirm your identity