Post-training

From base model to assistant

A base model from pretraining (module 10) only continues text. Ask it a question and it may answer, or write five more questions. Post-training is everything after pretraining that turns it into a useful assistant: following instructions, using the chat template, calling tools, reasoning, refusing harmful requests, and being good at the tasks you care about.

It’s much cheaper than pretraining (thousands to millions of examples instead of trillions of tokens), and it’s where small teams can actually compete. The usual stages:

1. SFT (supervised fine-tuning): learn from good example answers.
2. Preference tuning (RLHF/PPO, DPO): learn which of two answers people prefer.
3. RL with verifiable rewards: practise on tasks where a program checks the result.
4. Distillation: learn from a stronger teacher model.

SFT: imitation

SFT is the same next-token training as pretraining, but on curated conversations rendered with the chat template (module 02). One key difference: the loss is usually computed only on the assistant’s tokens. The model shouldn’t learn to write the user’s messages, only the answers.

from trl import SFTConfig, SFTTrainer
trainer = SFTTrainer(
    model="Qwen/Qwen3-0.6B",
    train_dataset=dataset,                      # rows of {"messages": [...]}
    args=SFTConfig(assistant_only_loss=True, learning_rate=2e-5, num_train_epochs=2),
)
trainer.train()

Data matters most. A few thousand excellent, diverse examples beat a million mediocre ones (LIMA). For agents, an SFT example is a whole session: system prompt, tool definitions, tool calls, tool results, final answer.

Full fine-tune vs LoRA vs QLoRA

How the weights get updated is a separate choice from what you train on:

• Full fine-tune: update every weight. Needs about 16 bytes per parameter for weights, gradients and Adam states (module 01).
• LoRA (Hu et al., 2021): freeze the original weights and learn a low-rank update W + B·A for selected matrices, with A [r × in] and B [out × r]. Only A and B get gradients and optimizer states.
• QLoRA (Dettmers et al., 2023): LoRA on top of a frozen base stored in 4-bit NF4. It saves the most memory but adds quantization error to the base during training.

LoRA: how many parameters you actually trainreal Qwen3-0.6B matrix shapes · 28 layers · 596M parameters

rank 4rank 8rank 16rank 32rank 64rank 128

attention only (q,k,v,o)attention + MLP (all linear)

Practical notes:

• Rank (r) is the adapter’s capacity. 16-64 is common. lora_alpha scales the update and is often set to r or 2r.
• Target all linear layers, not just attention. Thinking Machines found LoRA matches full fine-tuning in most post-training settings when applied to all layers with a higher learning rate (roughly 10×), and that it does especially well for RL (LoRA Without Regret).
• Merging: after training, W + B·A can be added into the base weights, giving a normal model with no extra inference cost. Or keep the adapter separate (a few MB) and swap it at runtime.
• Catastrophic forgetting: fine-tuning on a narrow dataset can erase general skills. Mixing in some general “replay” data and LoRA’s smaller updates both help (Biderman et al., 2024).

Preference tuning: RLHF, PPO, DPO

Some qualities are easier to judge than to demonstrate: helpfulness, tone, safety. RLHF (InstructGPT):

1. Collect pairs of answers to the same prompt and have people pick the better one.
2. Train a reward model to predict those choices.
3. Optimize the policy (the LLM) against the reward model with PPO (Schulman et al., 2017), with a KL penalty that keeps it close to the SFT model so it doesn’t learn to game the reward.

DPO (Rafailov et al., 2023) skips the reward model and the RL loop: a simple loss directly increases the probability of the preferred answer relative to the rejected one, compared with a frozen reference model. Much easier to run, so it became the default for open models.

RL with verifiable rewards

For math, code and agent tasks you don’t need human judges: a program can check the answer. Run the tests, compare the number, check whether the file exists. This is RLVR (named in Tülu 3), and it’s what produced the reasoning models.

GRPO (DeepSeekMath) is the most used algorithm:

1. For one prompt, sample a group of answers (say 8).
2. Score each with the verifier.
3. Each answer’s advantage is its score relative to the group’s average. Better-than-average answers get reinforced, worse ones suppressed.
4. No separate value model is needed (unlike PPO), which saves a lot of memory.

DeepSeek-R1 showed that RL like this, with only correctness rewards, makes models learn to write long chains of thought, check their work and backtrack. Those are the reasoning traces you see in <think> blocks.

What makes RL hard in practice isn’t the algorithm, it’s the environments: tasks at the right difficulty (too easy or too hard gives no learning signal), verifiers that can’t be gamed (reward hacking: deleting the failing tests counts as “all tests pass”), and sandboxes that are fast enough to run thousands of attempts.

Agentic RL in real harnesses

Everything above trains on prompts and answers. A coding agent doesn’t work like that: it runs inside a harness (OpenCode, Claude Code, Codex, Terminus-2, Mini-SWE-Agent…), the program that runs the loop, gives the model its tools, writes the context it reads, parses what it sends back and decides when to stop. Agentic RL is RL on whole multi-step tool-using sessions, and the obvious place to do it is inside the harnesses people actually use.

This section follows one public study, The ultimate guide to multi-harness RL (Hugging Face, updated 1 October 2026). All of the numbers below are theirs, copied from the article’s data files. We didn’t measure them.

The problem: scores come with a harness attached

With the same weights, their base model (LFM2.5-2.6B) solved 62% of the test tasks under Mini-SWE-Agent and 33% under Claude Code. Another measurement quoted in the article put GLM-5.2 at 23% in one harness and 52% in another on SWE-bench Pro. A model that never trained in your harness calls tools it doesn’t have or writes output it can’t parse. Train it in one harness and it picks up that harness’s habits. Module 14 makes the same point for evaluation: always name the harness next to the score.

White-box vs black-box environments

An RL environment owns the task, the state, the tools and the reward. The harness runs the loop. In a white-box setup the trainer drives: it samples an action, calls env.step(), reads the observation and samples again, so it already holds every token. This is what TRL’s GRPOTrainer expects by default. A real harness is black-box: it drives the loop itself and only calls a model API. So the trainer has to get the tokens back some other way.

The stack: OpenEnv + Harbor + capture proxy + TRL

• OpenEnv (GitHub): a standard interface for RL environments with Gymnasium-style reset / step / state, typed actions and observations, packaged as Docker and publishable to the Hub. It started as a Meta PyTorch + Hugging Face project and is BSD-3-licensed. It doesn’t train anything or define rewards.
• Harbor supplies the tasks and the sandboxes, with the harness picked per rollout. It’s the same tool Otter’s baseline.sh uses to run Terminal-Bench.
• The capture proxy pretends to be a normal model provider. The harness gets a base URL and an API key, and the key is really a session id for that one rollout. The proxy accepts the four API dialects harnesses speak, converts them to chat completions for vLLM, and replies in the caller’s own format.
• TRL trains on what the proxy recorded, using async GRPO.

Why the exact tokens matter

RL updates the probability of the tokens the model actually sampled. If you only keep the text and tokenize it again later, you can get different token ids than the ones that were generated (module 02), and the update then teaches the wrong thing. So for every call the proxy stores the prompt’s token ids, the sampled token ids and a logprob for each sampled token:

vllm serve <model> --return-tokens-as-token-ids --logprobs-mode processed_logprobs

It also samples from the full distribution, with no top_p or top_k (module 03). Truncating the distribution makes the samples differ from what the policy itself would produce, and in RL that’s a known route to entropy collapse. In their runs, moving top_p to 1.0 took the importance ratio (trainer probability ÷ sampler probability, ideally 1) from 0.985-0.993 to 0.9984-0.9999.

Hosted APIs (OpenAI, Anthropic, HF Inference Providers) don’t return these, so they can evaluate a harness but can’t be trained through.

Harnesses also retry, spawn subagents and compact their context, so a rollout is a tree of calls, not one clean chat. The proxy links each call to the earlier call that forms the longest exact token prefix of its prompt. Claude Code rewrites its history as it goes, so each of its rollouts turned into about eight training rows.

The setup

• Tasks: 1,000 training tasks (data analysis built from Kaggle notebooks) and 250 held-out test tasks, each tested once under each of the 4 harnesses (1,000 cells).
• Runs: OpenCode-only vs multi-harness, where each GRPO group of 8 rollouts uses one of the four harnesses.
• Async GRPO: one H100 trains while a second serves the policy with vLLM and keeps generating rollouts. The catch is that rollouts come from weights usually 2 optimizer steps old, never more than 4.
• Hardware: 2 H100s per run, plus one sandbox per rollout (1 CPU, 4 GB). Training took about 32 h (OpenCode only) and 46 h (multi-harness), not counting eval and restarts. They don’t give a cost.

The reward: when the whole group is right

GRPO learns from differences inside a group. If all 8 rollouts are correct, a correctness-only reward scores them all the same, and that group teaches nothing. In earlier Qwen runs with correctness alone, 35-58% of optimizer steps taught nothing, and tool calls per rollout crept from 13 to 41, because nothing told the model to stop exploring.

For LFM they added a tool-efficiency bonus of at most 0.1, paid only on correct answers (a wrong answer still scores 0, so giving up early earns nothing). In 17.5% (OpenCode-only) and 22.6% (multi-harness) of groups, every rollout was correct and the bonus was the only signal. The multi-harness model made 31% fewer tool calls than the base model on tasks both solved. There was no LFM run without the bonus, so its exact effect isn’t isolated.

Results

Same model, four harnesses: what training where doesLFM2.5-2.6B on 250 held-out tasks × 4 harnesses, pass@1 (data from the FineEnvs article)

RL · OpenCode onlyRL · multi-harnessSFT · OpenCode onlySFT · multi-harness

LFM2.5-2.6B curvesQwen3.5-2B curves

• A model gets better where it trains. OpenCode-only RL took OpenCode from 33.6% to 58% at the final step, while the three harnesses it never saw moved much less. Multi-harness RL improved every harness and finished at 54% overall, against 52% for OpenCode-only. The article calls that 2-point overall gap noise. The real difference is where the gains landed.
• Imitation did less than RL. SFT on 3,189 successful sessions from a 27B teacher reached 47.5% (OpenCode data) and 43.1% (all harnesses), below both RL runs. Multi-harness SFT still cut tool calls by 24%, but its flat overall score hid gains of 4-9 points in three harnesses and a 17-point drop under Mini-SWE-Agent.
• Exact tokens aren’t enough. The Qwen3.5-2B runs (bottom chart) rose and then fell back. One run’s answers grew past the output limit evaluation allowed, and the other kept calling tools without submitting. The LFM runs used the same output limit in training and eval plus a reward that values finishing, and didn’t decline, though the model and task pool changed too.
• Steps are a poor unit. Both LFM runs took 1,000 steps, but the multi-harness one saw 626 distinct tasks against 555 and 451M training tokens against 162M.

Caveats they state: one small model, one task family, one attempt per cell, and the RL rows in the comparison are the best checkpoint picked on the test set.

What it means for Otter (our reading)

• Otter’s order (SFT on verified teacher sessions first, RL later) fits this. This study suggests the RL step is where most of the gain is.
• Train and evaluate in the harnesses Otter will actually be used in, not just Terminus-2, and report each harness separately.
• Add an efficiency bonus that only pays on correct answers, and keep training and eval output limits identical. That’s especially relevant because the runs that collapsed were Qwen3.5-family, the same family as Ornith.
• Budget: at their scale, 2 H100s for 32-46 h is about 64-92 H100-hours for a 2.6B model. A 9B model needs more. Measure on a short pilot before committing.
• Their training scripts, with HF Jobs and Slurm instructions, are linked from the article’s “Reproducing it” section. The SFT data is public as FineEnvs/SmolDataEnvs-multiharness-sft.

Distillation

A strong teacher model teaches a smaller student:

• Sequence-level (the cheap, common way): the teacher writes answers or full agent sessions, you keep the good ones, and you SFT the student on them. DeepSeek-R1’s small “distill” models were made this way from R1’s reasoning traces.
• Logit-level (Hinton et al., 2015): the student learns to match the teacher’s whole probability distribution for each token, which carries much more information than the single sampled token. Requires the same tokenizer.
• On-policy distillation (GKD): the student writes its own answers and the teacher scores every token, so the student learns from its own mistakes. Needs access to the teacher’s logprobs on text the student wrote.

Case study: the Otter-9B plan

Otter is your project to post-train a 9B model for agentic coding. Its plan (from the Otter repo’s TRAINING_PLAN_FULL.md and design docs) maps directly onto this module:

stage	Otter’s choice	why
base model	Ornith-1.5-9B (Qwen3.5 architecture)	already strong at agent tasks; same tokenizer as a big teacher
data	teacher agent trajectories, kept only if verified successful (rejection sampling), plus 20-30% general replay data	distillation via SFT; replay against forgetting
method	BF16 LoRA, not QLoRA	Unsloth advises against 4-bit QLoRA for Qwen3.5-family models; ~22 GB for 9B
later	on-policy distillation, then GRPO-family RL in verified environments (see agentic RL in real harnesses)	the published consensus recipe the plan summarizes: SFT → on-policy distillation → RL → merge
evaluation	Terminal-Bench 2.1 baseline before and after	measure the gain, not the vibe (module 14)

Tools

• TRL (Hugging Face): SFT, DPO, GRPO, GKD trainers.
• Unsloth: fast, memory-efficient LoRA/QLoRA on one GPU.
• Axolotl, LLaMA-Factory: config-file driven fine-tuning (the training recipes from module 13).
• mlx-lm: LoRA fine-tuning on a Mac.
• verl, prime-rl, SkyRL: RL at scale.

Where to go deeper

• Nathan Lambert, RLHF Book.
• Hugging Face, LLM course: fine-tuning chapters.
• Tülu 3 report: a complete open post-training recipe with data and code.