The Post-Training Ladder: From Base Model to Aligned Assistant

A pretrained language model is a fluent next-token predictor with no notion of instructions, formatting, or human preference. Turning it into a useful assistant is a staged process, and Hugging Face TRL gives you a battle-tested trainer for every rung of the ladder. TRL v1 ships stable SFTTrainer, DPOTrainer, GRPOTrainer, RewardTrainer, and DistillationTrainer classes, all built on top of the Transformers v5 model definitions and the Trainer API you already know.

The canonical sequence is: SFT (teach the model the task and chat format) → preference tuning (align outputs to what humans actually prefer) using either DPO or an online RL method like GRPO. Each stage narrows the model's behavior. Skipping SFT and jumping straight to RL almost always produces instability, because the policy has no sane starting distribution to stay close to.

This guide walks the ladder in order, with realistic TRL code and the failure modes that send teams looking for structured post-training help. We will cover packing and chat templates in SFT, the reference model and beta in DPO, reward functions and KL control in GRPO, and how to keep the whole pipeline from silently degrading via reward hacking.

Stage 1: Supervised Fine-Tuning with SFTTrainer

SFT is straightforward maximum-likelihood training on (prompt, response) pairs, but two details make or break it: the chat template and packing. The chat template is a Jinja string stored on the tokenizer that converts a list of role/content messages into the exact token sequence the model was trained on. Get it wrong and every downstream stage inherits a formatting mismatch.

from datasets import load_dataset
from trl import SFTTrainer, SFTConfig

dataset = load_dataset("trl-lib/Capybara", split="train")

config = SFTConfig(
    output_dir="Qwen3-SFT",
    max_length=4096,
    packing=True,                 # concatenate samples to fill the context
    packing_strategy="bfd",       # best-fit-decreasing bin packing
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=2e-5,
    bf16=True,
    assistant_only_loss=True,     # mask loss on prompt tokens
    logging_steps=10,
)

trainer = SFTTrainer(
    model="Qwen/Qwen3-4B-Base",
    args=config,
    train_dataset=dataset,
)
trainer.train()

Two settings deserve emphasis. packing=True concatenates multiple short samples into full-length sequences so you stop wasting compute on padding — on chat data this often doubles throughput. assistant_only_loss=True masks the loss on the prompt and system tokens so the model learns to generate responses rather than memorize prompts; this relies on the chat template emitting the assistant-token boundaries correctly.

When you pass a model string, SFTTrainer loads the tokenizer and applies its chat template automatically for conversational datasets. If you are fine-tuning a base model with no template, set one explicitly before training or your DPO/GRPO stages will drift. Teams doing this at scale usually pair SFT with Accelerate for multi-GPU sharding and FSDP so a 4B–70B run fits in memory.

Stage 2: Preference Tuning with DPOTrainer

Direct Preference Optimization skips the explicit reward model of classic RLHF. Instead it trains directly on triplets of (prompt, chosen, rejected), optimizing the policy to raise the log-probability of chosen responses relative to rejected ones, while a frozen reference model anchors the policy so it does not collapse. The strength of that anchor is the single most important hyperparameter, beta.

from datasets import load_dataset
from trl import DPOTrainer, DPOConfig

# Each row: {"prompt": ..., "chosen": ..., "rejected": ...}
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

config = DPOConfig(
    output_dir="Qwen3-DPO",
    beta=0.1,                     # KL strength vs the reference model
    learning_rate=5e-7,           # much smaller than SFT
    max_length=2048,
    max_prompt_length=1024,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=16,
    bf16=True,
    loss_type="sigmoid",          # standard DPO; also ipo, robust, etc.
)

trainer = DPOTrainer(
    model="Qwen3-SFT",            # start from your SFT checkpoint
    args=config,
    train_dataset=dataset,
)
trainer.train()

Understanding beta. A low beta (0.01–0.05) lets the policy move far from the reference and learn strong preferences, but risks degenerate, repetitive, or over-optimized outputs. A high beta (0.3–0.5) keeps outputs close to the SFT model but barely shifts behavior. Most teams start at 0.1 and tune from validation win-rate. The learning rate must be dramatically lower than SFT — 5e-7 is typical — because DPO gradients are sharp.

The reference model. By default DPOTrainer creates a frozen copy of the initial policy as the reference, roughly doubling memory. With PEFT/LoRA you can drop the separate copy: TRL computes reference log-probs by disabling the adapter, so a single set of base weights serves both roles. That trick is what makes DPO on a single 80GB GPU practical for 7B–14B models. If your preference pairs are noisy, expect the win-rate to plateau; that is a data problem, not a hyperparameter one. Deeper diagnosis is exactly what our DPO job support and broader TRL support engagements focus on.

Stage 3: Online RL with GRPOTrainer

Group Relative Policy Optimization is where post-training gets both powerful and dangerous. Unlike DPO, which trains on a fixed offline dataset, GRPO is online RL: for each prompt the model generates a group of completions, each is scored by one or more reward functions, and the advantage of each completion is computed relative to the group mean — no separate value/critic network required. That group-relative baseline is what makes GRPO cheaper than PPO while still working well for reasoning and verifiable-reward tasks.

from datasets import load_dataset
from trl import GRPOTrainer, GRPOConfig

dataset = load_dataset("trl-lib/tldr", split="train")

# Reward functions receive completions and return one float per completion.
def reward_len(completions, **kwargs):
    target = 50
    return [-abs(len(c.split()) - target) for c in completions]

def reward_format(completions, **kwargs):
    return [1.0 if c.strip().endswith(".") else 0.0 for c in completions]

config = GRPOConfig(
    output_dir="Qwen3-GRPO",
    num_generations=8,            # group size G per prompt
    max_completion_length=256,
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,
    beta=0.04,                    # KL penalty toward the reference model
    learning_rate=1e-6,
    use_vllm=True,                # offload generation to vLLM for speed
    bf16=True,
)

trainer = GRPOTrainer(
    model="Qwen3-SFT",
    args=config,
    reward_funcs=[reward_len, reward_format],
    train_dataset=dataset,
)
trainer.train()

Reward functions are plain Python callables. They receive the decoded completions (plus prompts and any dataset columns as kwargs) and return a list of floats. You can stack several — a correctness check, a format check, a length penalty — and TRL sums them. For verifiable domains like math or code, the reward is often a unit-test pass or a regex match, which is far more robust than a learned reward model.

num_generations is the group size G. Larger groups give a lower-variance advantage estimate but multiply generation cost and memory linearly. Values of 4–16 are common; 8 is a sane default. beta here is the KL penalty pulling the policy back toward the reference model — set it too low and the policy exploits the reward; too high and it never learns.

Why GRPO runs out of memory

GRPO is memory-heavy because three things live at once: the trainable policy, the frozen reference model for the KL term, and a batch of generated sequences that scales with num_generations × max_completion_length. The usual OOM fixes: enable use_vllm=True to run generation in a dedicated vLLM process (colocated or on a separate GPU), lower num_generations or max_completion_length, use LoRA to drop the reference-model copy, and lean on Accelerate + DeepSpeed ZeRO-3 to shard the policy. Generation, not the backward pass, is typically the bottleneck — which is why fast serving matters even during training.

Instability and reward hacking

The failure signature of GRPO is a reward curve that keeps climbing while sample quality falls off a cliff. This is reward hacking: the model finds a shortcut that satisfies the letter of your reward function while violating its intent. A length reward gets gamed with padding filler; a "contains the answer" reward gets gamed by dumping every possible answer. Mitigations: combine multiple orthogonal rewards, add a KL penalty (that is what beta is for), cap reward magnitudes, and always eyeball raw completions rather than trusting the scalar. If your KL suddenly spikes, the policy has bolted from the reference — lower the learning rate or raise beta. This class of debugging is the most common reason teams reach for GRPO job support.

When to Use Each Method

  • SFT only — when you have high-quality demonstration data and need the model to learn a task or format. It is the cheapest, most stable stage and the foundation for everything else. Never skip it.
  • DPO — when you can collect or synthesize preference pairs cheaply and want a stable, offline, single-pass alignment step. It is the default for most instruction-following and tone/safety alignment because it is far easier to run than online RL.
  • GRPO — when the objective is verifiable (math, code, tool use, structured extraction) so you can write a programmatic reward, and when you have the GPU budget for online generation. It shines exactly where preference data is hard to author but correctness is checkable.

A pragmatic pipeline for a reasoning model looks like: SFT on demonstrations → DPO to fix tone and format → GRPO with verifiable rewards to push accuracy. Each stage should be gated by evaluation, not vibes.

Evaluating Every Stage with Lighteval

Post-training without evaluation is guessing. Lighteval is Hugging Face's evaluation harness for running standardized benchmarks (MMLU, GSM8K, IFEval, and custom suites) against a checkpoint, and it plugs into the same ecosystem as TRL. Run it after every rung of the ladder so you can attribute regressions to a specific stage.

# Evaluate an SFT/DPO/GRPO checkpoint on a benchmark suite
lighteval accelerate \
  "model_name=Qwen3-GRPO,dtype=bfloat16" \
  "leaderboard|gsm8k|0|0" \
  --output-dir ./eval-results

The pattern that catches problems early: log training reward/loss AND an external benchmark. If GRPO reward rises but GSM8K accuracy falls, you have caught reward hacking before it ships. Pair that with a small held-out set of raw prompts you read by hand — automated metrics miss degeneration that a human spots in ten seconds. Teams moving these pipelines into production usually formalize this loop as a CI gate, which is a core part of Hugging Face production support.

Get Expert Help With Your TRL Pipeline

SFT, DPO, and GRPO each have their own failure modes — masked-loss bugs, mis-tuned beta, exploding KL, generation OOM, and reward hacking that only shows up at eval time. If you are building a post-training stack and want experienced engineers to review your configs, unblock a stuck run, or prep for an ML engineering interview, our Hugging Face proxy job support team works alongside you in real time. Reach out and ship your aligned model with confidence.