A pretrained model can autocomplete anything and follow nothing — closing that gap is the difference between a demo and a product.
You fine-tuned a model on your own data and it still ignores half your instructions, rambles when it should stop, or refuses perfectly harmless requests. Closing that gap is what post-training is: the stages that turn a raw pretrained large language model (LLM) — a system that only predicts the next token — into one that follows instructions helpfully. You build the whole stack in the order practitioners actually build it. First, supervised fine-tuning (SFT) done correctly: data curation, chat templates, loss masking, sequence packing, and LoRA (low-rank adaptation). Then train a reward model on pairwise human preferences, and watch a policy learn to game it. Then run full reinforcement learning from human feedback (RLHF) using proximal policy optimization (PPO) — all four models it juggles, the penalty that keeps the policy near its starting point, and the rollout loop — before switching to the simpler reward-model-free alternatives led by direct preference optimization (DPO). Later modules cover reinforcement learning from verifiable rewards (RLVR) for math and code, training and evaluating reasoning models, synthetic data pipelines, numeric release gates, and safety tuning. Python-first on Hugging Face TRL, PEFT, and transformers, with runnable code throughout.
Built by Lakshya Kumar
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyFinished the tasks? Take the prompt to your AI and get tested on it. We copy the prompt and open the app — just paste it in.
Most failed fine-tunes are data and masking bugs, not model bugs — get the boring parts right and RL has far less work left to do.
Turn thousands of "this one's better" clicks into a number a model can optimize — then find all the ways that number lies to you.
Four models running at once and one fragile loop — run it, watch it diverge, and learn the handful of knobs that keep it stable.
One loss function, no rollouts, no reward model — DPO gets most of RLHF's benefit for a fraction of the compute and the grief.
When a test suite or a math checker can grade the output, you train on truth instead of taste — and drop the reward model entirely.
Long chains of thought aren't prompted into existence, they're trained — see what the run optimizes and when longer stops helping.
Human labels run out long before your model is done — build the loop that manufactures, filters, and refreshes data without collapsing.
Reward scores climb while users get worse answers — set numeric release gates that catch that before the checkpoint ever ships.
Every point of capability you win can cost you a refusal boundary — calibrate that tradeoff and gate it with a pre-ship checklist.
Complete all modules, then submit the required number of capstone projects. Each must earn a passing rating from an admin reviewer.
Take a small open-weight base model and post-train it — SFT first, then either DPO or GRPO — so the resulting model measurably beats the base model on a target benchmark or reward metric of your choice (a capability benchmark subset, a task-specific verifiable reward, or a reward-model score). Submit the full pipeline (data, training configs, code), the before/after eval numbers, and a written analysis of which specific stage or hyperparameter change moved the needle most and which changes didn't help.
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
I'm taking a "Post-Training & RL for LLMs" course. It covers, in order: the post-training landscape (base vs instruct, capability vs alignment, the alignment tax); SFT done right (curation, chat templates, loss masking, packing, LoRA); reward modeling (preference data, Bradley-Terry, calibration, reward hacking, over-optimization); RLHF with PPO (the four models, GAE, the KL penalty, the clipped objective, instabilities); direct preference optimization (DPO's derivation, beta, IPO, KTO, ORPO); RL with verifiable rewards (GRPO, RLVR for math and code, writing verifiers); reasoning models (long chain-of-thought, self-consistency, distillation, process vs outcome reward); synthetic and preference data pipelines (generation, filtering, dedup, LLM-as-judge, self-instruct, rejection sampling); evaluating post-training (capability vs alignment benchmarks, release gates, contamination); and safety and alignment tuning (refusal calibration, Constitutional AI, RLAIF, red-teaming, jailbreak resistance). My context: 1. The model I want to post-train (size, base checkpoint, open-weight or hosted) is: [describe] 2. What I want it to do differently after post-training: [follow instructions better / prefer a certain style or safety posture / solve verifiable math or code tasks / reason step by step] 3. My current data situation: [no labeled data / some SFT examples / some preference pairs / a verifier I could write] 4. My compute budget: [a single consumer GPU / a small cloud GPU budget / a serious multi-GPU budget] Given that, answer: - Which module should I start with, and why, given what I already have? - Name the specific post-training path (SFT-only, DPO, GRPO/RLVR, or full PPO) you'd recommend for my goal and budget, using the decision framework from Module 1. - Name the single most likely mistake I'll make in my first attempt, based on my stated data and compute situation. - If I only have one week, what's the smallest version of this pipeline that would still teach me something real? - Name one thing this course will NOT solve for me, so I have the right expectations going in.
Train a reward model on real or synthetic preference data, held-out-accuracy-validate it, then deliberately construct an RL or best-of-N loop that exploits a specific weakness in it (length bias or another axis) to demonstrate reward hacking concretely. Then apply at least one mitigation (length normalization, ensembling, a held-out judge check) and show the exploit margin shrink. Submit the reward model, the demonstrated hack, and the before/after mitigation results.
Write and audit your own verifier for a math or code task, then run GRPO (group-relative advantage, no value model) against it on a small model across a meaningfully difficulty-varied prompt set. Report the pass-rate curve across training, the zero-variance-group fraction, and at least one genuine reward-shaping gap you found and fixed.
The DPO derivation behind all of Module 5.