Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
Supervised fine-tuning (SFT) is the first and most important fine-tuning step: it takes a pretrained base model (which just predicts next tokens) and turns it into an instruction follower. The recipe is simple — train on (instruction, response) pairs with the loss computed only on response tokens — but the execution has many footguns: wrong chat template, loss computed on the prompt, too many epochs (overfitting on small datasets), wrong sequence packing, and missing EOS token that causes the model to generate forever. Getting SFT right is the prerequisite for everything in this course.
Supervised fine-tuning has one critical gotcha: the loss must be computed only on the assistant's response tokens, not the user's prompt. If the prompt tokens are included, the model learns to predict the question — wasting capacity and often hurting instruction-following. The DataCollatorForCompletionOnlyLM handles this by setting prompt-token labels to -100, which PyTorch's CrossEntropyLoss ignores. The chat template also matters: a wrong separator token means the model never learns where the response begins.
from datasets import load_dataset; ds = load_dataset('tatsu-lab/alpaca', split='train'); print(ds[0]). What fields does it have? How would you apply format_example to it?DataCollatorForCompletionOnlyLM's output for a sample and verify that the label for prompt tokens is -100 (which PyTorch ignores in CrossEntropyLoss). This is the loss masking.[len(tokenizer(format_example(d))['input_ids']) for d in ds[:100]]. What is the 95th-percentile length? This informs your max_seq_length setting in SFTTrainer.packing=True in SFTTrainer): instead of padding short sequences to max_seq_length, you concatenate multiple sequences into one long one. Estimate how much it speeds up training for a dataset where 80% of sequences are <256 tokens and max_seq_length=2048.# pip install trl transformers datasets peft accelerate bitsandbytes
from trl import SFTTrainer, DataCollatorForCompletionOnlyLM
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments
from datasets import Dataset
import torch
# Tiny synthetic dataset — in practice use Alpaca, Dolly, or your own
data = [
{"instruction": "Explain gradient descent in one sentence.",
"response": "Gradient descent iteratively moves model parameters in the direction that reduces the loss by subtracting a fraction (learning rate) of the gradient."},
{"instruction": "What is a transformer?",
"response": "A transformer is a neural network architecture that processes tokens in parallel using self-attention, enabling it to model long-range dependencies efficiently."},
{"instruction": "What does LoRA stand for?",
"response": "LoRA stands for Low-Rank Adaptation — a parameter-efficient fine-tuning method that adds small trainable matrices to frozen pretrained weights."},
]
# Chat template: format each example as the model expects
def format_example(ex):
return f"<|user|>\n{ex['instruction']}<|end|>\n<|assistant|>\n{ex['response']}<|end|>"
formatted = [{"text": format_example(d)} for d in data]
dataset = Dataset.from_list(formatted)
print("Example formatted input:")
print(formatted[0]["text"])
print()
print("Key: loss is computed ONLY on the assistant response tokens.")
print("The DataCollatorForCompletionOnlyLM handles masking the prompt tokens.")
print()
print("Training config (reference — requires actual model to run):")
training_args = TrainingArguments(
output_dir="/tmp/sft-test",
num_train_epochs=3,
per_device_train_batch_size=1,
gradient_accumulation_steps=4, # effective batch size = 4
learning_rate=2e-4,
fp16=False, bf16=True,
logging_steps=1,
save_strategy="epoch",
warmup_ratio=0.03,
)
print(training_args)python3 main.py