Supervised Fine-Tuning

At the end of Part IV you had a fully pretrained base model. It is a marvel: trained on trillions of tokens, it can complete text with uncanny fluency, has absorbed a vast amount of world knowledge, and predicts the next token better than any system before it. And yet, if you tried to USE it as a chatbot, you would be disappointed. This chapter explains why — and how to fix it.

The reason is simple once you see it. A base model was trained on one objective and one objective only: predict the next token in web text. It is a text-completion engine. It has never been taught to answer questions, follow instructions, or hold a conversation. It only knows how to continue whatever text it is given, in the style of its training data.

Suppose you give a base model the prompt 'What is the capital of France?'. You might expect 'Paris'. But the base model does not 'answer' — it CONTINUES. On the web, a line like that is often followed by more quiz questions, so the base model might continue with another question instead of answering:

Neither response is 'wrong' as text completion — the base model is doing exactly what it was trained to do. The problem is a MISMATCH between what it was trained to do (continue text) and what we want it to do (helpfully answer). Supervised fine-tuning is how we close that gap: we take the capable base model and teach it the NEW behaviour of being a helpful assistant.

Modern assistant models are built in two big stages. First, PRETRAINING (Parts III–IV) gives the model its raw capabilities and knowledge — this is the expensive part, costing millions of dollars. Then, POST-TRAINING (this Part V) shapes that raw capability into helpful, harmless, honest behaviour — this is comparatively cheap, but it is what makes the model usable.

Instruction tuning — the most common form of SFT — is conceptually the simplest idea in this entire book. We show the model many examples of instructions paired with good responses, and we train it to produce those responses. That is it. The model learns, by imitation, to respond helpfully to instructions it has never seen.

The training data for instruction tuning is a collection of (instruction, response) pairs. Each pair is a demonstration of the behaviour we want. Here are a few examples of what such pairs look like:

Notice the diversity: summarization, translation, creative writing, explanation, coding. The model is not trained on one task — it is trained on MANY tasks, all framed as 'here is an instruction, here is a good response'. This diversity is the key. By seeing thousands of different instructions, the model learns the GENERAL skill of 'follow the instruction helpfully', which then transfers to brand-new instructions at test time.

Early instruction-tuning datasets were built by converting existing NLP datasets into instruction format (the FLAN and T0 approach). Modern datasets use a mix of sources: human-written demonstrations (expensive but high-quality), examples distilled from a stronger model (cheap and scalable), and curated, filtered collections. We cover data curation in detail in Section 22.5; for now, the key point is that the data is a set of demonstrations of good behaviour.

Here is good news for the beginner: the SFT training objective is the SAME next-token cross-entropy loss you already know from Chapter 15. We are not learning a new kind of training. We take the instruction-response pairs, format them into sequences, and train the model to predict each next token — exactly as in pretraining. The model architecture does not change at all.

There is just ONE important twist, and understanding it is essential: we usually do NOT want the model to be penalized for failing to predict the INSTRUCTION tokens — only the RESPONSE tokens. We mask out the loss on the prompt. Let us build up to why.

First, we concatenate the instruction and response into a single sequence, with special markers showing where each begins (more on these markers in Section 22.4). Schematically, one training example becomes:

During pretraining, the model predicts EVERY token. But in SFT, the instruction is GIVEN — it is the input, not something the model should learn to generate. If we trained the model to predict the instruction tokens too, we would waste capacity teaching it to generate instructions (which we never want it to do) instead of focusing on generating good responses. So we apply a LOSS MASK: we compute the loss only on the response tokens and ignore the prompt tokens.

In Section 22.3 we used informal markers like '' and ''. Real models use precise, fixed formats called CHAT TEMPLATES, built from special tokens. Getting these exactly right is one of the most common stumbling blocks for beginners, so we will go slowly and carefully here.

A conversation has structure: there are turns, and each turn has a ROLE — system (instructions about how to behave), user (the human), or assistant (the model). The model needs to know where each turn begins and ends, and who is speaking. Special tokens mark these boundaries. The model is TRAINED with these exact tokens, so at inference time you must use the SAME tokens, or the model will be confused.

One widely-used format is ChatML, which wraps each turn in '<|im_start|>' and '<|im_end|>' tokens with the role name. Here is a full conversation in ChatML format:

There is no single standard chat template — each model family invented its own. LLaMA-2 uses '[INST]' and '[/INST]' markers; LLaMA-3 uses a different header-based format; Mistral, Gemma, and others each differ. The content is the same conversation; only the wrapping tokens differ. This is why you must use the template that MATCHES the model you are fine-tuning or running.

If SFT is just training on demonstrations, then the demonstrations are everything. The single most important factor in SFT success is DATA QUALITY. And the most surprising, important finding in this area is that you need far LESS data than you might expect — but it must be GOOD.

The LIMA paper (Zhou et al., 2023) made a striking claim, backed by experiment: a model fine-tuned on just 1,000 carefully-curated, high-quality examples could rival models trained on tens of thousands. They called the explanation the 'Superficial Alignment Hypothesis': a model's knowledge and capabilities are learned almost entirely during pretraining; SFT mostly teaches the model which of its existing abilities and FORMATS to use when responding.

There are three main ways to obtain instruction data, each with trade-offs:

The two responses above answer the same instruction, but the left (chosen) one is the kind of demonstration you want in your SFT set: accurate, complete, well-structured, with a concrete example. The right (rejected) one is vague, lowercase, and unhelpful. Training on responses like the right one teaches the model to be vague and unhelpful. This is why curation — actively keeping the good and discarding the bad — is the heart of SFT data work.

We now have all the pieces: the data (instruction-response pairs), the format (chat template), and the objective (masked next-token loss). Let us assemble them into a complete, working SFT training loop. We will build it step by step so every line is clear.

The training loop is almost identical to the pretraining loop of Chapter 15. The only differences are the masked labels and the much smaller dataset and learning rate. Notice how familiar this looks — SFT really is just focused fine-tuning.

The training loop in Section 22.6 does FULL fine-tuning: it updates every one of the model's billions of parameters. This works, but it has serious practical problems that motivate the parameter-efficient methods in the rest of the chapter.

Recall the memory budget from Chapter 18: full fine-tuning with AdamW needs about 16 bytes per parameter (the model, gradients, and two optimizer moments, plus the fp32 master copy). For a 7-billion-parameter model, that is about 112GB — more than a single 80GB GPU can hold. To fully fine-tune even a modest model, you need multiple expensive GPUs and the distributed-training machinery of Chapter 18.

Full fine-tuning produces a complete copy of the model for each task. If you fine-tune a 7B model for ten different tasks, you have ten 14GB checkpoints — 140GB of storage, and you must load a whole new model to switch tasks. This does not scale when you want many specialized variants.

When you update all parameters on a narrow SFT dataset, the model can 'forget' some of its pretrained knowledge — a phenomenon called catastrophic forgetting. The aggressive updates that teach the new behaviour can overwrite capabilities the model had before. Conservative learning rates help, but the risk grows with the amount of fine-tuning.

LoRA (Low-Rank Adaptation; Hu et al., 2021) is the most important parameter-efficient fine-tuning method. It is elegant, effective, and widely used. We met it briefly in Chapter 20; here we develop it carefully and from the ground up, because understanding it well is essential for modern fine-tuning.

Full fine-tuning learns an update ΔW to each weight matrix W, giving a new weight W + ΔW. The update ΔW has the same shape as W — for a 4096×4096 matrix, that is ~16.8 million numbers per matrix. LoRA's insight: this update ΔW tends to have LOW RANK. That means it can be FACTORED into the product of two much smaller matrices, capturing almost the same change with far fewer numbers.

Let us make the savings concrete. For a 4096×4096 weight matrix (d = 4096) with LoRA rank r = 8: full fine-tuning trains 4096×4096 ≈ 16.8 million parameters, while LoRA trains 2×4096×8 = 65,536 parameters — a 256× reduction. And because the original W is FROZEN, we only need to store and optimize those 65k LoRA parameters per matrix, not the whole model.

During the forward pass, instead of computing y = Wx, LoRA computes y = Wx + B(Ax). The frozen W does its usual work, and the small low-rank path B(Ax) adds the learned adjustment. Crucially, only A and B receive gradients; W never changes. Here is the structure:

LoRA has two main knobs that beginners must understand:

A subtle but important detail: LoRA initializes B to ZERO and A to small random values. This means that at the start of training, BA = 0, so the model behaves EXACTLY like the original pretrained model. Training then gradually grows the update from zero. This is why LoRA fine-tuning is stable: it begins as the unchanged base model and departs from it smoothly.

A wonderful property of LoRA: after training, you can MERGE the update back into the original weights by computing W + (α/r)BA once, producing a normal weight matrix. The merged model has the EXACT same architecture and inference speed as the original — the LoRA path adds zero inference cost. You can also keep the adapter separate and swap different LoRA adapters in and out to switch tasks instantly, since each adapter is just a few megabytes.

LoRA dramatically reduces the TRAINABLE parameters, but you still need to hold the full frozen base model in memory for the forward and backward passes. For a 70B model in bf16, that frozen model alone is ~140GB — still too large for a single GPU. QLoRA (Dettmers et al., 2023) solves this with a clever combination: store the frozen base model in 4-BIT precision, while training the LoRA adapters in higher precision.

Quantization (covered fully in Chapter 27) stores numbers with fewer bits. QLoRA quantizes the frozen base model to just 4 bits per parameter — a 4× reduction from bf16's 16 bits. Since the base model is frozen and never updated, the precision loss from quantizing it is tolerable. The LoRA adapters, which ARE trained, stay in higher precision (bf16) so their gradients are accurate. The result: a 70B model fine-tunable on a single 48GB GPU.

LoRA is the most popular PEFT method, but it is part of a broader family. Understanding the alternatives helps you appreciate why LoRA won and when another method might fit better. All share the same goal: adapt a frozen model by training only a small number of new parameters.

Adapters (Houlsby et al., 2019) were the original parameter-efficient method: small bottleneck layers (down-project, nonlinearity, up-project) inserted inside each Transformer block, with the rest of the model frozen. They work well but, unlike LoRA, add extra layers that increase inference latency and cannot be merged away. LoRA's mergeability — zero inference cost — is a major reason it became dominant.

Prompt tuning and prefix tuning take a different approach: instead of modifying the weights, they prepend trainable 'soft' vectors to the input (prompt tuning) or to the attention keys and values (prefix tuning). The model weights are entirely frozen; only these few prepended vectors are learned. They are extremely parameter-efficient but generally less powerful than LoRA for difficult adaptations.

Let us consolidate everything into a practical recipe you could follow to fine-tune a base model into an instruction-following assistant. This section is the hands-on synthesis of the chapter.

After fine-tuning, how do you know it worked? And when results disappoint, how do you diagnose the cause? SFT has characteristic success signals and failure modes, and recognizing them turns debugging from guesswork into method.

Evaluation of an instruction-tuned model is harder than perplexity (Chapter 21), because we care about open-ended helpfulness, not just prediction. The main approaches: hold out some instruction-response pairs and check the model's responses qualitatively; use an automated 'LLM-as-judge' where a strong model rates response quality; and run instruction-following benchmarks (like IFEval or AlpacaEval) that score how well the model obeys instructions. Reading actual generated responses remains the single most informative check.

Exercises 1–11 are pen-and-paper or derivations; 12–22 require code.

Further reading: “Training language models to follow instructions with human feedback” (Ouyang et al., 2022, InstructGPT) for the original SFT+RLHF recipe. “LIMA: Less Is More for Alignment” (Zhou et al., 2023) for the superficial alignment hypothesis. “Finetuned Language Models Are Zero-Shot Learners” (Wei et al., 2021, FLAN). “LoRA: Low-Rank Adaptation of Large Language Models” (Hu et al., 2021) and “QLoRA: Efficient Finetuning of Quantized LLMs” (Dettmers et al., 2023). “Parameter-Efficient Transfer Learning for NLP” (Houlsby et al., 2019, adapters). The Hugging Face PEFT and TRL library documentation for hands-on tooling.

Next → Chapter 23: Reinforcement Learning from Human Feedback

SFT taught the model to follow instructions by imitating good demonstrations — but imitation has limits. The model can only be as good as its demonstrations, and for many qualities (helpfulness, harmlessness, nuanced judgment) it is far easier for humans to COMPARE two responses than to WRITE the ideal one. Chapter 23 introduces Reinforcement Learning from Human Feedback (RLHF): instead of imitating demonstrations, we train a reward model on human PREFERENCES between responses, then use reinforcement learning to optimize the model against that reward. This is how SFT models become the polished, aligned assistants you actually interact with — and the preference pairs you saw in this chapter become the central training signal.