Parameter-Efficient Fine-Tuning

We have spent the whole book building and aligning a model. Now we have to RUN it — serve it to users, fast and cheaply. This turns out to be a completely different engineering challenge from training, with its own bottlenecks, its own metrics, and its own bag of tricks. Part VI begins here, with the techniques that make a trained model practical to deploy.

Here is the key economic fact. Training a model is a ONE-TIME cost — expensive, but paid once. Inference is a RECURRING cost — paid for every single token the model ever generates, for every user, forever. A popular model serves billions of tokens per day. Over a model's deployed life, the total inference cost dwarfs the training cost. This is why inference optimization matters so much: shaving 20% off inference cost saves far more, in absolute terms, than almost any training optimization.

Inference optimization juggles two goals that often conflict. LATENCY is how fast a single user gets their response — it matters for interactive use (a chatbot must feel responsive). THROUGHPUT is how many tokens you can generate across all users per second — it determines cost-efficiency (more throughput means serving more users per GPU). Many techniques trade one for the other, and the right balance depends on the application.

To optimize inference, you must understand that generating a response happens in TWO distinct phases with very different performance characteristics. Confusing them is a common beginner mistake; distinguishing them is the foundation for everything else in this chapter.

When you send a prompt, the model first PROCESSES the entire prompt in one go — this is prefill. All the prompt's tokens are fed through the model together, in parallel, to produce the first output token and to populate the KV cache (Section 27.3). Because all prompt tokens are processed at once, prefill is COMPUTE-BOUND: it does a lot of matrix multiplication and keeps the GPU's compute units busy.

After prefill, the model GENERATES the response one token at a time — this is decode. Each step takes the single most recent token, runs it through the model, and produces the next token, repeating until done. Because each step processes only ONE token, decode is MEMORY-BANDWIDTH-BOUND: the tiny computation for one token is dwarfed by the time spent reading the model weights and KV cache from memory.

We met the KV cache in Chapters 13 and 19. It is so central to inference that we revisit it here as the object around which all of inference optimization revolves. Understanding it deeply makes the rest of the chapter click into place.

Recall how attention works: each new token attends to ALL previous tokens, using their keys (K) and values (V). Without a cache, generating each new token would require recomputing the keys and values for every previous token — enormously wasteful, since those don't change. The KV CACHE stores the keys and values of all previous tokens, so each new token only computes ITS OWN K and V and reads the rest from the cache. This turns generation from O(T²) recomputation into O(T) with caching — it is what makes generation practical.

The KV cache solves a compute problem but creates a MEMORY problem. It grows with every token generated, and at long context lengths and large batch sizes it becomes the dominant consumer of GPU memory — often larger than the model weights themselves. Recall the formula from Chapter 19:

86GB of KV cache for one batch — more than a single GPU's memory. This is why so much of inference optimization targets the cache: shrinking it (GQA from Chapter 19, quantizing it), managing its memory efficiently (PagedAttention, Section 27.7), and reusing it across requests (prefix caching, Section 27.10). The KV cache is the resource that everything fights over.

You cannot optimize what you cannot measure. Inference has a specific vocabulary of metrics, and using the right ones is essential. Let us define them carefully, because they map directly onto the prefill/decode phases from Section 27.2.

These metrics fit together simply. The total time a user waits for a complete response is roughly the time to the first token (TTFT, set by prefill) plus the time per subsequent token (TPOT, set by decode) times the number of tokens generated. For an interactive chatbot, a low TTFT makes it feel responsive (the answer starts quickly), and a low TPOT makes it read smoothly (tokens stream fast enough to read).

The single most impactful inference optimization is QUANTIZATION: representing the model's weights (and sometimes activations and the KV cache) with fewer bits. A model trained in 16-bit precision can often be run in 8-bit or even 4-bit with little quality loss — halving or quartering its memory footprint and, because decode is memory-bound (Section 27.2), speeding it up substantially.

Recall that decode is bottlenecked by READING the weights from memory, not by computing with them. If you store the weights in 4 bits instead of 16, there is 4× LESS data to read per token — so decode gets up to 4× faster, AND the model takes 4× less memory (leaving more room for KV cache and bigger batches). Quantization attacks the exact bottleneck of inference.

Quantization maps a range of floating-point values onto a small set of integers. Instead of storing each weight as a 16-bit float, you store a scale factor per group of weights and represent each weight as a small integer that, multiplied by the scale, approximates the original. The art is choosing the mapping so that the approximation loses as little quality as possible.

Most LLM inference quantization is WEIGHT-ONLY: only the weights are stored in low precision, while computation happens in higher precision (the weights are dequantized on the fly). This is because weights are the memory bottleneck at decode time, and quantizing activations too is harder (activations have outliers that are sensitive to precision loss). Weight-only int4 is the sweet spot for most deployments.

The naive quantization of Section 27.5 works, but smarter methods lose far less quality at the same bit-width. Three names dominate practice — GPTQ, AWQ, and GGUF — and beginners are often confused about how they differ. Let us clear that up.

GPTQ (Frantar et al., 2022) is a clever post-training quantization method. Instead of quantizing each weight independently, it quantizes them in a way that MINIMIZES the resulting error on a small calibration dataset, adjusting the remaining weights to compensate for the error introduced by each quantized weight. This careful, error-aware approach lets GPTQ quantize to 4 bits with minimal quality loss. It is one of the most widely-used 4-bit methods for GPU inference.

AWQ (Activation-aware Weight Quantization; Lin et al., 2023) is based on a key observation: not all weights are equally important. A small fraction of weights — those that multiply large activations — matter disproportionately for quality. AWQ identifies these important weights (by looking at activation magnitudes) and protects them from quantization error by scaling, while aggressively quantizing the rest. This activation-awareness often gives slightly better quality than GPTQ at the same bit-width.

GGUF is frequently confused with GPTQ and AWQ, but it is a different KIND of thing: it is a FILE FORMAT (used by llama.cpp), not a quantization algorithm. GGUF files store quantized models in a portable format optimized for running on CPUs, laptops, and Apple Silicon — enabling LLMs to run locally on consumer hardware without a GPU. GGUF supports many quantization levels (Q4_K_M, Q5_K_M, Q8_0, etc.), each a different bit-width and scheme. When you download a model to run locally on a Mac or PC, it is usually GGUF.

We saw in Section 27.3 that the KV cache is a huge consumer of memory. But there is a second, subtler problem: traditional serving WASTES much of the memory it allocates to the cache. PagedAttention (Kwon et al., 2023), the technique behind the vLLM serving engine, solves this and was a major leap in serving efficiency.

Traditionally, when a request arrives, the server allocates one big CONTIGUOUS block of memory for its KV cache, sized for the maximum possible length. But most responses are far shorter than the maximum — so most of that reserved block sits empty and unusable by other requests. This is INTERNAL FRAGMENTATION: memory reserved but wasted. Studies found traditional serving wasted 60–80% of KV-cache memory this way.

The payoff is large. By allocating cache memory in small blocks on demand, PagedAttention nearly eliminates the wasted memory, letting the server fit FAR more concurrent requests in the same GPU memory — vLLM reported up to 24× higher throughput than prior systems, largely from this. More requests fit, so larger batches are possible, so throughput soars.

Batching — processing many requests together — is how we get throughput, because it lets us read the weights ONCE and use them for MANY requests (recall the decode bottleneck from Section 27.2). But naive batching wastes the GPU. Continuous batching, the other half of vLLM's magic, fixes this and is one of the most important throughput optimizations.

With STATIC (naive) batching, you collect a batch of requests, run them all together until ALL of them finish, then start the next batch. The problem: requests finish at different times — one needs 10 tokens, another needs 500. With static batching, the finished short requests sit idle, wasting their GPU slots, while the batch waits for the longest request. The GPU is underused, and new requests wait for the whole batch to clear.

Continuous batching (also called in-flight or dynamic batching) operates at the level of individual DECODE STEPS. After every single token-generation step, the scheduler checks: did any request finish? If so, remove it and immediately admit a waiting request into the freed slot. The batch is continuously reshuffled, so the GPU is always working on a full batch of ACTIVE requests — no idle slots waiting for stragglers.

Speculative decoding (Leviathan et al., 2023; Chen et al., 2023) is a clever technique that speeds up generation with NO loss in output quality — the output is provably identical to normal decoding. It directly attacks the decode bottleneck (one token per expensive forward pass) by generating SEVERAL tokens per pass of the big model. It feels like magic the first time you see it.

The insight: a small, fast 'draft' model can GUESS the next several tokens cheaply, and the big model can VERIFY all those guesses in a SINGLE forward pass (because verifying tokens in parallel is cheap — it is just one prefill-like pass). If the guesses are right, you got several tokens for the price of one big-model pass. If a guess is wrong, you discard it from that point and continue. Because the big model only ever ACCEPTS tokens it would have generated anyway, the output is identical to normal decoding.

Speculative decoding helps because many tokens are EASY to predict — common words, completions of obvious phrases, code boilerplate. The small draft model gets these right most of the time, so the big model verifies several at once. The speedup depends on the ACCEPTANCE RATE: how often the draft's guesses are correct. A good draft model (well-matched to the big one) achieves high acceptance and 2–3× speedups.

Beyond the big techniques, a collection of smaller but valuable optimizations round out a modern inference stack. Each targets a specific inefficiency, and together they add up to substantial gains.

Many requests share a common prefix — the same long system prompt, the same few-shot examples, the same document being asked about repeatedly. Prefix caching stores the KV cache for that shared prefix and REUSES it across all requests that share it, instead of recomputing the prefill every time. For applications with a large fixed system prompt, this can dramatically cut TTFT and cost. It builds directly on PagedAttention's block-sharing (Section 27.7).

A very long prompt makes prefill take a long time, which delays the decode of other requests sharing the batch (and spikes TTFT). Chunked prefill splits a long prefill into smaller chunks and INTERLEAVES them with the decode steps of other requests. This keeps decode latency smooth for everyone, rather than letting one giant prompt monopolize the GPU. It balances the compute-heavy prefill against the latency-sensitive decode.

One more inference topic every practitioner must understand: HOW the next token is chosen from the model's output distribution. The model outputs a probability for every possible next token; the DECODING STRATEGY decides which one to actually emit. This affects output quality, diversity, and even speed, and the knobs (temperature, top-k, top-p) are ones you will tune constantly.

Temperature is the most important sampling knob. It scales the logits before the softmax: a temperature below 1 SHARPENS the distribution (more confident, more deterministic), while above 1 FLATTENS it (more random, more diverse). Temperature 0 is equivalent to greedy decoding (always the top token). For factual tasks you want low temperature (precise, consistent); for creative writing you want higher (varied, surprising).

You will almost never implement these optimizations yourself. Instead, you use a SERVING ENGINE — software that bundles quantization, PagedAttention, continuous batching, speculative decoding, and the rest into a single high-performance system. Knowing the major engines and what each is for completes the practical picture.

Tying the chapter together, here is the journey of a request through a modern serving engine, touching the optimizations we covered:

Exercises 1–11 are pen-and-paper or derivations; 12–22 require code.

Further reading: “Efficient Memory Management for LLM Serving with PagedAttention” (Kwon et al., 2023, vLLM). “GPTQ” (Frantar et al., 2022) and “AWQ” (Lin et al., 2023) for quantization. “Fast Inference from Transformers via Speculative Decoding” (Leviathan et al., 2023) and “Accelerating LLM Decoding with Speculative Sampling” (Chen et al., 2023). “Medusa” (Cai et al., 2024) for multi-head speculation. “Orca” (Yu et al., 2022) for continuous batching. The vLLM, TensorRT-LLM, and llama.cpp documentation. “The Curious Case of Neural Text Degeneration” (Holtzman et al., 2019) for top-p sampling.

Next → Chapter 28: Tool Calling & Function Calling

You can now run a model fast and cheaply. But a model alone is limited to what is in its weights — it cannot look up today's weather, run a calculation reliably, query a database, or take actions in the world. Chapter 28 gives the model TOOLS: the ability to call functions, APIs, and external systems. We will see how a model is trained and prompted to decide WHEN to use a tool, format the call, and incorporate the result — the foundation of agents. The actor-to-actor message flow you saw in this chapter's diagrams becomes the agentic loop, where the model and its tools converse to accomplish tasks beyond what any single forward pass could.