Positional Encoding

You have built every component: linear layers, activations, normalization, dropout, residuals (Chapter 10), the autograd to train them (Chapter 11), and multi-head attention (Chapter 12). This chapter assembles them into the complete Transformer. We start with the bird's-eye view, then build each piece, then write the whole thing from scratch.

Every Transformer, regardless of variant, has the same three-stage structure: an input stage that turns tokens into vectors, a stack of identical processing blocks, and an output stage that turns vectors back into token predictions.

The middle stage — the stack of N identical blocks — does the heavy lifting. Each block refines the representation, mixing context via attention and processing it via the feed-forward network. The input and output stages are comparatively simple: a lookup table in, a linear projection out.

The input stage converts a sequence of integer token IDs into a sequence of vectors the network can process. This requires two things: a token embedding that maps each token to a learned vector, and a positional encoding that tells the network where each token sits in the sequence.

The token embedding is a lookup table: a matrix E of shape (V, d) where row i is the embedding of token i. Looking up a sequence of token IDs gives a sequence of d-dimensional vectors. This is exactly the embedding idea from Chapter 8, now learned end-to-end with the rest of the model.

Here is a subtle but critical fact: self-attention is permutation-equivariant. If you shuffle the input tokens, the outputs shuffle the same way — attention has no inherent notion of order. 'dog bites man' and 'man bites dog' would be processed identically. Position must be added explicitly.

[Missing Component: attnNote]

The original Transformer used fixed sinusoids of geometrically increasing wavelength. Each dimension of the encoding oscillates at a different frequency, giving every position a unique fingerprint and — crucially — letting the model attend to relative positions via linear combinations.

The residual connections introduced in Chapter 10 are more than an optimization trick in the Transformer — they form the residual stream, a powerful conceptual lens (Elhage et al., 2021) for understanding how the whole network communicates. Every block reads from and writes to this shared stream.

Each sublayer (attention or FFN) does not replace the representation — it adds to it. The running sum x flows unchanged through every residual connection; each sublayer reads the current x, computes an update, and adds it back. The stream is a shared workspace that all blocks contribute to.

Where you place the LayerNorm relative to the residual add is one of the most consequential design choices. The original Transformer used Post-LN; nearly all modern LLMs use Pre-LN, which keeps the residual stream clean and stabilizes training at depth.

After attention mixes information across positions, the feed-forward network (FFN) processes each position independently. It is deceptively simple — two linear layers with a nonlinearity — yet it holds roughly two-thirds of a Transformer's parameters and is increasingly understood to store much of the model's factual knowledge.

The FFN expands the representation to a wider hidden dimension (typically 4× the model dimension), applies a nonlinearity, then projects back. The expansion gives the network room to compute rich nonlinear features per position; the projection returns to the residual-stream dimension so the output can be added back.

We now have every piece. The Transformer block combines a multi-head attention sublayer and a feed-forward sublayer, each wrapped in a Pre-LN residual connection. This is the unit that gets stacked N times.

The same block can be arranged in three ways, giving three model families. The difference comes down to two choices: is the self-attention masked (causal) or not, and is there a separate encoder feeding the decoder via cross-attention?

The largest and most capable language models — GPT, LLaMA, Claude, Gemini — are decoder-only. This was not obvious in 2018, when encoder-only (BERT) and encoder-decoder (T5) were equally prominent. The decoder-only design won for LLMs because of its simplicity and the power of the next-token objective.

After the last block, the residual stream holds a rich representation of each position. The output stage converts this back into a probability distribution over the vocabulary: a final LayerNorm, then a linear projection (the unembedding or 'LM head') from model dimension d to vocabulary size V, then softmax.

A common trick: tie the unembedding matrix W_U to the token embedding matrix E (using Eᵀ as the unembedding). This saves V×d parameters — 38 million for GPT-2 — and often improves quality, since the same vocabulary geometry serves both reading and writing tokens.

We now assemble everything into a complete, working decoder-only Transformer — a miniature GPT. This is the culmination of Part III: a model you understand line by line, from token IDs to output logits.

A trained decoder-only Transformer generates text autoregressively: it predicts the next token, appends it to the sequence, and repeats. The causal mask ensures each prediction depends only on previous tokens, so this loop is consistent with how the model was trained.

Every real Transformer is a specific configuration of the architecture you just built. Here is how the major models fill in the blanks — the same skeleton, different sizes and refinements.

Notice the pattern: the architecture is stable, but two things change over time. Scale grows relentlessly (12 → 96 layers; 768 → 12288 dimensions), and a handful of refinements accumulate (RoPE replaces learned positions; RMSNorm replaces LayerNorm; SwiGLU replaces GELU; GQA reduces KV memory). The core — stacked Pre-LN blocks of attention and FFN — is unchanged from 2017.

A Transformer can have a subtle bug and still train — just worse — making errors hard to spot. Here are the most common implementation pitfalls and how to catch them.

Exercises 1–10 are pen-and-paper or derivations; 11–20 require code.

Further reading: “Attention Is All You Need” (Vaswani et al., 2017) — the original architecture. “Language Models are Unsupervised Multitask Learners” (Radford et al., 2019) for GPT-2's decoder-only design. “RoFormer” (Su et al., 2021) for RoPE. “LLaMA” (Touvron et al., 2023) for the modern recipe. Andrej Karpathy's nanoGPT and his 'Let's build GPT' video — the best hands-on companion to this chapter. “A Mathematical Framework for Transformer Circuits” (Elhage et al., 2021) for the residual-stream view.

Next → Chapter 14: Tokenization

You have built a complete Transformer that maps token IDs to predictions — but where do token IDs come from? Chapter 14 fills the one remaining gap: how raw text becomes the integer sequences the model consumes. We will build Byte-Pair Encoding from scratch, compare it to WordPiece and Unigram, explore the surprising ways tokenization shapes model behaviour (arithmetic, multilingual fairness, the 'SolidGoldMagikarp' glitch tokens), and understand why tokenization is both essential and a persistent source of model quirks.