2 Transformers

A transformer repeatedly updates a sequence of vector representations. At each layer, attention lets positions exchange information, while a feed-forward network transforms the features at each position. Residual connections preserve a path for information and gradients through the stack.

The original architecture used an encoder–decoder design for sequence transduction. Many autoregressive language models use a decoder-style stack with causal self-attention. These are related architectures, not identical diagrams. The original formulation is described in Attention Is All You Need.

Follow one sequence

1. Tokenization maps text to discrete token IDs. A token need not be a word.
2. Embedding maps each ID to a learned vector.
3. Position information lets the computation distinguish order. The method depends on the architecture.
4. Transformer blocks update the representations using attention and feed-forward sublayers.
5. An output projection produces a score, or logit, for each vocabulary item.
6. Softmax and decoding turn logits into a next-token choice.

For a decoder language model, the representation at a position must not depend on future tokens during next-token prediction. A causal mask enforces this restriction inside attention.

Start with attention

The attention mechanism is the first detailed chapter. It covers queries, keys, values, scaling, masking, and a small executable example. If the shapes feel unfamiliar, read linear algebra for language models first.

Keep architecture and implementation separate

An equation describes a computation. A production implementation chooses how to schedule it, where to store intermediate tensors, and which numerical precision to use. Two implementations can represent the same attention function while having very different memory use and execution time.

That distinction becomes especially useful when moving from an introductory attention matrix to caching and serving. The training and inference overview provides the next bridge.