0 Building an LLM from scratch

A language model assigns probabilities to sequences of tokens. A useful way to study one is to follow a single question: how does a sequence of discrete symbols become a prediction about what comes next?

For tokens $x_1,\ldots,x_T$, the chain rule gives

$$p_\theta(x_1,\ldots,x_T) = \prod_{t=1}^{T}p_\theta(x_t\mid x_1,\ldots,x_{t-1}). \tag{0.1}$$(0.1)

This factorization is the starting point for autoregressive language modeling. The model represents each conditional distribution; a decoding procedure turns those distributions into a generated sequence. Training, architecture, and inference answer different parts of this problem.

A path through the subject

1. Learn the language of representations

Vectors, matrices, and inner products let us describe a model without relying on a particular library. Begin with linear algebra for language models. Keep track of shapes: many apparently difficult equations become ordinary matrix multiplication once the dimensions are explicit.

2. Follow information through a transformer

Read the transformer overview, then work through the attention mechanism. Attention connects a token’s representation to other positions in its context. A complete transformer also needs position information, nonlinear transformations, normalization, and residual connections.

3. Separate learning from generation

Training and inference explains what changes when parameters are learned and what changes when a fixed model generates text. This distinction makes later topics such as fine-tuning, caching, and quantization easier to place.

4. Put the model inside a system

A model can propose an action, but software must decide how that proposal affects the world. Continue with agentic systems and infrastructure to connect modeling choices with execution, measurement, and reliability.

How to read these chapters

The chapters use small examples, explicit assumptions, and references to primary sources. Equations explain the computation; code makes the assumptions executable. Neither should be treated as a substitute for the other.

You can read in sequence or follow internal links. When an idea needs more detail, it can grow into its own page without changing the surrounding chapters.

Start with one concrete object

When a new concept feels abstract, choose a short sequence and write down the shape of every tensor. Then ask which axis means “position,” which means “feature,” and which operation mixes them.