An LLM is a next-token predictor.
Everything else is plumbing.

The core task: P(next token | everything so far)

Strip away the mystique and a language model computes one thing: given a sequence of tokens, a probability distribution over which token comes next. That’s it. To write a sentence, you sample one token from that distribution, append it to the input, and ask again: autoregression. The model never plans the whole sentence; it just keeps answering “what’s next?” extremely well.

The forward pass, end to end

Every stage in this guide is one box in the pipeline below, the same pipeline drawn in the left rail. Text comes in, integers flow through a stack of identical Transformer blocks, and a final projection turns the result into a score for every possible next token.

Mental model to hold onto: tokens travel down a wide “conveyor belt” (the residual stream). Each block reads the belt, computes a small update, and adds it back. Nothing gets overwritten: information accumulates layer by layer until the last position holds a rich summary of “what should come next.”

Text isn’t numbers. Fix that first.

Neural networks eat numbers, not characters. Tokenization is the lookup table that maps pieces of text to integer IDs and back. The simplest scheme is character-level: every distinct character gets an ID. Real models use Byte-Pair Encoding (BPE), which learns a vocabulary of frequent chunks, so " tokenization" might become two tokens, not twelve characters.

# Character-level tokenizer: the simplest thing that works.
# (Real models swap this for BPE, e.g. tiktoken, but the interface is identical:
#  text -> list[int] -> text.)

text = open("input.txt").read()
chars = sorted(set(text))          # the vocabulary
vocab_size = len(chars)

stoi = {ch: i for i, ch in enumerate(chars)}   # string  -> int
itos = {i: ch for i, ch in enumerate(chars)}   # int     -> string

encode = lambda s: [stoi[c] for c in s]        # "hi" -> [40, 41]
decode = lambda ids: "".join(itos[i] for i in ids)

print(encode("hello"))             # [...]
print(decode(encode("hello")))     # "hello"

The only thing the model ever sees is integers. Everything downstream (embeddings, attention, the final prediction) operates on token IDs. Vocabulary size becomes a key dimension: it’s the width of the model’s output layer.

Turn IDs into meaning, and add a sense of place

Token ID 42 means nothing on its own; 42 isn’t “bigger” than 41. So each ID indexes into a learned embedding table: a big matrix where row i is a trainable vector for token i. During training these vectors arrange themselves so that tokens used in similar ways land near each other in space. Meaning becomes geometry.

But attention (next section) treats its input as an unordered set: it has no built-in notion of sequence. "dog bites man" and "man bites dog" would look identical. The fix: add a positional encoding to each token so position is baked into the vector itself.

import torch, torch.nn as nn

class Embeddings(nn.Module):
    def __init__(self, vocab_size, block_size, n_embd):
        super().__init__()
        self.tok_emb = nn.Embedding(vocab_size, n_embd)   # one row per token
        self.pos_emb = nn.Embedding(block_size, n_embd)   # one row per position
        # block_size = max context length the model can ever see at once.

    def forward(self, idx):                 # idx: (B, T) integer token ids
        B, T = idx.shape
        pos = torch.arange(T, device=idx.device)          # 0,1,2,...,T-1
        x = self.tok_emb(idx) + self.pos_emb(pos)          # (B,T,C) + (T,C) broadcasts
        return x                                           # meaning + position, fused

Let every token gather what it needs from the others

This is the mechanism that made Transformers work. For each token, attention asks: which earlier tokens are relevant to me, and what should I pull from them? It answers with three learned projections of every token:

• Query (Q): what this token is looking for.
• Key (K): what each token advertises about itself.
• Value (V): the actual information a token will hand over if attended to.

Compare a token’s query against every key with a dot product → a relevance score. Softmax those scores into weights that sum to 1. Then take a weighted average of the values. Each token walks away with a custom blend of information from the tokens it cared about.

The formula, dissected

Real Transformers run several attention operations in parallel: multi-head attention. Each head gets its own Q/K/V projections and can specialize: one head tracks subject–verb agreement, another links pronouns to their referents, another watches punctuation. Their outputs are concatenated and mixed.

import math, torch, torch.nn as nn
from torch.nn import functional as F

class CausalSelfAttention(nn.Module):
    def __init__(self, cfg):
        super().__init__()
        assert cfg.n_embd % cfg.n_head == 0
        # one big linear produces Q, K, V for every head at once (efficiency):
        self.c_attn = nn.Linear(cfg.n_embd, 3 * cfg.n_embd)
        self.c_proj = nn.Linear(cfg.n_embd, cfg.n_embd)   # mix heads back together
        self.n_head, self.n_embd = cfg.n_head, cfg.n_embd
        # lower-triangular mask: position i may attend to j only if j <= i
        self.register_buffer("mask",
            torch.tril(torch.ones(cfg.block_size, cfg.block_size))
                 .view(1, 1, cfg.block_size, cfg.block_size))

    def forward(self, x):                          # x: (B, T, C)
        B, T, C = x.shape
        q, k, v = self.c_attn(x).split(self.n_embd, dim=2)
        # split channels into heads -> (B, nh, T, hs)
        hs = C // self.n_head
        q = q.view(B, T, self.n_head, hs).transpose(1, 2)
        k = k.view(B, T, self.n_head, hs).transpose(1, 2)
        v = v.view(B, T, self.n_head, hs).transpose(1, 2)

        att = (q @ k.transpose(-2, -1)) / math.sqrt(hs)        # (B, nh, T, T) relevance
        att = att.masked_fill(self.mask[:, :, :T, :T] == 0, float('-inf'))  # block future
        att = F.softmax(att, dim=-1)                           # weights sum to 1 per row
        y = att @ v                                            # weighted blend of values

        y = y.transpose(1, 2).contiguous().view(B, T, C)       # re-merge heads
        return self.c_proj(y)

Communication, then computation: repeated

A Transformer is just one block stacked N times. Each block does two things in sequence. Attention lets tokens talk to each other and exchange information (communication). Then a small MLP processes each token independently, “thinking” about what it just gathered (computation). Wrapping both: residual connections and layer normalization.

The math inside the wrapper

class MLP(nn.Module):
    """Per-token computation. Expand 4x, apply nonlinearity, project back."""
    def __init__(self, cfg):
        super().__init__()
        self.fc   = nn.Linear(cfg.n_embd, 4 * cfg.n_embd)
        self.proj = nn.Linear(4 * cfg.n_embd, cfg.n_embd)
        self.act  = nn.GELU()
    def forward(self, x):
        return self.proj(self.act(self.fc(x)))

class Block(nn.Module):
    """Pre-norm: normalize -> sublayer -> add to residual stream."""
    def __init__(self, cfg):
        super().__init__()
        self.ln1 = nn.LayerNorm(cfg.n_embd)
        self.attn = CausalSelfAttention(cfg)
        self.ln2 = nn.LayerNorm(cfg.n_embd)
        self.mlp = MLP(cfg)
    def forward(self, x):
        x = x + self.attn(self.ln1(x))   # communicate, then add
        x = x + self.mlp(self.ln2(x))    # compute,     then add
        return x

Stack the blocks, then read out a prediction

Now wire it all together. Embed the tokens, run them through N identical blocks, apply one final LayerNorm, then project the result to logits: one raw score per vocabulary entry, for every position. A softmax over those scores gives the next-token distribution. That projection is the lm_head, and a classic trick called weight tying reuses the embedding table as the output matrix (the map “word → vector” run in reverse), saving parameters and improving quality.

The math of the read-out

from dataclasses import dataclass

@dataclass
class GPTConfig:
    vocab_size: int = 65       # from your tokenizer
    block_size: int = 256      # max context length
    n_layer: int = 6
    n_head:  int = 6
    n_embd:  int = 384

class GPT(nn.Module):
    def __init__(self, cfg):
        super().__init__()
        self.cfg = cfg
        self.tok_emb = nn.Embedding(cfg.vocab_size, cfg.n_embd)
        self.pos_emb = nn.Embedding(cfg.block_size, cfg.n_embd)
        self.blocks  = nn.ModuleList([Block(cfg) for _ in range(cfg.n_layer)])
        self.ln_f    = nn.LayerNorm(cfg.n_embd)
        self.lm_head = nn.Linear(cfg.n_embd, cfg.vocab_size, bias=False)
        self.lm_head.weight = self.tok_emb.weight     # weight tying

    def forward(self, idx, targets=None):
        B, T = idx.shape
        pos = torch.arange(T, device=idx.device)
        x = self.tok_emb(idx) + self.pos_emb(pos)     # (B,T,C)
        for block in self.blocks:
            x = block(x)
        x = self.ln_f(x)
        logits = self.lm_head(x)                      # (B, T, vocab_size)

        loss = None
        if targets is not None:
            # compare prediction at every position to the *actual* next token
            loss = F.cross_entropy(logits.view(-1, logits.size(-1)),
                                   targets.view(-1))
        return logits, loss

That’s a complete, trainable GPT in ~60 lines of model code. Everything from here is about fitting it (training) and using it (generation).

Measure surprise, then nudge the weights

Training data is free: any text is a stack of next-token questions with the answers attached. Take a chunk of tokens; the inputs are positions 0…T-1 and the targets are the same chunk shifted by one. At every position the model predicts the next token, and we already know what it actually was.

The math of learning

import torch

def get_batch(data, block_size, batch_size, device):
    # data is a 1-D tensor of token ids (the whole corpus)
    ix = torch.randint(len(data) - block_size, (batch_size,))
    x = torch.stack([data[i     : i+block_size]   for i in ix])  # inputs
    y = torch.stack([data[i+1   : i+block_size+1] for i in ix])  # targets = shift by 1
    return x.to(device), y.to(device)

model = GPT(GPTConfig()).to(device)
opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1)

for step in range(max_steps):
    xb, yb = get_batch(train_data, cfg.block_size, batch_size=64, device=device)
    logits, loss = model(xb, yb)        # forward: prediction + surprise

    opt.zero_grad(set_to_none=True)
    loss.backward()                     # backprop: gradient for every weight
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)  # clip spikes
    opt.step()                          # AdamW nudges the weights

    if step % 200 == 0:
        print(f"step {step}: loss {loss.item():.3f}")

Sanity check: with vocab size V, a totally untrained model is just guessing, so initial loss ≈ ln(V). Watching it fall below that on the first steps is your “it’s alive” signal.

From a distribution to actual words

To generate, run the model, take the logits at the last position, turn them into probabilities, and pick a token. Append it to the input and repeat. How you pick is the difference between a boring model and a good one. The knobs:

• Temperature: divide logits by T before softmax. T < 1 sharpens the distribution (safe, repetitive); T > 1 flattens it (creative, riskier); T → 0 is pure greedy/argmax.
• Top-k: keep only the k highest-probability tokens, renormalize, sample. Cuts off the long tail of nonsense.
• Top-p (nucleus): keep the smallest set of tokens whose probability sums to p. Adapts its cutoff to how confident the model is.

@torch.no_grad()
def generate(model, idx, max_new_tokens, temperature=1.0, top_k=None):
    model.eval()
    for _ in range(max_new_tokens):
        idx_cond = idx[:, -model.cfg.block_size:]    # never exceed the context window
        logits, _ = model(idx_cond)
        logits = logits[:, -1, :] / temperature      # only the last position matters

        if top_k is not None:                        # optional top-k filtering
            v, _ = torch.topk(logits, top_k)
            logits[logits < v[:, [-1]]] = -float('inf')

        probs = F.softmax(logits, dim=-1)
        idx_next = torch.multinomial(probs, num_samples=1)   # sample one token
        idx = torch.cat((idx, idx_next), dim=1)              # append, then loop
    return idx

# usage:
context = torch.tensor([encode("ROMEO:")], device=device)
out = generate(model, context, max_new_tokens=300, temperature=0.8, top_k=40)
print(decode(out[0].tolist()))

Speed note: naively re-running the whole sequence every step is wasteful, since past tokens’ keys and values never change. Production models cache them (the KV cache) so each new token is cheap. Same math, far less compute.

What separates your nanoGPT from a frontier model

You now have the complete skeleton. A frontier model is the same skeleton: bigger, fed more, aligned, and engineered for scale. The conceptual additions worth knowing next, roughly in order of leverage:

Where to go to actually run this

• Karpathy’s nanoGPT (MIT License): the canonical minimal, trainable GPT. This guide’s model code in §5 and §6 follows its structure and naming (c_attn, c_proj, block_size), and borrows its “communication, then computation” block framing directly. Pair it with his “Let’s build GPT from scratch” video.
• “Attention Is All You Need” (2017): the original Transformer paper. Now readable, given everything above.
• The Illustrated Transformer (Jay Alammar): more diagrams if you want another visual pass on attention.
• Build it on tiny Shakespeare first. ~1MB of text, char-level, the config in §6 trains on a single GPU in minutes and produces recognizable (if nonsensical) Shakespeare. That moment is the whole point.