From pure noise
to a picture.

why generation is hard

A 512×512 RGB image is ~786,000 numbers. Almost every possible combination of those numbers is static. The images that look like something (faces, dogs, mountains) sit on a vanishingly thin manifold inside that enormous space.

"Generating an image" means sampling a new point that lands on that manifold. We never get an equation for it. Instead we collect millions of real images and train a model to learn the shape of the data, well enough to invent new points that belong.

Older approaches attacked this directly. GANs pit a generator against a critic (powerful but unstable, prone to mode collapse). VAEs learn a smooth compressed code (stable but blurry). Diffusion won the image race because it turns one impossibly hard jump into thousands of trivially easy steps.

the one idea everything rests on

Drop ink in water. It diffuses into a uniform cloud, and you can write down exactly how. Running that backwards, reassembling the drop, is the hard part. Diffusion models exploit an asymmetry: the destruction is free and known; we only pay to learn the reversal.

the forward process, precisely

At each step we mix in a little Gaussian noise. How much is set by a fixed noise schedule β₁…β_T (tiny at first, larger later). One step:

Doing 1000 steps to build one training example would be painfully slow. The key fact: a sum of Gaussians is still Gaussian, so we can jump to any timestep t in one shot. Define αₜ = 1−βₜ and the cumulative product ᾱₜ = α₁·α₂···αₜ. Then:

That single line is the entire data pipeline. ᾱₜ is just "how much original image survives": it slides from ≈1 (clean) to ≈0 (pure noise). It's exactly the blend you were dragging in the hero.

import torch

def cosine_schedule(T, s=0.008):
    # smoother than linear; preserves detail at low t (Nichol & Dhariwal 2021)
    t = torch.linspace(0, T, T+1)
    f = torch.cos(((t / T + s) / (1 + s)) * torch.pi / 2) ** 2
    abar = f / f[0]
    betas = (1 - abar[1:] / abar[:-1]).clamp(1e-5, 0.999)
    return betas

betas   = cosine_schedule(1000)
alphas  = 1. - betas
abar    = torch.cumprod(alphas, dim=0)         # ᾱₜ for every t

def q_sample(x0, t, eps):
    # jump straight to noise level t: the whole training pipeline
    a  = abar[t].sqrt().view(-1,1,1,1)
    am = (1 - abar[t]).sqrt().view(-1,1,1,1)
    return a * x0 + am * eps

the objective, and why it's so simple

You might expect a complicated objective. Instead, we hand the network a noisy image xₜ and its timestep t, and ask it to answer one question: what noise did I add? Predict ε. That's it.

Mean-squared error between the true noise and the network's guess. If it can name the noise, subtracting a slice of it nudges the image toward the clean manifold.

And the training loop is shockingly short: sample an image, sample a random timestep, noise it, predict, backprop:

for x0 in dataloader:                 # x0: a batch of real images
    t   = torch.randint(0, T, (x0.size(0),))   # random noise level per image
    eps = torch.randn_like(x0)                  # the noise we will hide
    xt  = q_sample(x0, t, eps)               # noisy version (closed form)

    pred = model(xt, t)                          # εθ: guess the noise
    loss = F.mse_loss(pred, eps)             # that's the whole objective

    loss.backward(); opt.step(); opt.zero_grad()

Every difference between this toy and Stable Diffusion is about scale and control, not a different idea. The objective above never really changes.

the body of the denoiser

The denoiser must output a value per pixel, yet to denoise sensibly it needs global context (is this a cat's ear or a cloud?). The U-Net resolves this tension: an encoder that shrinks the image while widening understanding, a decoder that rebuilds full resolution, and skip connections that hand fine detail straight across so it isn't lost in compression.

How the network knows how noisy the input is

The timestep t is a single integer, but feeding a raw number in works poorly. Instead we expand it into a sinusoidal embedding: a vector of sines and cosines at many frequencies, the same trick transformers use for word positions.

class ResBlock(nn.Module):
    def __init__(self, c_in, c_out, t_dim):
        super().__init__()
        self.norm1 = nn.GroupNorm(8, c_in)
        self.conv1 = nn.Conv2d(c_in, c_out, 3, padding=1)
        self.temb  = nn.Linear(t_dim, c_out)          # inject timestep
        self.norm2 = nn.GroupNorm(8, c_out)
        self.conv2 = nn.Conv2d(c_out, c_out, 3, padding=1)
        self.skip  = nn.Conv2d(c_in, c_out, 1) if c_in!=c_out else nn.Identity()

    def forward(self, x, t):
        h = self.conv1(F.silu(self.norm1(x)))
        h = h + self.temb(t)[:, :, None, None]    # broadcast t over H,W
        h = self.conv2(F.silu(self.norm2(h)))
        return h + self.skip(x)

turning a denoiser into a generator

To generate, start from pure noise x_T and apply the trained denoiser repeatedly. At each step: predict the noise, remove a calibrated slice of it, and add back a touch of fresh randomness (this keeps outputs diverse rather than collapsing to one blurry average).

@torch.no_grad()
def ddpm_sample(model, shape):
    x = torch.randn(shape)                      # start: pure noise x_T
    for t in reversed(range(T)):              # walk T → 0
        eps = model(x, torch.full((shape[0],), t))
        a, ab, b = alphas[t], abar[t], betas[t]
        mean = (x - b/(1-ab).sqrt() * eps) / a.sqrt()
        x = mean + (b.sqrt() * torch.randn_like(x) if t > 0 else 0)
    return x                                  # a fresh sample on the manifold

Below is that exact loop running live. For a 2-D point cloud the true score is computable in closed form, so this is the real reverse process (the same math as sample.py), not a canned animation. Every run starts from fresh noise and lands somewhere new on the gold manifold.

from random images to "an astronaut on a horse"

So far the model generates something from the data, but not what you asked for. Control means feeding a condition c alongside the noisy image so the denoiser becomes εθ(xₜ, t, c). For text, three pieces click together:

1 · Encode the words

A frozen text encoder (CLIP, or T5 in newer models) turns your prompt into a sequence of vectors: a numerical summary of meaning the U-Net can read.

2 · Cross-attention injects meaning

Inside the U-Net's attention blocks, the image features form the queries while the text vectors supply keys and values. Each image region asks "which words am I responsible for?" and pulls in their guidance. This is the literal bridge from language to pixels.

3 · Classifier-free guidance: the "prompt strength" dial

During training, drop the text ~10% of the time so the same network can run conditioned and unconditioned. At sampling, compute both and push away from the generic toward the prompted:

the trick that put it on consumer GPUs

Running diffusion on full 512×512 pixels is brutally expensive. Stable Diffusion's key move: compress first. A pretrained autoencoder (VAE) squeezes the image ~8× per side into a small latent tensor (e.g. 64×64×4 instead of 512×512×3, roughly a 48× reduction in elements). Do all the diffusion there, then decode once at the end.

Stack the pieces and you have the real system: VAE (compress/decompress) + U-Net (denoise latents) + text encoder (condition) + scheduler (the sampling loop). Nothing in the earlier sections changed; it all just moved into a smaller space.

adding a time axis

A video is a stack of frames. Denoise each one independently and you get flicker: every frame invents its own details. The fix is to let frames talk to each other so motion is smooth and objects stay the same object.

The cutting edge (Sora-class systems) drops the U-Net for a Diffusion Transformer over spacetime patches of a 3D latent, but conceptually it's still §02's idea: denoise a compressed representation, now with time folded in.

where the field is going + your path

Two shifts define modern systems (SD3, Flux, Sora-class). Worth knowing so you're learning the current thing, not the 2021 thing:

But you learn this by climbing a ladder, smallest rung first. Each rung is runnable in an afternoon to a weekend and proves you understood the one before: