Machine learning,
before the deep end.

A traditional program is a rule you write by hand: if income > X and debt < Y then approve. Machine learning flips that. You hand the computer examples (past loans, and whether each defaulted) and it searches for the rule itself. The promise is that a rule fit to data can be more accurate, and far less work, than a rule you guess.

Almost everything in this guide is supervised learning: each example comes with the right answer attached, and the model learns to reproduce it on new cases. Below is the whole idea in one picture. Each point is a past applicant, colored by outcome. Your job is to place one dividing line by moving the threshold. That is exactly what a classifier does, except it will find the line by optimizing a number instead of by eye.

Take one loan applicant: income, existing debt, and whether they defaulted. Income and debt go in, the default outcome comes out, and the whole applicant is just one row in a spreadsheet. Every example in a dataset is another row built the same way.

Every example is a row: some features (the inputs, x) and a label (the target, y). A model is a function with adjustable knobs. To fit it we need a single number that says how wrong it currently is: a loss function. Learning is nothing more than turning the knobs to make that number small.

For regression the workhorse is squared error. For a constant prediction c, the loss is the average of (c − y)². Squaring punishes big misses hard, which is why a single outlier drags the answer toward it. Its cousin, absolute error, grows only linearly and shrugs off outliers. Move the prediction line below and watch the two losses disagree: squared error is minimized at the mean of the data, absolute error at the median.

If you studied for an exam by memorizing last year's exact questions and answers, you would ace that exam and flunk this year's. Studying so what you learn holds up on new questions is the whole game, and it is exactly what splitting data into train and test sets is built to measure.

The goal is not to memorize the examples you were given. It is to do well on examples you have not seen. So we hold some data back. We train on one part and test on the other, and the test score is our honest estimate of future performance.

Below, one straight line is fit by least squares on the training points only. The test points are scored but never touched during fitting. Slide the split: with tiny training sets the line is unstable and test error is high and noisy; with more training data the fit settles and training and test error converge.

Picture every past home sale plotted as a dot, square footage on one axis, sale price on the other. You eyeball a line through the cloud that comes closest to all the points at once. That is linear regression, before any formula gets involved.

The simplest useful model draws a straight line: ŷ = w·x + b. Fit it by minimizing squared error and something remarkable happens: you do not need to search at all. Setting the derivatives to zero gives a formula, the ordinary least squares (OLS) solution, that lands on the best line in one shot.

Drag any point in the figure and the amber line re-solves instantly, because it is computing that formula on the current points, not animating a guess. The vertical stubs are residuals, the errors OLS is squaring and summing. Push the rightmost point up with the outlier slider and watch how far one bad measurement tilts the whole line: squared error has no defense against outliers.

Most models have no closed-form solution, so we search. Gradient descent is the universal method: compute the slope of the loss with respect to each knob, then step every knob a little way downhill. Repeat. For squared error on a line the gradient is exact and cheap, and you can watch the line crawl toward the same answer OLS found in one step.

Step it by hand or press run. The loss trace falls fast then flattens. Now push the learning rate η up: too small and it crawls, too large and it overshoots the valley and the loss explodes. That single knob is the whole art of training by gradient descent, here and in every neural network.

Say you want the chance a tumor is malignant given its size, not a flat yes or no, but a number like 73%. A straight line can shoot past 100% or dip below 0%, which is nonsense for a probability. Logistic regression fixes that by squashing the line through an S-shaped curve that always lands between 0 and 1.

To predict a category, not a number, we squash the linear score through the sigmoid so the output is a probability between 0 and 1. Fitting maximizes the probability the model assigns to the correct labels (equivalently, minimizes cross-entropy loss) by gradient descent. The result is still a straight decision boundary, the line where the probability crosses 50%.

Press run and watch the boundary swing into place as real gradient steps update the weights. The background is shaded by the model's predicted probability: confident where it is dark, unsure near the line. Despite the name, logistic regression is a classifier, and it is still the first model most practitioners try.

A more flexible model always fits the training data better. That is a trap. Below, real polynomials of rising degree are fit to a handful of training points drawn from a curved truth plus noise. Drag the degree up: training error falls all the way to zero as the curve threads every point. But test error traces a U: it drops as the model gains enough flexibility to capture the real shape, then climbs as the model starts memorizing the noise.

A model with huge weights swings wildly on a hunch: a tiny wiggle in one feature can send the prediction flying. Regularization puts it on a leash, still flexible enough to bend to real patterns, but it has to justify a big swing with strong evidence or get pulled back toward zero.

Instead of limiting flexibility by lowering the degree, keep the flexible model but penalize large weights. Add λ·(size of the weights) to the loss and the fit is pulled toward gentler curves. Ridge penalizes the squared weights (L2) and shrinks them smoothly; lasso penalizes absolute weights (L1) and drives some to exactly zero, selecting features.

A degree-9 polynomial is fit below. At λ = 0 it is the wild overfit from the last chapter. Raise λ and the fitted curve calms down, the total weight size collapses, and test error again traces a U with a minimum at a healthy amount of shrinkage. Same flexibility, tamed by a penalty.

A single train/test split wastes data and depends on luck: a different split gives a different score. k-fold cross-validation uses all of it. Chop the data into k equal folds. Train on k−1 of them, test on the one left out, and rotate so every fold is the test set exactly once. Average the k scores. That average is a far steadier estimate of true performance.

Step through the folds below. Each step holds out one fold, fits a degree-d polynomial on the rest, and records that fold's error. The bars accumulate; their mean is the CV score. Change the degree and re-run: cross-validation will prefer a moderate degree over a wildly flexible one, without you ever touching a final test set.

How much would more data help? A learning curve plots error against training-set size. It diagnoses your model at a glance. Slide the training size below. With few points the model fits them almost perfectly (low train error) while flailing on new data (high test error): a wide gap that screams overfitting. As data grows the two curves march toward each other and meet at the model's floor.

The laziest possible model does no fitting at all. To classify a new point, look at its k nearest neighbors in the training set and take a vote. That is the whole algorithm. Yet it draws boundaries of any shape, because it makes no assumption that the classes are linearly separable.

k controls smoothness. At k = 1 every training point owns its patch: training accuracy is a perfect 100%, but the boundary is jagged and it has memorized the noise. Raise k and the boundary smooths as votes average over a neighborhood; training accuracy drops but the region shading becomes reasonable. This is the bias-variance tradeoff again, now as one integer.

A decision tree asks a sequence of yes/no questions about the features, carving the space into axis-aligned rectangles and predicting the majority class in each. To choose each split it tries every threshold on every feature and keeps the one that most reduces impurity, measured by the Gini index (the chance of mislabeling a random point if you guessed by the box's class mix).

The tree below is grown for real to the depth you set. At depth 1 it is a single line. Add depth and it keeps slicing, driving training accuracy up and the number of leaves up, until each leaf is pure. Deep trees fit anything, which is exactly why they overfit, and why the next guide combines many shallow ones instead.

Among all lines that separate two classes, which is best? The support vector machine picks the one with the widest margin, the largest gap to the nearest points of either class. Those nearest points, the support vectors, are the only ones that matter; move any other and the boundary does not budge. We train it here with a real subgradient method (Pegasos) on the hinge loss.

But this data is a ring: no straight line can split it. The fix is the kernel trick. Map the points into a richer feature space (here, add x², y², xy) where they become linearly separable, find the max-margin plane there, and the boundary curves back in the original space. Toggle between the linear and lifted models and watch the accuracy jump.

Sometimes there are no labels, only points, and the question is whether they fall into natural groups. k-means answers it with a two-step loop, Lloyd's algorithm: (1) assign each point to the nearest of k centroids, (2) update each centroid to the mean of its points. Repeat. Each round can only lower the total within-cluster distance (the inertia), so it always converges.

Step the loop below and watch the centroids slide to the centers of mass while points recolor by allegiance. The inertia readout drops every round and then stops changing: that is convergence. Reset drops the centroids in new random spots; sometimes k-means lands in a worse local optimum, which is why in practice you run it several times.

Picture a scatter of points shaped like a tilted cigar, almost all of the spread running along its long axis. If you had to describe each point with a single number, you would measure how far along that long axis it falls. PCA finds that axis: it is the first principal component.

When features are correlated, the data really lives along a few directions. PCA finds them. It computes the covariance matrix of the (centered) data and takes its eigenvectors: the first, PC1, points along the direction of greatest variance; the second is perpendicular to it. Projecting onto the top few components compresses the data while keeping most of its spread.

The figure computes a real 2×2 eigen-decomposition on the point cloud. Turn the correlation up and the cloud stretches into a cigar; PC1 aligns with it and the variance explained by that single axis climbs from about 50% toward 100%. At that point one number nearly describes each point: two dimensions have collapsed to one.

Accuracy lies when classes are imbalanced: a model that always predicts "healthy" scores 99% on a disease that afflicts 1%. The honest breakdown is the confusion matrix, the four ways a binary prediction can land: true positives, false positives, true negatives, false negatives. From those come the two numbers that actually matter.

Each example below has a model score; the classes overlap. Slide the threshold: raise it and you demand more confidence before calling something positive, so precision rises while recall falls (you miss more). Lower it and the trade reverses. There is no free lunch, only the operating point you choose.

To judge a scorer across all thresholds at once, sweep the threshold from high to low and plot the true positive rate against the false positive rate. That path is the ROC curve. A perfect model hugs the top-left corner; random guessing is the diagonal. The area under it, the AUC, is a single threshold-free summary: the probability the model scores a random positive above a random negative.

The threshold slider moves the operating point along the fixed ROC curve. Notice the AUC does not change as you slide: it is a property of the scores themselves, not of where you cut. Choosing a threshold is choosing where on this curve to live.

Here is the whole toolbox on one dataset. The same training points fit four different classifiers from this guide; each is judged on the same held-out test set. Click through them and compare not just the boundaries but the numbers: the linear models draw a straight cut, the tree draws boxes, kNN and the kernel SVM bend around the shape. No model is best everywhere, which is the real lesson of classical ML: know the toolbox, and let held-out data choose.