Bias, variance, over/under-fitting

Background

A fundamental framework from statistics (Geman et al., 1992) for diagnosing model behavior
Explains why making a model more complex helps on training data but can hurt on test data
Critical for choosing model capacity, regularization strength, and training set size

The Concept

Every prediction error breaks into three parts: bias (the model is systematically wrong), variance (the model is too sensitive to which training examples it saw), and irreducible noise. Reducing one often increases the other - the bias-variance trade-off.

How it Links

Explains overfitting and underfitting seen in every ML algorithm
Motivates regularization (ridge, dropout, early stopping)
Modern double-descent shows deep nets can interpolate training data and still generalize

Concrete Example

A ruler drawn between two clusters of points (high bias, low variance) vs a wobbly curve that threads through every training point (low bias, high variance). The sweet spot generalizes best to new points.

Prerequisites

Math you need (probability + statistics), The ML workflow

Learning outcomes

Decompose expected squared error into bias-squared, variance, and irreducible noise.
Diagnose under-fitting vs over-fitting from learning curves alone.
Explain regularisation as a deliberate trade of variance for bias.
State the double-descent phenomenon and explain why it does not contradict the classical view.

TL;DR

Expected squared error of an estimator decomposes into bias-squared plus variance plus irreducible noise. Bias is how wrong your average model is; variance is how much your model wobbles when you change the data. Regularisation trades variance for bias on purpose. Double descent extends, rather than contradicts, this picture in heavily over-parameterised models.

Intuition

One-sentence definitions

Bias = how wrong your average model is, no matter how much data you give it.

Variance = how much your model changes when you change the training set.

Irreducible noise = the part of the target that no model can predict because the universe is stochastic.

A high-bias model is rigid: it cannot match the data even in expectation (a linear model trying to fit a sine wave). A high-variance model is twitchy: it fits each training set so closely that two different training sets yield wildly different models (a 20-degree polynomial on 50 points).

The classical view, due to Geman, Bienenstock, and Doursat [2], is that increasing model capacity decreases bias and increases variance, with a sweet spot somewhere in the middle.

The decomposition

Derivation of the bias-variance decomposition

Setup

Suppose the true relationship between input $x$ and target $y$ is

y = f(x) + \varepsilon, \qquad \mathbb{E}[\varepsilon] = 0, \quad \mathrm{Var}(\varepsilon) = \sigma^{2}.

We fit a model $\hat{f}$ on a random training set $\mathcal{D}$ , so $\hat{f}$ is itself a random variable depending on $\mathcal{D}$ . Fix a query point $x$ and compute the expected squared error

\mathrm{Err}(x) = \mathbb{E}_{\mathcal{D}, \varepsilon}\!\left[ (y - \hat{f}(x))^{2} \right].

Decomposition

Add and subtract $\mathbb{E}_{\mathcal{D}}[\hat{f}(x)]$ :

y - \hat{f}(x) = \underbrace{\bigl(f(x) - \mathbb{E}_{\mathcal{D}}[\hat{f}(x)]\bigr)}_{\text{bias}} + \underbrace{\bigl(\mathbb{E}_{\mathcal{D}}[\hat{f}(x)] - \hat{f}(x)\bigr)}_{-\text{variance term}} + \underbrace{\varepsilon}_{\text{noise}}.

Squaring and taking expectation, the cross terms vanish because $\varepsilon$ is independent of $\mathcal{D}$ and $\mathbb{E}_{\mathcal{D}}\!\left[\mathbb{E}_{\mathcal{D}}[\hat{f}(x)] - \hat{f}(x)\right] = 0$ by definition. We get

\mathrm{Err}(x) = \underbrace{\bigl(f(x) - \mathbb{E}_{\mathcal{D}}[\hat{f}(x)]\bigr)^{2}}_{\mathrm{Bias}^{2}} + \underbrace{\mathbb{E}_{\mathcal{D}}\!\left[\bigl(\hat{f}(x) - \mathbb{E}_{\mathcal{D}}[\hat{f}(x)]\bigr)^{2}\right]}_{\mathrm{Variance}} + \underbrace{\sigma^{2}}_{\text{irreducible}}.

This is the canonical bias-variance decomposition for squared error [1].

Consequences

The error has a non-negative floor $\sigma^{2}$ that no model can break.
Bias and variance can be traded against each other but not driven simultaneously to zero with finite data.
The decomposition is exact for squared error and holds in expectation, not for any particular fit.

The decomposition does not generalise cleanly to 0/1 loss or cross-entropy; for those, analogous decompositions exist (Domingos 2000) but require more care. Bishop [5] works through the squared-error case in chapter 3 in full.

Diagnosing the regime

Symptom	High bias (under-fit)	High variance (over-fit)
Training error	High.	Very low (near zero).
Validation error	High, close to training error.	Much higher than training error.
Effect of more data	Small or none. Bias is structural.	Large. Variance shrinks with N.
Effect of more capacity	Helps; reduces bias.	Hurts; increases variance further.
Typical fix	Bigger model; richer features; remove regularisation; train longer.	Smaller model; more data; add regularisation; early stopping; data augmentation.
Learning-curve shape	Two flat lines high up, close together.	Training curve drops to zero; validation curve plateaus or rebounds upward.

Interactive: polynomial fit demo

The cleanest illustration of the bias-variance trade-off is fitting polynomials of varying degree to a noisy sine. Low-degree polynomials are biased; high-degree ones overfit.

Polynomial fit — bias vs variance

Degree3train MSE = 0.0578 test MSE = 0.0518

Dashed gray = sin(2πx); blue = polynomial fit; green dots = noisy training points.

Interactive Demo

Explore the bias-variance trade-off through two linked visualizations: an error curve showing how training and test error change with model complexity, and a fitted curve view showing how a polynomial of the selected degree fits the noisy data.

Bias-Variance Explorer

Adjust the polynomial degree to see how model complexity affects training vs test error.

Model complexity (degree)4Good fit

Training error: 1.040

Test error: 0.280

Error vs Model Complexity

Data points and fitted curve

--- true sin(x) --- fitted polynomial ● data points

What to try: Drag the complexity slider from 1 to 10. At degree 1 the line cannot follow the curve (high bias). At degree 9-10 the curve threads through every noise point (high variance). The sweet spot is around degree 3-5 where test error is lowest.

Regularisation

Regularisation is a deliberate trade

Regularisation injects bias into the model on purpose so that variance drops further than bias rises and the net error decreases. Concrete examples:

L2 (ridge): quadratic penalty on weights. Equivalent to a Gaussian prior. Shrinks all weights toward zero.
L1 (Lasso): linear penalty. Equivalent to a Laplace prior. Sets many weights to exactly zero (sparsity).
Early stopping: halt training before the loss bottoms out on training. Empirically equivalent to L2 with a particular schedule.
Dropout: randomly zero out activations during training. Effectively averages over an exponential ensemble of subnetworks.
Data augmentation: synthetic perturbations of training inputs. A prior over invariances.

Each of these increases bias (the average prediction is now smoother / sparser / averaged) and decreases variance. The optimum is found by cross-validation or a held-out set.

Double descent

For decades the textbook risk curve looked U-shaped: training error monotonically decreasing with capacity, test error first decreasing then rising as variance dominates. Belkin, Hsu, Ma, and Mandal [3] and then Nakkiran et al. [4] showed that the picture is incomplete. Beyond the interpolation threshold (where the model fits the training data exactly), test error often decreases again.

Double descent in one paragraph

The classical bias-variance curve assumes you stop at “good fit on training”. If you keep growing the model past the point where it has zero training error, you enter an over-parameterised regime where many interpolating solutions exist; SGD-style optimisers tend to pick the smoothest of these (an implicit regularisation effect), and test error drops a second time. This is double descent [3][4].

The classical view was right within its assumptions; modern deep learning operates in a regime the classical view was never meant to describe.

Why double descent does not contradict the classical view

The bias-variance decomposition still holds; what changes is the shape of the variance curve. In the over-parameterised regime, variance drops because there are many interpolating solutions and the optimiser’s implicit bias picks ones that generalise. The decomposition is still

\mathrm{Err} = \mathrm{Bias}^{2} + \mathrm{Variance} + \sigma^{2},

but variance is no longer monotonically increasing in capacity once you cross the interpolation threshold. The classical picture is the left half of a wider curve.

Practical consequence: in deep learning, “more parameters than samples” is not automatically over-fitting. Whether it generalises depends on architecture, optimiser, and regularisation - not on parameter count alone.

Pop quiz

Check yourself

Score: 0 / 5(0 answered)

1. A model has 0.05 training error and 0.32 validation error. Which regime is it in?
2. A model has 0.30 training error and 0.32 validation error. Which regime is it in?
3. Which of these is the bias-variance decomposition of expected squared error?
4. Adding L2 regularisation to a high-variance model typically:
5. The double-descent phenomenon means:

Interview Q&A

Derive the bias-variance decomposition for expected squared error.onsite

Assume the truth is $y = f(x) + \varepsilon$ with $\mathbb{E}[\varepsilon] = 0$ and $\mathrm{Var}(\varepsilon) = \sigma^{2}$ . Fit a random model $\hat{f}$ on a random dataset $\mathcal{D}$ . Fix a query point $x$ and let $\bar{f}(x) = \mathbb{E}_{\mathcal{D}}[\hat{f}(x)]$ .

Then

\mathbb{E}\!\left[(y - \hat{f}(x))^{2}\right] = \mathbb{E}\!\left[ (f(x) + \varepsilon - \hat{f}(x))^{2} \right].

Insert $\bar{f}(x)$ :

= \mathbb{E}\!\left[ \bigl((f(x) - \bar{f}(x)) + (\bar{f}(x) - \hat{f}(x)) + \varepsilon\bigr)^{2}\right].

Expand. The cross terms $(f - \bar{f})(\bar{f} - \hat{f})$ have expectation zero because $\mathbb{E}_{\mathcal{D}}[\bar{f}(x) - \hat{f}(x)] = 0$ . The cross terms with $\varepsilon$ vanish because $\varepsilon$ is independent of $\mathcal{D}$ and zero-mean.

What remains is

\mathrm{Err}(x) = \underbrace{(f(x) - \bar{f}(x))^{2}}_{\mathrm{Bias}^{2}} + \underbrace{\mathbb{E}_{\mathcal{D}}\!\left[(\hat{f}(x) - \bar{f}(x))^{2}\right]}_{\mathrm{Variance}} + \underbrace{\sigma^{2}}_{\text{noise}}. \quad \blacksquare

The decomposition is exact for squared error and has no analogue this clean for 0/1 or cross-entropy losses.

From learning curves alone, how do you decide whether to add data, capacity, or regularisation?phone

Look at the gap between training and validation error.

Both high, close together: under-fitting / high bias. Add capacity, add features, or remove regularisation. More data will not help.
Training near zero, validation much higher: over-fitting / high variance. Add data first (cheapest), then regularisation (early stopping, dropout, weight decay), then reduce capacity if those fail.
Both low: you are done. Stop and ship.

The diagnostic is paraphrased from Andrew Ng’s recipe in Machine Learning Yearning and is consistent with the bias-variance decomposition: more data shrinks variance, more capacity shrinks bias, regularisation trades variance for bias.

Reconcile classical bias-variance with double descent.research

The classical curve [2] assumes you stop training at the interpolation threshold or earlier: you find a model that fits the training data well but does not memorise it. Past that point, the classical view says variance dominates and test error rises.

Double descent [3][4] shows that if you keep growing the model past the interpolation threshold, test error drops a second time. Three things resolve the apparent contradiction:

Variance is not monotonic in capacity. In the over-parameterised regime, many models interpolate the training data perfectly. SGD’s implicit bias picks ones with low norm or low complexity, which generalise.
The decomposition still holds. Bias-squared + variance + noise still equals total error. What changes is the shape of the variance curve.
The “sweet spot” of the classical picture is now a local minimum, not a global one. With enough capacity (and enough data) you can do better at extreme over-parameterisation.

Practically: if your training error is near zero and validation error is poor, the right move is either to add more regularisation (classical) or to make the model much larger (modern). The two are not equivalent; which works depends on the data and architecture.

Why does L1 regularisation produce sparsity but L2 does not?onsite

Geometrically, the L1 norm has corners on the axes; the L2 norm is smooth and round. Optimisation under a constraint $\lVert w \rVert_{1} \le t$ will often hit a corner of the L1 ball (where coordinates are exactly zero), whereas optimisation under $\lVert w \rVert_{2} \le t$ hits a point on a smooth sphere (where all coordinates are typically non-zero but small).

Algebraically, the L1 sub-gradient is constant in magnitude (sign of the weight), so a small update can drive a weight all the way to zero and leave it there. The L2 gradient is proportional to the weight, so weights asymptote to zero but never reach it.

Bayesian view: L1 corresponds to a Laplace prior (spiky at zero, heavy-tailed); L2 corresponds to a Gaussian prior (smooth around zero). The Laplace’s spike at zero translates into a non-zero posterior probability that the weight is exactly zero.

References

Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer. https://hastie.su.domains/ElemStatLearn/
Geman, S., Bienenstock, E., and Doursat, R. (1992). Neural Networks and the Bias/Variance Dilemma. Neural Computation 4(1), 1-58. doi:10.1162/neco.1992.4.1.1
Belkin, M., Hsu, D., Ma, S., and Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias-variance trade-off. PNAS 116(32), 15849-15854. doi:10.1073/pnas.1903070116
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. (2019). Deep Double Descent: Where Bigger Models and More Data Hurt. arXiv:1912.02292
Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. ISBN 978-0-387-31073-2

Data & leakage Evaluation metrics