Math you need

Background

Three mathematical pillars underpin virtually all of ML: linear algebra, calculus, and probability
You do not need a PhD-level treatment - you need fluency with specific operations that recur constantly
Understanding the math lets you read papers, debug models, and design architectures instead of just calling APIs

The Concept

ML math boils down to: vectors and matrices for representing data and parameters, derivatives and gradients for optimization, and probability distributions for modeling uncertainty and designing losses.

How it Links

Linear algebra is used in every layer of a neural network (matrix multiply)
Calculus powers backpropagation and all gradient-based optimization
Probability theory underlies Bayes classifiers, VAEs, diffusion models, and model calibration

Concrete Example

Training a neural network is literally: represent data as matrices (linear algebra), compute a loss, take its derivative with respect to weights (calculus), and update weights proportionally to how much each contributed to the error (probability via cross-entropy loss).

Prerequisites

High-school algebra and single-variable calculus., AI / ML / DL / GenAI map (helpful context)

Learning outcomes

Read a modern ML paper without getting stuck on notation.
Differentiate a scalar loss with respect to a matrix without consulting a table.
Distinguish maximum likelihood from maximum a posteriori, and write down both objectives.
Compute the entropy of a distribution and the KL divergence between two.
Identify which mathematical tool a given paper is using, even when it is not labelled.

TL;DR

You do not need every result in every textbook; you need fluency in five chapters: linear algebra (vectors, matrices, eigendecomposition, SVD), calculus (gradients, chain rule, Jacobians), probability (random variables, Bayes, expectation), statistics (estimators, MLE/MAP), and information theory (entropy, KL divergence). The Mathematics for Machine Learning book [1] is a single 400-page reference for all five.

Intuition

The 80/20 of ML math

You do not need every theorem in every chapter. You need:

Linear algebra: vectors, matrices, products, norms, eigendecomposition, SVD.
Calculus: gradients, chain rule, Jacobians, the matrix-cookbook identities.
Probability: Bayes rule, expectations, common distributions, the law of total expectation.
Statistics: maximum likelihood, MAP, bias and variance of estimators.
Information theory: entropy, cross-entropy, KL divergence, mutual information.

Five short chapters across three books [1][3][4] cover everything you will see in a modern ML paper.

Linear algebra

TL;DR

Everything in ML is a vector or a matrix. Linear algebra is the language; eigendecomposition and SVD are the verbs.

Linear algebra: the operators that show up everywhere

Vectors and norms

A vector $\mathbf{x} \in \mathbb{R}^d$ has a length (norm). The two you will see constantly:

\lVert \mathbf{x} \rVert_2 = \sqrt{\sum_{i=1}^d x_i^2}, \qquad \lVert \mathbf{x} \rVert_1 = \sum_{i=1}^d |x_i|.

The $L_2$ norm penalises large coordinates quadratically; the $L_1$ norm penalises them linearly and is the source of the sparsity-inducing behaviour of Lasso regression.

Matrix multiplication

For $A \in \mathbb{R}^{m \times k}$ and $B \in \mathbb{R}^{k \times n}$ , the product $C = AB \in \mathbb{R}^{m \times n}$ has entries

C_{ij} = \sum_{\ell=1}^{k} A_{i\ell} B_{\ell j}.

A neural-network forward pass is a sequence of matrix multiplies interleaved with element-wise nonlinearities.

Eigendecomposition

A square matrix $A \in \mathbb{R}^{n \times n}$ that has $n$ linearly independent eigenvectors can be written

A = Q \Lambda Q^{-1},

where $\Lambda$ is diagonal with the eigenvalues and $Q$ ‘s columns are the eigenvectors. PCA finds the eigendecomposition of the empirical covariance matrix.

Singular Value Decomposition (SVD)

Every real matrix $A \in \mathbb{R}^{m \times n}$ has a decomposition

A = U \Sigma V^{\top},

where $U \in \mathbb{R}^{m \times m}$ and $V \in \mathbb{R}^{n \times n}$ are orthogonal and $\Sigma \in \mathbb{R}^{m \times n}$ is diagonal with non-negative entries (the singular values). SVD generalises eigendecomposition to non-square and singular matrices and is the workhorse behind low-rank approximation, recommender systems, and LoRA fine-tuning of LLMs.

Pseudo-inverse and least squares

The least-squares problem $\min_{\mathbf{w}} \lVert A\mathbf{w} - \mathbf{b}\rVert_2^2$ has solution

\mathbf{w}^{\star} = (A^{\top}A)^{-1} A^{\top} \mathbf{b}

when $A$ has full column rank, and more generally $\mathbf{w}^{\star} = A^{+}\mathbf{b}$ where $A^{+}$ is the Moore-Penrose pseudo-inverse. This is the closed-form for ordinary linear regression.

Positive (semi-)definiteness

A symmetric matrix $A$ is positive definite if $\mathbf{x}^{\top}A\mathbf{x} > 0$ for all $\mathbf{x} \neq 0$ (positive semi-definite for $\geq 0$ ). Covariance matrices and Hessians of convex functions are PSD; PSD matrices have non-negative eigenvalues and a unique square root.

For a deeper treatment, Strang [5] is the canonical applied reference.

Calculus

TL;DR

Optimisation runs on gradients. The chain rule, applied through a computation graph, is what backpropagation actually computes.

Gradient descent on f(x) = x²

Learning rate0.10x = 2.5000 f(x) = 6.2500 ∇f(x) = 5.0000

Calculus: gradients, Jacobians, Hessians, and Taylor expansion

Gradient of a scalar function

For $f : \mathbb{R}^d \to \mathbb{R}$ , the gradient is the vector of partial derivatives:

\nabla f(\mathbf{x}) = \left( \frac{\partial f}{\partial x_1}, \ldots, \frac{\partial f}{\partial x_d} \right)^{\top}.

It points in the direction of steepest ascent; $-\nabla f$ is the descent direction used by SGD.

Chain rule

For $f : \mathbb{R}^n \to \mathbb{R}$ and $g : \mathbb{R}^m \to \mathbb{R}^n$ ,

\frac{\partial (f \circ g)}{\partial x_i} = \sum_{j=1}^{n} \frac{\partial f}{\partial g_j} \frac{\partial g_j}{\partial x_i}.

Backpropagation is the chain rule applied through a computation DAG, with intermediate gradients reused.

Jacobian

For a vector-valued function $\mathbf{f} : \mathbb{R}^n \to \mathbb{R}^m$ , the Jacobian is the $m \times n$ matrix

J_{ij} = \frac{\partial f_i}{\partial x_j}.

The Jacobian-vector product (JVP) and vector-Jacobian product (VJP) are the building blocks of forward-mode and reverse-mode autodiff respectively.

Hessian

The Hessian of $f : \mathbb{R}^d \to \mathbb{R}$ is the $d \times d$ matrix of second partials:

H_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}.

Newton’s method uses $H^{-1}$ ; second-order optimisers in deep learning approximate it because the full Hessian is $O(d^2)$ memory.

Taylor expansion

Around $\mathbf{x}_0$ , a smooth function admits the second-order Taylor expansion

f(\mathbf{x}_0 + \boldsymbol{\delta}) \approx f(\mathbf{x}_0) + \nabla f(\mathbf{x}_0)^{\top} \boldsymbol{\delta} + \tfrac{1}{2} \boldsymbol{\delta}^{\top} H(\mathbf{x}_0) \boldsymbol{\delta}.

Gradient descent minimises the linear part; Newton’s method minimises the full quadratic.

Matrix calculus identities

Three identities you will use without thinking:

\frac{\partial \, \mathbf{a}^{\top} \mathbf{x}}{\partial \mathbf{x}} = \mathbf{a}, \qquad \frac{\partial \, \mathbf{x}^{\top} A \mathbf{x}}{\partial \mathbf{x}} = (A + A^{\top})\mathbf{x}, \qquad \frac{\partial \, \lVert \mathbf{x} \rVert_2^2}{\partial \mathbf{x}} = 2\mathbf{x}.

For more, consult the Matrix Cookbook (Petersen and Pedersen 2012).

Convexity

A function $f$ is convex if for all $\mathbf{x}, \mathbf{y}$ and $t \in [0, 1]$ ,

f(t\mathbf{x} + (1-t)\mathbf{y}) \le t f(\mathbf{x}) + (1-t) f(\mathbf{y}).

For convex $f$ , every local minimum is global. Linear regression, logistic regression, and SVMs have convex losses; deep nets do not. Boyd and Vandenberghe [2] is the canonical reference.

Probability

TL;DR

ML is statistics in disguise. The Bayes rule, expectations, and a handful of distributions are the moves you reach for.

Probability: random variables, Bayes, distributions

Bayes rule

For events $A$ and $B$ with $P(B) > 0$ ,

P(A \mid B) = \frac{P(B \mid A) P(A)}{P(B)}.

For a parameter $\theta$ given data $\mathcal{D}$ ,

P(\theta \mid \mathcal{D}) = \frac{P(\mathcal{D} \mid \theta) P(\theta)}{P(\mathcal{D})} \propto P(\mathcal{D} \mid \theta) P(\theta).

The posterior is proportional to the likelihood times the prior; the denominator is the (often intractable) evidence.

Expectation and variance

For a random variable $X$ ,

\mathbb{E}[X] = \int x\, p(x)\, dx, \qquad \mathrm{Var}(X) = \mathbb{E}[(X - \mathbb{E}[X])^2] = \mathbb{E}[X^2] - \mathbb{E}[X]^2.

The law of total expectation: $\mathbb{E}[X] = \mathbb{E}[\mathbb{E}[X \mid Y]]$ .

Common distributions

Distribution	Support	Parameters	Use in ML
Bernoulli	{0, 1}	p in [0, 1]	Binary classification labels; output of sigmoid.
Categorical	{1, ..., K}	p_1, ..., p_K with sum 1	Multiclass classification; output of softmax.
Binomial	{0, ..., n}	n, p	Counts of successes in n trials.
Gaussian (normal)	R	mu, sigma^2	Noise model in regression; prior on weights.
Multivariate Gaussian	R^d	mu, Sigma	Latent variables in VAEs; Gaussian processes.
Beta	[0, 1]	alpha, beta	Conjugate prior for Bernoulli; smoothing rates.
Dirichlet	(K-1)-simplex	alpha in R^K_+	Conjugate prior for Categorical; topic models.
Exponential	[0, infinity)	lambda > 0	Inter-arrival times; survival analysis.

Statistics

TL;DR

Estimators have bias, variance, and consistency. MLE and MAP are the two most common ways to fit them.

Statistics: estimators, MLE, MAP, hypothesis testing

Estimator properties

An estimator $\hat{\theta}$ of $\theta$ has:

Bias: $\mathrm{Bias}(\hat{\theta}) = \mathbb{E}[\hat{\theta}] - \theta$ .
Variance: $\mathrm{Var}(\hat{\theta}) = \mathbb{E}[(\hat{\theta} - \mathbb{E}[\hat{\theta}])^2]$ .
Consistency: $\hat{\theta}_n \xrightarrow{p} \theta$ as $n \to \infty$ .

Mean squared error decomposes as $\mathrm{MSE}(\hat{\theta}) = \mathrm{Bias}(\hat{\theta})^2 + \mathrm{Var}(\hat{\theta})$ . This decomposition is the backbone of the bias-variance tradeoff (see Bias, variance, over/under-fitting).

Maximum Likelihood Estimation (MLE)

Given iid data $\mathcal{D} = \{x_1, \ldots, x_n\}$ with density $p(x \mid \theta)$ ,

\hat{\theta}_{\mathrm{MLE}} = \arg\max_{\theta}\; \prod_{i=1}^{n} p(x_i \mid \theta) = \arg\max_{\theta}\; \sum_{i=1}^{n} \log p(x_i \mid \theta).

For the Gaussian likelihood, MLE recovers the empirical mean and (uncorrected) variance. For a softmax classifier, MLE is exactly cross-entropy minimisation.

Maximum A Posteriori (MAP)

If we have a prior $p(\theta)$ ,

\hat{\theta}_{\mathrm{MAP}} = \arg\max_{\theta}\; \log p(\mathcal{D} \mid \theta) + \log p(\theta).

A Gaussian prior on $\theta$ corresponds exactly to L2 regularisation; a Laplace prior corresponds to L1 regularisation. This is why ridge and Lasso are sometimes called “MAP with Gaussian / Laplace prior”.

Confidence intervals and hypothesis tests

A 95% confidence interval is constructed so that, under repeated sampling, the interval covers the true parameter 95% of the time. It is not the probability that the parameter lies in this particular interval (that is the Bayesian credible interval).

Hypothesis testing fixes a null $H_0$ , computes a test statistic, and rejects $H_0$ if the p-value falls below a chosen significance level. In ML, hypothesis tests show up in A/B testing and in evaluating whether one model is significantly better than another (paired t-tests on cross-validation folds, McNemar’s test for classifiers).

Information theory

TL;DR

Cross-entropy is just the log-loss. KL divergence measures how surprised one distribution is by another. Mutual information measures shared structure.

Information theory: entropy, cross-entropy, KL, mutual information

Entropy

The entropy of a discrete distribution $p$ is

H(p) = -\sum_{x} p(x) \log p(x).

It is the minimum average number of bits (or nats, with natural log) required to encode samples from $p$ .

Cross-entropy

For two distributions $p$ (the truth) and $q$ (the model),

H(p, q) = -\sum_{x} p(x) \log q(x).

This is the average number of bits a code optimised for $q$ uses to encode samples from $p$ . In supervised classification, $p$ is a one-hot label and $q$ is the model’s softmax, so cross-entropy reduces to $-\log q(\text{true class})$ - exactly the negative log-likelihood, also known as log-loss.

KL divergence

The Kullback-Leibler divergence from $p$ to $q$ is

D_{\mathrm{KL}}(p \,\Vert\, q) = \sum_{x} p(x) \log \frac{p(x)}{q(x)} = H(p, q) - H(p).

It is non-negative, zero iff $p = q$ , and asymmetric in its arguments. Minimising cross-entropy with $p$ fixed (as in classification) is equivalent to minimising $D_{\mathrm{KL}}(p \Vert q)$ .

Mutual information

For two random variables $X$ and $Y$ ,

I(X; Y) = D_{\mathrm{KL}}(p(x, y) \,\Vert\, p(x) p(y)) = H(X) - H(X \mid Y).

Mutual information is the reduction in uncertainty about $X$ from observing $Y$ . It is symmetric and non-negative. In ML, mutual information underlies feature selection (mRMR), representation learning (InfoNCE, contrastive loss), and information-bottleneck analyses of deep nets.

For the canonical reference, Cover and Thomas [4] is the standard textbook.

Where each tool shows up

Tool	Where you meet it in ML
Matrix multiplication	Every neural-network forward pass; attention; embedding lookups.
Eigendecomposition / SVD	PCA; spectral methods; low-rank adaptation (LoRA).
Gradient + chain rule	Backpropagation; SGD, Adam, RMSprop.
Jacobian / Hessian	Second-order optimisers (L-BFGS, K-FAC, Newton); influence functions.
Bayes rule	Naive Bayes; Bayesian deep learning; diffusion models.
Gaussian distribution	Linear regression noise model; VAE prior; Gaussian processes.
Categorical / softmax	Multiclass classification output; LLM next-token distribution.
MLE	Default training objective for almost every supervised model.
MAP / regularisation	Ridge (L2), Lasso (L1), early stopping, weight decay.
Cross-entropy	Standard classification loss; training objective for LLMs.
KL divergence	VAE ELBO; PPO trust-region penalty in RLHF; knowledge distillation.
Mutual information	Feature selection; contrastive learning (InfoNCE); information-bottleneck.

Pop quiz

Check yourself

Score: 0 / 5(0 answered)

1. What is the gradient of f(x) = x^T A x with respect to x, when A is square but not necessarily symmetric?
2. Which of these statements about KL divergence is FALSE?
3. L2 regularisation on linear regression weights corresponds to which Bayesian setup?
4. Which decomposition exists for ANY real matrix A in R^{m x n}, square or not, full rank or not?
5. For a discrete distribution with support {a, b}, what is the entropy H(p) maximised at?

Interview Q&A

Derive the closed-form solution for ordinary least squares from scratch.phone

We want to minimise $L(\mathbf{w}) = \lVert X\mathbf{w} - \mathbf{y} \rVert_2^2 = (X\mathbf{w} - \mathbf{y})^{\top}(X\mathbf{w} - \mathbf{y})$ .

Expand:

L(\mathbf{w}) = \mathbf{w}^{\top} X^{\top} X \mathbf{w} - 2 \mathbf{y}^{\top} X \mathbf{w} + \mathbf{y}^{\top}\mathbf{y}.

Take the gradient w.r.t. $\mathbf{w}$ using $\partial \mathbf{w}^{\top} A \mathbf{w} / \partial \mathbf{w} = (A + A^{\top}) \mathbf{w} = 2 A \mathbf{w}$ for symmetric $A = X^{\top}X$ :

\nabla_{\mathbf{w}} L = 2 X^{\top} X \mathbf{w} - 2 X^{\top} \mathbf{y}.

Set to zero: $X^{\top}X \mathbf{w} = X^{\top}\mathbf{y}$ , i.e. the normal equations. If $X^{\top}X$ is invertible (X has full column rank),

\mathbf{w}^{\star} = (X^{\top}X)^{-1} X^{\top}\mathbf{y}.

If $X^{\top}X$ is singular, use the pseudo-inverse $\mathbf{w}^{\star} = X^{+}\mathbf{y}$ , computed via SVD.

Show that softmax + cross-entropy gives the same gradient as a logistic regression update on a single example.onsite

Softmax with logits $z_k$ : $q_k = e^{z_k} / \sum_j e^{z_j}$ . Cross-entropy with one-hot label $y$ (true class $c$ ):

L = -\log q_c = -z_c + \log \sum_j e^{z_j}.

Differentiate w.r.t. $z_k$ :

\frac{\partial L}{\partial z_k} = -\mathbb{1}[k = c] + \frac{e^{z_k}}{\sum_j e^{z_j}} = q_k - \mathbb{1}[k = c].

So the gradient of softmax + cross-entropy with respect to logits is (prediction - one-hot target). For binary classification this reduces to $q - y$ where $q$ is the sigmoid output - exactly the logistic regression gradient. The simplicity of this gradient is why softmax + cross-entropy is the default classification head.

What is the connection between MLE and minimising KL divergence?research

For iid data $\mathcal{D} = \{x_1, \ldots, x_n\}$ from an unknown true distribution $p^{\star}$ , the empirical distribution $\hat{p}$ puts mass $1/n$ on each $x_i$ . MLE chooses $\theta$ to maximise

\frac{1}{n} \sum_i \log q_{\theta}(x_i) = \mathbb{E}_{\hat{p}}[\log q_{\theta}].

Now,

D_{\mathrm{KL}}(\hat{p} \,\Vert\, q_{\theta}) = \mathbb{E}_{\hat{p}}[\log \hat{p}] - \mathbb{E}_{\hat{p}}[\log q_{\theta}].

The first term does not depend on $\theta$ , so minimising $D_{\mathrm{KL}}(\hat{p} \Vert q_{\theta})$ in $\theta$ is the same as maximising $\mathbb{E}_{\hat{p}}[\log q_{\theta}]$ - that is, MLE. As $n \to \infty$ , $\hat{p} \to p^{\star}$ in the appropriate sense, so MLE asymptotically minimises $D_{\mathrm{KL}}(p^{\star} \Vert q_{\theta})$ , the discrepancy between the truth and the model. This is why MLE is “the natural” estimator when you trust your model class to contain the truth, and why it goes wrong when the model class is misspecified.

A function f is convex. What does that buy you computationally?onsite

Three concrete things, paraphrased from Boyd and Vandenberghe [2].

Every local minimum is global. SGD or any first-order method that finds a stationary point finds an optimum.
Strong duality often holds. The dual problem gives a lower bound that matches the primal, which means cross-checking optimality is cheap.
Optimisation is well-conditioned. Convex losses (linear regression, logistic regression, SVM) admit polynomial-time solvers and lots of theory; non-convex losses (deep nets) do not, which is why “did the model train” is a real question for neural networks but never for logistic regression.

The flip side: convexity is restrictive. Almost no expressive function class is convex in its parameters. Deep learning trades convexity for expressiveness and absorbs the optimisation difficulty.

References

Deisenroth, M. P., Faisal, A. A., and Ong, C. S. (2020). Mathematics for Machine Learning. Cambridge University Press. https://mml-book.github.io/
Boyd, S. and Vandenberghe, L. (2004). Convex Optimization. Cambridge University Press. https://web.stanford.edu/~boyd/cvxbook/
Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. ISBN 978-0-387-31073-2
Cover, T. M. and Thomas, J. A. (2006). Elements of Information Theory (2nd ed.). Wiley. ISBN 978-0-471-24195-9
Strang, G. (2019). Linear Algebra and Learning from Data. Wellesley-Cambridge Press. ISBN 978-0-692-19638-0

AI / ML / DL / GenAI map The ML workflow