Interview Prep Guide

This page aggregates the highest-signal interview material from across the hub. There is no new theory here — everything links back to the page where it is derived. Use this as a navigator for focused preparation.

1. Interview levels explained

Phone screen (30—45 min): Concept recognition and high-level intuition. Expect: “What is regularization?”, “Why use batch norm?”, “What is the bias-variance tradeoff?”. No derivations. The bar is: could you explain this to a junior engineer?

Onsite technical (45—60 min per round): You will be asked to derive things on a whiteboard. Expect: “Derive the normal equations for linear regression”, “Walk me through backpropagation”, “How does the attention mechanism work — write the formula”. You should be able to get from “definition” to “working formula” for all core topics. Mathematical comfort matters.

Research interview (60 min): Deep engagement with a specific paper or research area. Expect: “What are the limitations of RLHF compared to DPO?”, “Explain the Mamba selective state space”, “Why does LoRA work — what assumption does it make about weight updates?”. The bar is paper-level understanding plus the ability to critique and extend.

Staff / principal interview: System design + technical depth. Expect: “Design an ML system for X at Google-scale”, “How would you evaluate a large-scale retrieval system?”, “What would you change about the Transformer architecture and why?”. Combines all levels plus architectural judgment.

2. Topic-by-topic question map

3. Derivations you must know cold

These are the derivations that recur most frequently across research-track interviews. Each item links to the page where it is worked through in full. Practice doing each from memory in under 5 minutes.

• Perceptron convergence: $T \le (R/\gamma)^2$ — see Perceptron. The bound states the number of updates is at most $(R / \gamma)^2$ where $R$ is the max norm of any example and $\gamma$ is the margin.
• Normal equations: $\hat{w} = (X^\top X)^{-1} X^\top y$ — derived in Linear Regression from setting the gradient of the MSE to zero.
• Ridge closed form: $\hat{w}_\lambda = (X^\top X + \lambda I)^{-1} X^\top y$ — same page; shows how regularization adds $\lambda$ to the diagonal to fix the ill-conditioned case.
• SVM dual: $\max_\alpha \sum \alpha_i - \frac{1}{2} \sum_{ij} \alpha_i \alpha_j y_i y_j x_i^\top x_j$ — SVMs. The kernel trick replaces $x_i^\top x_j$ with $k(x_i, x_j)$.
• Backprop chain rule: $\partial L / \partial W = \partial L / \partial z \cdot \partial z / \partial W$ — Backpropagation. Know this for softmax cross-entropy and ReLU activations specifically.
• VAE ELBO: $\mathcal{L} = \mathbb{E}_{q_\phi}[\log p_\theta(x|z)] - \text{KL}(q_\phi(z|x) \| p(z))$ — Autoencoders and VAEs. Reconstruction term minus KL penalty.
• LoRA parameter budget: if $W \in \mathbb{R}^{m \times n}$ and rank $r \ll \min(m,n)$, then $\Delta W = AB$ with $A \in \mathbb{R}^{m \times r}$, $B \in \mathbb{R}^{r \times n}$; parameters = $r(m + n)$ vs $mn$ — see Fine-tuning & PEFT.
• Scaled dot-product attention: why $\sqrt{d_k}$ — see Attention. Without scaling, dot products grow as $O(d_k)$ in magnitude, pushing softmax into the saturation region with near-zero gradients.
• BPE tokenization: greedy merge of the most frequent byte-pair until vocabulary size is reached — see Tokenization & Embeddings.
• DDPM ELBO: the training objective reduces to predicting the noise $\epsilon$ added at step $t$ — see Diffusion Models. The simplified objective is $\mathbb{E}_{t, x_0, \epsilon}[\| \epsilon - \epsilon_\theta(x_t, t) \|^2]$.

4. Comparison tables interviewers love to ask

These comparisons come up in nearly every ML interview. Study the key axes, not just the names.

XGBoost vs LightGBM vs CatBoost

Transformer vs Mamba vs RWKV vs RetNet

BERT vs GPT vs T5

Adam vs AdamW vs SGD + momentum

Dropout vs Batch Norm vs Layer Norm

RAG vs Fine-tuning

5. Common pitfalls interviewers probe

1. Confusing L1 and L2 regularization effects: L2 (ridge) shrinks all weights proportionally. L1 (lasso) produces exact zeros. Interviewers often ask “which produces sparse models and why?” — the answer is L1, because the L1 ball has corners on the axes.

2. KV cache memory calculation: interviewers often ask “why can’t you run Llama-2-70B with 100k context on a single A100?” The answer is the KV cache: $2 \times 80 \text{ layers} \times 2 \times 8192 \text{ hidden} \times 100{,}000 \text{ tokens} \times 2 \text{ bytes} \approx 500 \text{ GB}$, which exceeds the 80 GB VRAM of an A100.

3. Train-test leakage subtlety: scaling features (normalize, impute, etc.) using statistics computed on the full dataset (including test) constitutes leakage. Always fit preprocessors on training data only, then apply to test. See Data & Leakage.

4. Accuracy is a bad metric for class imbalance: a model that predicts the majority class always can achieve 99% accuracy on a 1% positive-class dataset. Always report precision, recall, F1, and AUC for imbalanced problems.

5. Confusing correlation and feature importance: a feature can have low Pearson correlation with the label but high tree feature importance (because importance is marginal over splits, which can capture interactions). Pearson correlation measures linear marginal dependence only.

6. The deadly triad in RL: off-policy learning + function approximation + bootstrapping can diverge. This is why DQN needs target networks and experience replay — the combination without those stabilizers diverges in practice. See Foundations of RL.

7. Assuming softmax outputs are probabilities: softmax outputs are valid probabilities (sum to 1, all positive), but they are not calibrated probabilities unless the model has been explicitly calibrated. Overconfident softmax outputs are common in deep models.

8. LoRA vs full fine-tuning on new tasks: LoRA is not always better. For tasks that require substantial distribution shift from the base model (entirely new format, new language), full fine-tuning of all layers often outperforms LoRA at the same parameter budget. LoRA is best for tasks that are stylistically similar to the base training distribution.

9. Cross-validation leakage with time-series data: random k-fold CV on time-series data leaks future information into training. Use temporal splits (train on past, validate on future) or time-series cross-validation.

10. Gradient accumulation changes effective batch norm behavior: if you use gradient accumulation to simulate a large batch, but with batch normalization, the batch norm statistics are computed on the small actual batch at each step, not the accumulated large batch. This can cause inconsistency. Prefer layer norm or group norm when using gradient accumulation.

6. 30-day study plan

A suggested order for someone starting from a solid math and Python foundation with 30 days before an interview. Adjust pace based on existing familiarity.