Practical Projects (Classical & Deep)

TL;DR

How to choose a project

Pick the project whose comparison story demonstrates the skill you want to be judged on:

• Statistical rigor and feature engineering: Titanic with XGBoost + SHAP or California housing regression.
• Model-family taxonomy: MNIST classifier comparison (kNN, RF, MLP, CNN on the same data).
• Modern deep learning: IMDB sentiment classifier (TF-IDF baseline -> LSTM -> Transformer fine-tune).
• Generative modeling: MNIST GAN or image colorization with autoencoder.
• Production ML systems: MovieLens recommender (cold-start, cost of evaluation, ranking metrics).

A good portfolio has at most one project per category, picked to showcase distinct skills. Three projects done deeply beats six done shallowly.

Project catalog at a glance

1. MNIST classifier comparison

Goal: demonstrate the bias-variance and capacity-vs-data trade-off across model families on a single dataset.

Prerequisites: /foundations/bias-variance-overfitting, /foundations/evaluation-metrics, /classical-ml/knn, /classical-ml/random-forests, /deep-learning/mlp, /deep-learning/cnns.

Dataset: MNIST handwritten digits, 60k train / 10k test, 28x28 grayscale, 10 classes [1]. Available at https://yann.lecun.com/exdb/mnist/ or via sklearn.datasets.fetch_openml('mnist_784') or torchvision.datasets.MNIST.

Starter steps:

1. Load MNIST, flatten to 784-dim, scale to $[0, 1]$, split 60k/10k. Compute the trivial baseline: predict the most common class. Report accuracy (~11%).
2. Fit KNeighborsClassifier(n_neighbors=5) on a 10k train subset (full 60k is slow). Report test accuracy and total wall-clock training + inference time.
3. Fit RandomForestClassifier(n_estimators=100) on the full 60k. Report accuracy and training time.
4. Train an MLP (2 hidden layers of 256 units, ReLU, dropout 0.2) for 10 epochs. Report accuracy and training time.
5. Train a 3-conv-layer CNN with the same training budget. Report accuracy and training time.

Evaluation criteria:

• kNN: 95-97% test accuracy, training instant, inference seconds per 10k.
• RF: 96-97% test accuracy, training ~30 seconds.
• MLP: 97-98% test accuracy, training a couple of minutes on CPU.
• CNN: 99.0-99.4% test accuracy, training under 10 minutes on a CPU; under 1 minute on GPU.
• Plot accuracy vs. wall-clock training time across all four; CNN should be on the Pareto frontier.

Stretch goals:

1. Add Fashion-MNIST as a second test domain; the gap between MLP and CNN widens.
2. Vary CNN depth/width and plot accuracy vs. parameter count.
3. Add data augmentation (random crops, rotations) and quantify the gain on the CNN.
4. Implement label-smoothing or mixup and report the regularization effect.

2. Titanic survival with XGBoost + SHAP

Goal: classical tabular ML pipeline end-to-end with explainability.

Prerequisites: /foundations/evaluation-metrics, /classical-ml/logistic-regression, /classical-ml/decision-trees, /classical-ml/gradient-boosting.

Dataset: Titanic passenger list, 891 train / 418 test, mixed categorical and numeric features. Available at https://www.kaggle.com/c/titanic/data.

Starter steps:

1. Load the data; impute missing ages with the median per Pclass-Sex group, missing embarked with the mode. One-hot encode categoricals.
2. Fit a LogisticRegression baseline. Report 5-fold cross-validated accuracy and AUC.
3. Fit XGBClassifier with default hyperparameters [6]. Report the same two metrics.
4. Tune max_depth, learning_rate, n_estimators via grid search with 5-fold CV. Report best parameters and the cross-validated metric.
5. Compute SHAP values [2] on the test set with shap.TreeExplainer. Plot the global summary (beeswarm) and one local explanation for a high-confidence and a low-confidence prediction.

Evaluation criteria:

• Logistic baseline: 5-fold CV accuracy 0.78-0.81, AUC 0.84-0.87.
• Tuned XGBoost: 5-fold CV accuracy 0.82-0.84, AUC 0.87-0.90.
• Held-out Kaggle leaderboard score: 0.78-0.81 typical (the test set is small so leaderboard variance is high).
• SHAP summary: Sex, Pclass, and Age should dominate; if any feature dominates that should not (e.g. PassengerId), you have a leakage bug.

Stretch goals:

1. Engineer features: title from Name, family size from SibSp + Parch, has-cabin indicator. Quantify the gain.
2. Replace one-hot with target encoding for categoricals; compare on the same CV split.
3. Add LightGBM and CatBoost; report timing and accuracy on the same CV.
4. Use SHAP interaction values to find the strongest 2-way interaction; compare with a hand-engineered interaction term in logistic regression.

3. California housing regression

Goal: regularized linear regression and cross-validation on a real economics dataset.

Prerequisites: /foundations/math-you-need, /classical-ml/linear-regression.

Dataset: California housing 1990 census [7], 20640 rows, 8 numeric features, target = median house value. Available via sklearn.datasets.fetch_california_housing().

Starter steps:

1. Load and scale features with StandardScaler. Train/test split 80/20.
2. Fit OLS, Ridge, Lasso, and ElasticNet on the train set; for each tune the regularization strength via 5-fold CV.
3. Report test RMSE and $R^2$ for each model. Plot the learning curve (train and test error vs. training set size) for OLS and Ridge.
4. Plot the Lasso coefficient path: $|\hat\beta_j|$ vs. $\log \alpha$, highlight the order in which coefficients enter.
5. Add polynomial features (degree 2, no interactions) and refit Ridge; report the gain.

Evaluation criteria:

• OLS: test RMSE ~0.72-0.74, $R^2$ ~0.59-0.61.
• Ridge with tuned $\alpha$: test RMSE ~0.71-0.73, $R^2$ ~0.60-0.62.
• Lasso with tuned $\alpha$: similar RMSE; coefficient sparsity > 0 with at least one feature driven to zero.
• Polynomial Ridge: test RMSE ~0.62-0.65, $R^2$ ~0.68-0.71.

Stretch goals:

1. Add a RandomForestRegressor and GradientBoostingRegressor baseline; compare on the same split and discuss why trees beat linear models on this dataset.
2. Quantify residual heteroskedasticity with a Breusch-Pagan test and add log-transform of the target; report the change in RMSE.
3. Spatial-aware split: hold out one geographic region and evaluate generalization. Report the train/test gap; this is the realistic deployment scenario.
4. Use 1-D cubic splines or a small additive model (pyGAM) and compare against Ridge.

4. Customer segmentation

Goal: unsupervised clustering on real transactional data with three different cluster shapes.

Prerequisites: /classical-ml/clustering, /classical-ml/dimensionality-reduction.

Dataset: UCI Online Retail (UK retail invoices, 2010-2011). Available at https://archive.ics.uci.edu/ml/datasets/online+retail. About 540k rows, fields include InvoiceNo, CustomerID, Country, Quantity, UnitPrice [8].

Starter steps:

1. Aggregate to one row per CustomerID with RFM features: Recency (days since last purchase), Frequency (number of invoices), Monetary (total spend). Filter to customers with at least 3 invoices.
2. Log-transform F and M, then standardize all three. Run PCA and inspect the explained-variance curve; project to 2-D for plotting.
3. Fit KMeans(n_clusters=k) for $k \in \{3, 4, 5, 6, 7, 8\}$. Plot the elbow (inertia vs. k) and silhouette score vs. k. Pick a $k$.
4. Fit DBSCAN(eps, min_samples) with reasonable hyperparameters; plot how many points are clustered vs. labeled noise.
5. Fit GaussianMixture(n_components=k) with the same $k$. Compare cluster assignments with K-Means via Adjusted Rand Index.

Evaluation criteria:

• Elbow at $k = 4$ or $5$ with silhouette $\ge 0.4$ on RFM features.
• DBSCAN should mark 5-15% of customers as noise; the rest cluster into 4-8 groups.
• GMM and K-Means should have ARI $\ge 0.6$ when both are well-tuned (the data is roughly globular in RFM space).
• Cluster sizes should not be wildly imbalanced; if one cluster has 95% of the data, RFM features need different scaling.

Stretch goals:

1. Engineer additional features: average basket size, return rate, country, weekday vs. weekend purchases. Re-segment and check stability.
2. Run hierarchical clustering with Ward linkage; cut the dendrogram and compare to K-Means.
3. Profile the segments: median R, F, M per cluster, top-5 product categories. Build a one-slide “persona” per segment.
4. Time-aware segmentation: split customers into “early 2011” and “late 2011” buckets and check whether segment membership is stable.

5. Image colorization with an autoencoder

Goal: image-to-image regression with a U-Net-like autoencoder.

Prerequisites: /deep-learning/cnns, /deep-learning/autoencoders-and-vaes.

Dataset: CIFAR-10 grayscale (synthesize from RGB via 0.2989 R + 0.587 G + 0.114 B) [3] for a small project, or Places365 256x256 for a real one. CIFAR-10 is at https://www.cs.toronto.edu/~kriz/cifar.html.

Starter steps:

1. Convert RGB images to LAB color space; the L channel is the grayscale input, the AB channels are the regression target. Normalize AB to roughly $[-1, 1]$.
2. Build a U-Net-style autoencoder: 4 down-sampling conv blocks, a bottleneck, 4 up-sampling conv blocks with skip connections. Output 2 channels (AB).
3. Train with $\mathcal{L} = \lVert \hat{AB} - AB \rVert_2^2$ for 30 epochs. Track validation loss; save the best checkpoint.
4. Evaluate on a held-out test set: report PSNR (peak signal-to-noise ratio) on the AB channels and L1 reconstruction error.
5. Inspect 16 sample reconstructions side-by-side: grayscale input, predicted AB combined with input L, and ground-truth color. Note systematic failure modes (skies, faces).

Evaluation criteria:

• Per-pixel L1 on AB after normalization: 0.05-0.10.
• PSNR on the colorized image: 22-26 dB (CIFAR-10 is low-resolution; bigger datasets give higher PSNR).
• Visual: skies and grass should look right; objects with category-specific colors (cars, fruit) will be desaturated because the L1 mean is the average plausible color.

Stretch goals:

1. Replace L2 loss with L1 + a perceptual loss (VGG feature matching). Compare reconstructions; perceptual loss reduces the gray-mean-ification.
2. Add a discriminator (cGAN-style: pix2pix loss). Compare on the same data.
3. Move to Places365 256x256 with a deeper U-Net; budget 3-5 epochs of training.
4. Quantify hue diversity (entropy of predicted hue) to detect “all-gray” mode collapse.

6. IMDB sentiment classifier

Goal: model-family progression on the same NLP task.

Prerequisites: /classical-ml/logistic-regression, /deep-learning/rnns-lstms-grus. Familiarity with Hugging Face transformers helps for the third stage.

Dataset: IMDB movie reviews, 25k train / 25k test, balanced binary sentiment [4]. Available at https://huggingface.co/datasets/imdb or https://ai.stanford.edu/~amaas/data/sentiment/.

Starter steps:

1. Baseline: TF-IDF unigrams + bigrams (top 50k features) + logistic regression with L2. Report test accuracy.
2. LSTM: train a 1-layer LSTM (hidden=128) on tokenized + embedded reviews (embedding dim 128, learned from scratch), max length 256. Train for 5 epochs.
3. Transformer fine-tune: load distilbert-base-uncased, fine-tune the classification head + last 2 layers for 3 epochs.
4. Build a confusion matrix for each model. Read 10 disagreement examples (LR is right, LSTM is wrong, and vice versa); note systematic patterns.
5. Plot accuracy vs. training time for the three models on a single chart.

Evaluation criteria:

• TF-IDF + LR: test accuracy 0.88-0.90.
• LSTM from scratch: test accuracy 0.86-0.89 (often loses to TF-IDF on this dataset because IMDB is small).
• DistilBERT fine-tune: test accuracy 0.92-0.94.
• Wall-clock: TF-IDF takes minutes on CPU, LSTM about an hour on CPU, DistilBERT about 30 minutes on a single GPU.

Stretch goals:

1. Ablation on the TF-IDF baseline: chars n-grams, max features, log-tf weighting. Quantify each.
2. Replace DistilBERT with roberta-large for a stronger ceiling; report accuracy and compute cost.
3. Adversarial robustness: replace 5% of words with synonyms and re-evaluate; the LR baseline often degrades less than the LSTM.
4. Few-shot: use only 1k labeled examples and compare TF-IDF + LR to fine-tuned DistilBERT to demonstrate the pretraining advantage.

7. MNIST GAN

Goal: train a generative model and evaluate sample quality and diversity.

Prerequisites: /deep-learning/cnns, /deep-learning/gans.

Dataset: MNIST again. The point is that you already know the digit distribution, so visual checks of sample quality and mode coverage are direct.

Starter steps:

1. Implement a small DCGAN: generator with 3 ConvTranspose layers and Tanh output, discriminator with 3 strided Conv layers and Sigmoid output. Latent dim 64.
2. Train for 50 epochs with Adam(lr=2e-4, betas=(0.5, 0.999)), batch size 128. Save 64 sample images per 1000 steps.
3. After training, sample 5000 images, classify them with a pretrained MNIST classifier, and plot the histogram of predicted digits. The histogram should be roughly uniform; any mode under-represented signals partial collapse.
4. Switch to WGAN-GP loss with $\lambda = 10$, 5 critic steps per generator step, and no BatchNorm in the critic. Retrain. Compare sample quality and the histogram.
5. Plot the critic loss curve from WGAN-GP; it should decrease (more negative) and roughly correlate with sample quality.

Evaluation criteria:

• DCGAN: visual samples are recognizable digits within 5-10 epochs; full coverage of all 10 modes within 30 epochs.
• WGAN-GP: training is more stable (less oscillation), critic loss curve is monotonic, sample quality similar after equal compute.
• Per-class proportion among 5000 samples: each class between 7% and 13% (uniform baseline is 10%); large deviations indicate mode collapse.

Stretch goals:

1. Conditional DCGAN: feed a one-hot digit label to both networks and sample digit-by-digit.
2. Compute FID against a held-out 10k MNIST set using a small pretrained MNIST feature extractor; lower is better.
3. Add spectral normalization to the discriminator and compare against gradient penalty on training stability.
4. Compare GAN samples to a simple VAE trained on the same data; the VAE will be blurrier but mode-covering.

8. MovieLens recommender

Goal: practical recommender pipeline with matrix factorization, evaluated by ranking metrics.

Prerequisites: /classical-ml/recommender-systems, /classical-ml/dimensionality-reduction (for the SVD baseline).

Dataset: MovieLens-100k (943 users, 1682 movies, 100k ratings) for a quick project, or MovieLens-1M / MovieLens-Latest for a real one [5]. Available at https://files.grouplens.org/datasets/movielens/.

Starter steps:

1. Load ratings; split chronologically (train = first 80% of each user’s ratings, test = last 20%) so the evaluation is realistic.
2. Baseline: popularity ranker (recommend top-rated unseen movies globally) and a user-mean predictor. Report MAE and RMSE on test ratings, and Recall@10 on top-N recommendations.
3. Matrix factorization: alternating least squares (ALS) or Funk-SVD with 64 factors, trained for 20 epochs. Report the same metrics.
4. Evaluation: for each user in the test set, score all unseen movies; compute Precision@10, Recall@10, NDCG@10. Plot training loss and validation NDCG by epoch.
5. Cold-start audit: report recall@10 separately on users with fewer than 5 training ratings; this is where collaborative filtering struggles.

Evaluation criteria:

• Popularity baseline: NDCG@10 ~0.10, Recall@10 ~0.05.
• ALS / matrix factorization: RMSE 0.90-0.95 on ML-100k, NDCG@10 0.30-0.40, Recall@10 0.20-0.30.
• Cold-start users: matrix factorization gain over popularity should be small or negative; this is the documented limitation.

Stretch goals:

1. Add implicit-feedback matrix factorization (treat any rating as a “click”, weight by confidence). Compare on Recall@10.
2. Two-tower neural recommender with side features (genre, year). Compare to ALS.
3. Diversity-aware re-ranking: introduce a determinantal-point-process or simple genre-diversity penalty and report NDCG vs. diversity Pareto.
4. Time-aware split: train on movies released before 2015 and evaluate on 2015+; quantify concept drift.

9. CIFAR-10 with a small ResNet (stretch)

Goal: train a deep CNN from scratch with proper data augmentation and learning-rate scheduling.

Prerequisites: /deep-learning/cnns, /deep-learning/regularization-in-deep-nets, /deep-learning/backprop-and-optimizers.

Dataset: CIFAR-10, 50k train / 10k test, 32x32 RGB, 10 classes [3]. Available at https://www.cs.toronto.edu/~kriz/cifar.html or via torchvision.datasets.CIFAR10.

Starter steps:

1. Build ResNet-20 (3 stages of 3 residual blocks, channels 16-32-64). Train for 100 epochs with SGD(momentum=0.9, weight_decay=5e-4), batch 128, cosine learning-rate schedule starting at 0.1.
2. Add data augmentation: random crop with 4-pixel padding, random horizontal flip, normalize. Track train/test accuracy.
3. Train a second model without augmentation; compare the train-test gap.
4. Plot the per-class confusion matrix on the test set; identify the two most-confused class pairs.
5. Evaluate the test-time accuracy with and without test-time augmentation (10-crop average).

Evaluation criteria:

• ResNet-20 with augmentation: 91-93% test accuracy.
• No augmentation: 87-89% test accuracy, with a much larger train-test gap.
• Most-confused pairs: cat vs. dog, deer vs. horse.

Stretch goals:

1. Mixup or CutMix augmentation; report the gain.
2. ResNet-56 or PreAct-ResNet; compare against ResNet-20 at equal compute.
3. Switch to AdamW and cosine schedule; document the optimizer ablation.
4. Profile training: time per epoch on CPU, single GPU, and amp+gpu.

10. Time-series forecasting with LSTM (stretch)

Goal: sequence modeling on real noisy time series with proper baselines.

Prerequisites: /deep-learning/rnns-lstms-grus, /foundations/evaluation-metrics.

Dataset: M5 Walmart sales (Kaggle, ~30k stores x SKUs, daily for ~5 years) at https://www.kaggle.com/c/m5-forecasting-accuracy, or NOAA daily weather at https://www.ncei.noaa.gov/data/global-historical-climatology-network-daily/. M5 is the better choice if you want to engage with the M-competitions literature.

Starter steps:

1. Pick one moderately busy SKU/series. Plot the time series; identify weekly and yearly seasonality.
2. Naive baseline: predict the same day-of-week average from the prior 4 weeks. Report MASE (mean absolute scaled error) on the last 28 days.
3. Exponential smoothing (ETS): with statsmodels or darts. Report MASE on the same holdout.
4. LSTM: 1-layer LSTM (hidden=64) with day-of-week + month one-hots and 28-day lookback. Train for 50 epochs, predict 28 days ahead.
5. Compare the three models on MASE. Plot one prediction window for each model overlaid with truth.

Evaluation criteria:

• Naive baseline: MASE ~1.0 by definition (depends on baseline used).
• ETS: MASE 0.80-0.95, often hard to beat on short horizons.
• LSTM: MASE 0.85-1.05; on a single series with limited data, LSTM frequently does not beat ETS, which is the textbook M-competition lesson.
• Plot the residual ACF for the best model; significant autocorrelation at lag 7 means you missed weekly seasonality.

Stretch goals:

1. Multi-series LSTM (one model trained jointly on 1000 series with series-id embedding); the cross-series pooling is where neural models start to win.
2. Add holiday and promotion regressors (M5 provides them); quantify the gain.
3. Try a Temporal Fusion Transformer (pytorch-forecasting); report MASE and training cost.
4. Build a probabilistic forecast (quantile loss); evaluate with pinball loss at 10/50/90 quantiles.

Pitfalls

Pop quiz

Interview Q&A