FoundationsThe ML workflow

The ML workflow

Background
  • ML is not just modeling - 80% of the work is data wrangling, feature engineering, and deployment infrastructure
  • CRISP-DM (1999) formalized the iterative pipeline structure still used today
  • Each stage has distinct failure modes and quality gates
The Concept

The ML workflow is the end-to-end process from raw data to a deployed, monitored model. It covers: problem framing, data collection and cleaning, feature engineering, model selection and training, evaluation, deployment, and monitoring.

How it Links
  • Every other ML concept lives inside a specific workflow stage
  • MLOps connects the development workflow to the production deployment lifecycle
  • Understanding the full pipeline prevents common mistakes like ignoring data drift after deployment
Concrete Example
Building a house price predictor: you define the target (sale price), collect historical listings (data collection), engineer features like age/size/location (feature engineering), train several models (model selection), evaluate on held-out recent sales (evaluation), deploy to an API (deployment), and monitor for neighborhood shifts (monitoring).
Prerequisites
Why ML differs from software engineering
Learning outcomes
  • Diagram the end-to-end ML lifecycle from problem framing to retraining.
  • List the differences between a research workflow and a production workflow.
  • Identify the stage at which most ML projects actually fail (it is rarely modelling).
  • Name three things that go wrong AFTER a model is deployed and how to detect them.

TL;DR

TL;DR

A real ML project is ten stages, only one of which is “train a model”. Most failed projects fail at problem framing or data, not at the loss function. Production workflows differ from research workflows along eight dimensions, monitoring and rollback being the largest.

Intuition

The end-to-end lifecycle has a shape worth memorising. Every stage feeds the next, and every stage can - and does - send you back to an earlier one.

1. Problem framing2. Data3. EDA4. Features5. Split6. Train7. Evaluate8. Deploy9. Monitor10. RetrainLoop back: drift detected, problem reframed, or new data arrives.

Figure 1. The end-to-end ML workflow as a directed graph with a feedback loop.

The ten stages

What actually happens at each stage

1. Problem framing

The most underrated stage. You translate a business or research goal into a machine-learning problem with a measurable objective. Bad framing produces a perfectly trained model that solves the wrong problem. Andrew Ng’s Machine Learning Yearning [1] spends half its content on framing and evaluation specifically because the modelling part is the easy bit.

Concretely, you decide:

  • What is the input and what is the output?
  • What metric defines success on a held-out set, and what threshold counts as “good enough”?
  • What is the cost of a false positive vs a false negative?
  • What is the deployment latency budget?

2. Data

Acquire, label, and store the dataset. This is typically the longest stage of any real project. Polyzotis et al. [4] survey the data-lifecycle challenges and identify schema management, versioning, and provenance as the top three.

3. Exploratory data analysis (EDA)

Look at the data before modelling. Plot every feature’s distribution, every pairwise scatter you can afford, and the target distribution. EDA is where you catch the wrong unit of measurement, the dataset that has 3% of rows duplicated, the population shift between train and serving, and the leak you would otherwise discover after deployment.

4. Feature engineering

Transform raw columns into model inputs: encoding, scaling, embedding, aggregating. For tabular and many time-series problems, feature engineering still beats more model complexity. For images and text with deep nets, the model usually does its own feature engineering.

5. Splitting

Choose a train/validation/test split that reflects how the model will be used. For tabular data: stratified random. For time-series: time-based. For groups (users, devices, patients): group-based. Splits are not optional and they are rarely random when time or grouping is involved. See Data & leakage for the long form.

6. Training

Fit the model. SGD or Adam over a chosen architecture, with a chosen loss and a chosen regulariser. This is the stage that gets the most textbook coverage and the least real-world time.

7. Evaluation

Measure performance on the held-out set, sliced by every dimension you care about (geography, demographic group, time of day). A globally good model that fails on a 5% slice is often a deal-breaker. See Evaluation metrics.

8. Deployment

Ship the model to a serving stack. Decisions here include batch versus online inference, which framework to serialise to (ONNX, TorchScript, native), how to A/B test against the previous model, and how to roll back when something breaks.

9. Monitoring

Once in production, the model is a moving target. You instrument:

  • Input drift: is today’s feature distribution still like training’s?
  • Prediction drift: is the model’s output distribution stable?
  • Performance drift: when ground truth eventually arrives, is online accuracy holding up?
  • Latency and error rates: ordinary infrastructure metrics still apply.

10. Retraining

When monitoring fires, you retrain. This can be a full retrain on a sliding window, an online incremental update, or a complete reframing of the problem. Most production teams run a scheduled retrain on a cadence (daily, weekly, monthly) plus on-demand retrains when alarms fire.

Research vs production

Research workflowProduction workflow
GoalBeat a benchmark by 0.5 points.Make a measurable business or product impact.
DatasetsStatic, public, well-defined (CIFAR, ImageNet, GLUE).Streaming, proprietary, noisy, drifting.
Iteration speedHours to days. Optimise for ideas tested per week.Days to weeks. Optimise for stability.
ReproducibilityReproduce a paper exactly.Reproduce yesterday's prediction; pin code+data+model+config.
MonitoringNone during training; eval at the end.Continuous: drift, freshness, latency, slice metrics, error rates.
DeploymentSubmit to a leaderboard or release a checkpoint.A/B-tested rollout, canary, rollback plan, SLOs.
StakeholdersReviewers, supervisors, the field.Product, ops, legal, customers, regulators.
Failure costA retracted paper.Money, trust, sometimes lives. Brittle pipelines amplify the cost.

Where projects actually fail

Problem framing is half the job

The single most common cause of failed ML projects is bad framing: optimising for an offline metric that does not move the business metric, scoring against labels that were generated by a process that does not match production, or solving a problem the user did not have. The fix is to spend more time with the team that owns the metric you care about than with the model itself.

Skipping EDA hides leakage and drift

Every horror story in Data & leakage was discoverable in EDA. Plotting target rate by date catches temporal leakage. Plotting feature distributions by class catches features that are downstream of the label. Skipping EDA to “save time” is the most expensive shortcut in the book.

Splits are not optional and are not random when time is involved

A random split on a time series is leakage by construction: you train on 2024 and test on 2023. The model predicts the past; the metric is meaningless. Use TimeSeriesSplit (sklearn) or write your own time-aware split, every time. Group-based splits are required when individual users, devices, or patients can appear in multiple rows; otherwise the model memorises identifiers.

Production-readiness checklist (Breck et al. 2017)

Before deploying, you should be able to point to:

  1. A documented training pipeline that any team member can reproduce.
  2. A schema for the input features with validation in CI.
  3. Held-out evaluation with sliced metrics, not just a single number.
  4. A canary deploy and a rollback plan.
  5. Drift alarms and a runbook for retraining.

The Breck et al. rubric [2] codifies 28 such checks across data, model, infrastructure, and monitoring.

What goes wrong AFTER deployment

Sculley et al. [3] and Huyen [5] compile a catalogue. The three most common are:

  1. Distributional shift. The training distribution drifts away from the serving distribution. Detected via input-feature drift alarms and (lagged) performance drop.
  2. Feedback loops. The model influences the data it sees: a recommender promotes items it predicts will be clicked, the user clicks them, and the next training round confirms the bias. Detected by counterfactual logging, fixed by exploration.
  3. Pipeline silent failures. A feature pipeline upstream of the model returns NaN, default, or stale values; the model produces predictions that look valid but are based on garbage. Detected via schema validation and feature-staleness monitors.

Interactive Demo

Click through the 7 stages of a real ML pipeline. Each stage shows a description, the common tools used there, and the most common pitfall that causes projects to fail at that stage.

ML Pipeline Explorer

Click any stage to see what happens there, the common tools used, and what typically goes wrong.

🎯

Problem Framing

Define the ML objective, business metric, and success criteria. Decide what to predict, for whom, and what action the prediction drives.

Common tools

Stakeholder interviewsNorth Star metric definitionBaseline heuristics

Common pitfall

Optimising the wrong metric - building a model that maximises offline accuracy while the business cares about revenue or retention.

What to try: Click any pipeline stage to learn what happens there and what can go wrong. Notice how failures at early stages (Problem Framing, Data Collection) tend to be harder to fix than failures at later ones.

Pop quiz

Check yourself
Score: 0 / 4(0 answered)
  1. 1. Which stage of the ML workflow is most often where projects actually fail?
  2. 2. What is the correct split strategy for a fraud-detection dataset where each user can appear in multiple rows over time?
  3. 3. A model deployed three months ago has a sudden 6-point AUC drop overnight. Most likely cause?
  4. 4. Which of these is NOT typically part of the production workflow?

Interview Q&A

Walk me through how you would scope a new ML project, before writing any model code.onsite

I would do five things, in order:

  1. Talk to the metric owner. What business or product number are we trying to move? What is the current value, what would success look like, and what is the cost of getting it wrong?
  2. Translate to an ML problem. Define inputs, outputs, an offline objective, and a connection between the offline metric and the business metric. If I cannot connect them, the framing is wrong.
  3. Audit the data. Where does it come from, who labels it, how often, what is the latency between event and label? If labels lag the prediction by weeks, online learning is off the table.
  4. Build the dumbest possible baseline. A constant predictor, a simple heuristic, or a logistic regression. The number it gets is the bar to beat. Many projects “succeed” by beating a baseline that does not exist; building it forces an honest measurement.
  5. Design the eval before training. Sliced metrics, A/B-testable lift, rollback plan. If I cannot evaluate the model offline in a way that predicts online behaviour, I scope the project differently.

This is paraphrased from Ng [1] and Huyen [5], both of whom argue that the modelling part is 10-20% of the work.

A model in production silently drifts. How do you detect it?phone

Three layers of monitoring.

  1. Input drift. Compare the current distribution of each feature to its training distribution using a statistical test (KS for continuous, chi-square for categorical) and alarm on p-value or population stability index above a threshold.
  2. Prediction drift. Track the distribution of model outputs over time. A sudden shift in the predicted-probability histogram is often the earliest signal that something is wrong.
  3. Performance drift (lagged). When ground truth eventually arrives, recompute online metrics on a rolling window. The lag is unavoidable but it is the gold standard.

For high-stakes systems, also run shadow predictions against the previous model and alarm on prediction disagreement above a baseline.

When does it make sense to retrain a model on a fixed schedule versus on-demand?research

Schedule when the data drifts predictably and the cost of stale predictions accumulates: ad CTR, recommendation systems, demand forecasting, anything tied to user behaviour. Daily or weekly retrains are common.

Retrain on-demand when drift is rare but catastrophic: medical imaging, fraud detection during a new fraud pattern, regulatory model updates. The trigger is a monitoring alarm, not a calendar.

In practice most teams do both: a slow schedule for hygiene and event-driven retrains for incidents. The tradeoff is operational complexity (more pipelines to maintain) versus latency to react. Polyzotis et al. [4] survey several patterns and conclude that most production teams converge on a hybrid.

References

  1. Ng, A. (2018). Machine Learning Yearning. deeplearning.ai. https://www.deeplearning.ai/machine-learning-yearning/
  2. Breck, E., Cai, S., Nielsen, E., Salib, M., and Sculley, D. (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. IEEE Big Data. doi:10.1109/BigData.2017.8258038
  3. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., and Dennison, D. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS 28. https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
  4. Polyzotis, N., Roy, S., Whang, S. E., and Zinkevich, M. (2018). Data Lifecycle Challenges in Production Machine Learning: A Survey. SIGMOD Record 47(2), 17-28. https://sigmodrecord.org/publications/sigmodRecord/1806/pdfs/04_vision_polyzotis.pdf
  5. Huyen, C. (2022). Designing Machine Learning Systems. O'Reilly. ISBN 978-1-098-10796-3
Math you needData & leakage