MLOps

Background

MLOps emerged ~2017 as ML systems at scale exposed the need for DevOps practices applied to ML pipelines
Hidden Technical Debt in Machine Learning Systems (Sculley et al. 2015) documented 12 categories of ML technical debt
Kubeflow, MLflow, and SageMaker Pipelines are the dominant open-source and cloud MLOps platforms

The Concept

MLOps applies software engineering and DevOps practices to ML: version-controlled data and models, automated training pipelines, CI/CD for model deployment, monitoring for data drift and model degradation, and reproducible experiments. It is the discipline that turns research notebooks into production systems.

How it Links

Connects to the ML workflow - MLOps automates and governs every workflow stage in production
Data drift monitoring connects to covariate shift and the statistical tests for distribution change
Feature stores connect to feature engineering and data pipeline engineering

Concrete Example

A fraud detection model: without MLOps it degrades silently as transaction patterns shift. With MLOps: daily data ingestion pipeline, weekly retraining trigger, A/B testing framework comparing old vs new model, drift detector that alarms when feature distributions shift, model registry versioning every trained artifact with its training data hash.

Prerequisites

The ML Workflow, Data and Leakage, Evaluation Metrics

Learning outcomes

Identify the major categories of hidden technical debt in ML systems (CACE, glue code, pipeline jungles, dead experimental codepaths) from the Sculley et al. 2015 taxonomy.
Design an experiment tracking setup and explain what artifacts must be versioned to reproduce a training run.
Distinguish the four monitoring axes (data drift, prediction drift, concept drift, label drift) and state which is observable without ground-truth labels.
Describe shadow deployment, canarying, and A/B testing and state when each is appropriate for a model release.
Apply the Breck et al. 2017 ML Test Score checklist as a production-readiness rubric.

TL;DR

MLOps is the set of engineering practices that make ML systems reproducible, monitorable, and safely deployable. Sculley et al. [1] identified that most ML technical debt is not in the model code but in the surrounding infrastructure: training pipelines, feature engineering, data dependencies, and monitoring. The Breck et al. [2] ML Test Score operationalises this as a 28-point rubric. The key operational insight is that models degrade silently: inputs change, the world changes, and model accuracy falls without any software error. Monitoring, staged rollouts (shadow, canary, A/B), and automated retraining pipelines are the engineering response.

Intuition

Traditional software fails loudly: a bug throws an exception or returns a wrong answer traceable to a line of code. ML systems fail silently: the model continues to return predictions, but the predictions are gradually less accurate as the distribution of inputs shifts away from the training distribution. A deployed model can be subtly broken for weeks before anyone notices, if monitoring is absent.

The second challenge is reproducibility. A model trained by one engineer may not be reproducible by another if the random seed, library version, feature computation, or data split was not recorded. Production ML requires treating training as a versioned artifact: inputs, code, hyperparameters, and outputs must all be pinned so that any run can be reproduced.

ML systems are mostly not ML

The Sculley et al. [1] paper opens with the observation that in a typical ML system, the actual model training code is a small box surrounded by a much larger infrastructure: data collection, feature engineering, configuration, resource management, analysis tools, process management tools, serving infrastructure, and monitoring. Improving the model code yields diminishing returns if the surrounding infrastructure is brittle. Most production ML failures are infrastructure failures, not modelling failures.

Hidden technical debt

Sculley et al. [1] introduced the concept of ML-specific technical debt: patterns in ML systems that appear harmless but accumulate maintenance cost over time.

CACE principle. Changing Anything Changes Everything. In ML, every component is entangled: changing the input features changes the model’s learned representations; changing the training data distribution changes performance on held-out distributions; changing the downstream serving environment can silently change which inputs the model sees. This entanglement makes it difficult to understand the effects of any single change.

Glue code. ML systems frequently use general-purpose libraries (scikit-learn, PyTorch, HuggingFace) that do not natively fit together. The code that glues them together---format converters, data loaders, preprocessing pipelines---accumulates silently and is rarely tested. This glue code is often undocumented and fragile.

Pipeline jungles. Incremental feature engineering creates complex branching data preparation pipelines. Different features are extracted by different scripts written at different times. No one understands the full pipeline from raw data to model input. Dead branches, duplicate computations, and format incompatibilities accumulate over months.

Dead experimental codepaths. Every model experiment leaves behind commented-out code, unused flags, and conditional branches. Over time the production code becomes cluttered with “if we ever want to try X” branches that are never cleaned up and may interact badly with new features.

Undeclared consumers. If a model’s output format or semantics changes, downstream consumers may break silently. Without a clear contract between the model and its consumers, any update to the model is a potentially breaking change.

The MLOps lifecycle

2015Hidden Technical Debt (Sculley et al.)paper
First systematic taxonomy of ML-specific technical debt. Identifies CACE, glue code, pipeline jungles, undeclared consumers. Sets the conceptual foundation for what would become MLOps.
2017ML Test Score (Breck et al.)paper
28-point rubric for production-readiness across four axes: features and data, model development, ML infrastructure, monitoring. Operationalises the Sculley debt taxonomy.
2018MLflow (Databricks)system
Open-source platform for experiment tracking, model packaging, registry, and deployment. Standardises the logging of params, metrics, and artifacts across ML frameworks.
2019Feast (feature store), W&Bsystem
Feast open-sources the feature store concept (online + offline feature serving from a shared registry). Weights & Biases popularises cloud-hosted experiment tracking with rich visualisation.
2020Kubeflow, SageMaker Pipelinessystem
Kubernetes-native and managed cloud ML pipelines become mainstream. End-to-end pipelines from data ingestion to serving, with DAG orchestration.

Experiment tracking

Every training run should log: hyperparameters, dataset version and split seed, model architecture, training curve (loss and metrics per epoch), final evaluation metrics, model artifacts (weights, tokenizer), and environment (library versions, hardware). Without this log, the best model found during exploration is irreproducible and the performance difference from a later run cannot be attributed to any specific change.

Tools: MLflow (open-source, self-hosted, integrates with most frameworks), Weights & Biases (cloud-hosted, richer UI, sweep orchestration), TensorBoard (TensorFlow/PyTorch, metrics only, no artifact registry).

Key principle: log everything to the tracking server from the start of the project, even during prototyping. Retrofitting logging to a mature codebase is expensive and often incomplete.

Feature stores

A feature store decouples feature engineering from model training and serving. Features are computed once, stored in both an offline store (a data warehouse or lakehouse, e.g., BigQuery, Snowflake) for training and an online store (a low-latency key-value store, e.g., Redis, DynamoDB) for serving. Models read features from the same registry in both environments, eliminating training-serving skew: the most common source of production ML failures.

The key discipline: every feature used by a production model is versioned in the feature store. If a feature computation changes, the old version remains available for models trained on it, and the new version is available for future models.

Model registry

A model registry stores model versions with their metadata: training dataset version, hyperparameters, evaluation metrics, and deployment history. The registry enforces a promotion workflow: models move through stages (staging, production, archived) with approval gates. No model should be deployed to production without a registry entry, which provides the audit trail needed for compliance and debugging.

CI/CD for ML

Continuous integration for ML adds data validation and model validation steps to the standard code-CI pipeline.

A minimal ML CI pipeline on a pull request should:

Run unit tests on the pipeline code.
Run data validation: check that the training data schema matches expectations, that no critical features have shifted significantly, and that no label leakage is detectable.
Run a training dry-run on a small data slice to confirm the pipeline executes end-to-end.
Evaluate the new model on a fixed holdout set and compare to the current production model (champion/challenger test).

Only if all gates pass does the PR merge and the model advance to staging.

The four monitoring axes

Monitoring a deployed ML model requires tracking four distinct phenomena, each detectable at different times.

Drift type	What changes	Observable without labels?	Latency to detect	Response
Data drift (covariate shift)	Distribution of input features P(X) shifts	Yes	Immediate (next request)	Retrain; alert; investigate data pipeline
Prediction drift	Distribution of model outputs P(Y_hat) shifts	Yes	Immediate	May indicate data drift or model issue; investigate
Concept drift	True relationship P(Y\|X) changes	No (needs ground truth)	Days to weeks (depends on label latency)	Retrain on recent data; recalibrate
Label drift	Distribution of true labels P(Y) shifts	No (needs ground truth)	Days to weeks	Retrain; update class weights

Data drift and prediction drift are observable in near-real-time from the request stream. Concept drift and label drift require ground truth labels, which may arrive hours to weeks after the prediction (e.g., a credit model gets ground truth labels when a loan is repaid or defaulted, months later).

Population Stability Index (PSI) is the standard measure for feature drift:

\text{PSI} = \sum_i (p_i - q_i) \ln\frac{p_i}{q_i},

where $p_i$ and $q_i$ are the proportions of samples in bin $i$ for the reference and monitoring periods. PSI below 0.1 indicates no drift; 0.1-0.2 is moderate drift; above 0.2 indicates significant shift requiring investigation.

Deployment patterns

Pattern	User traffic to new model	Risk	Rollback	Best for
Shadow deployment	None (new model gets copies of live traffic, results not served)	Zero	Instant	Validating latency and correctness before any exposure
Canary release	Small fraction (1-5%)	Very low	Instant (route all traffic back)	Validating new model on real users before full rollout
A/B test	Fixed split (e.g., 50/50 or 90/10)	Low	Fast	Measuring business metrics (CTR, revenue) with statistical significance
Blue-green deployment	100% switched over	Medium (all-or-nothing)	Instant (keep old version running)	Zero-downtime version cutover after canary validation
Multi-armed bandit	Dynamic (based on observed rewards)	Medium	Adjust allocation	Online ad serving, recommendation, real-time optimisation

Shadow deployment is the correct first step for any major model change: run the new model on production traffic, log its outputs, but serve only the old model’s outputs to users. This validates latency (the new model can handle production request rates), correctness (predictions look reasonable on real inputs), and the absence of regressions before any user is affected.

Canary release follows shadow: 1-5% of traffic sees the new model. Monitor key business metrics (error rate, downstream conversion) for 24-48 hours. If metrics are within acceptable bounds, gradually increase the percentage (10%, 25%, 50%, 100%). Automate rollback: if P99 latency exceeds a threshold or error rate spikes, route all traffic back to the old model immediately.

A/B testing is a controlled experiment to measure the causal effect of the model change on business metrics. It requires a power calculation to determine the sample size needed for the desired effect size and significance level ( $\alpha = 0.05$ , $\beta = 0.2$ are typical). Common pitfalls: novelty effects (users behave differently on new features for the first few days), network effects (A/B treatment leaks between users who interact), and peeking (stopping the experiment early when metrics look good inflates Type I error).

The ML Test Score

Breck et al. [2] proposed a 28-point ML Test Score rubric across four domains:

Features and data (9 tests): schema validation on all inputs, test data and training data drawn from same distribution, no feature leakage, features important enough to justify engineering cost, feature generation is tested and versioned, data pipeline is monitored for anomalies.

Model development (5 tests): offline performance is tracked against a baseline, model is not overtrained on the test set, hyperparameter search is reproducible, model is compared to a simple rule-based baseline.

ML infrastructure (5 tests): model can be reproduced from a fixed training code version + data version + config, the serving code is tested, the model can be served within latency SLA.

Monitoring (9 tests): prediction distribution is monitored, feature distributions are monitored, model is retrained when drift is detected, business metric is tracked end-to-end.

A score of 0-4 is typical at initial deployment. A score of 12+ indicates a production-grade ML system. Most teams should aim for 12+ before calling a model “production-ready”.

Pitfalls

Training-serving skew

Training-serving skew is the single most common production ML failure: the features computed at serving time differ from the features computed during training, because the two codepaths have diverged. The fix is a feature store that computes each feature exactly once and serves it from both the offline store (for training) and the online store (for serving). If a feature store is unavailable, at minimum write a single compute_feature() function called in both paths and add a regression test that checks numerical equality on a fixed example.

Drift without ground truth

Detecting data drift is easy (no labels needed). Detecting model degradation from concept drift requires ground truth labels, which may arrive long after the prediction. If your label latency is measured in weeks, you may not know your model has degraded until significant damage has been done. Mitigations: monitor surrogate signals that correlate with ground truth (CTR as a proxy for relevance, user complaints as a proxy for quality), and establish a human evaluation sampling pipeline for spot-checking.

The CACE trap

Every ML experiment should change exactly one variable at a time: same data, same hyperparameters, different architecture; or same architecture, different training data. Changing multiple things at once makes it impossible to attribute performance differences to any specific change, and over time your experiment log becomes uninterpretable. Enforce this with your experiment tracking tool: require a diff against the parent run whenever logging a new run.

Pop quiz

Check yourself

Score: 0 / 4(0 answered)

1. Which type of drift can be detected immediately from the request stream, without any ground-truth labels?
2. The CACE principle in ML technical debt means:
3. Shadow deployment differs from canary release in that:
4. According to Breck et al. 2017, what ML Test Score indicates a production-grade system?

Interview Q&A

A model's production performance has degraded over 3 months. How do you diagnose which type of drift is responsible?onsite

Work through the four axes in order of observability.

First, check data drift (no labels needed): compute PSI for each input feature against the training distribution. If any feature has PSI above 0.2, that feature’s distribution has shifted significantly. Also check for missing values, new categories, or format changes in the input pipeline.

Second, check prediction drift (no labels needed): plot the distribution of model output scores over time. If the score distribution has shifted, that is consistent with data drift or concept drift.

Third, gather ground truth labels (may require a sampling pipeline or delayed feedback) and check concept drift: compute accuracy, AUC, or calibration on recent labelled examples. If performance is significantly below the baseline, the relationship $P(Y|X)$ has changed.

Finally, check label drift: has the base rate of the target class changed? A shift in base rates can explain reduced AUC even without concept drift.

Remediation: for data drift, investigate the data pipeline and retrain on recent data. For concept drift, retrain with a sliding window of recent labelled examples and consider sample reweighting. For label drift, recalibrate the threshold and update class weights.

Design an A/B test to decide whether to deploy a new recommendation model. What statistical design decisions must you make?staff

Effect size and power. Specify the minimum detectable effect (MDE): the smallest improvement in your primary metric (e.g., 1% relative increase in CTR) that would be business-significant. Run a power calculation: with significance level $\alpha = 0.05$ and power $1 - \beta = 0.80$ , the required sample size per variant is $n \approx 16 \sigma^2 / \delta^2$ , where $\sigma^2$ is the variance of the metric and $\delta$ is the MDE.

Randomisation unit. Assign users (not requests) to variants, so the same user always sees the same model. Request-level randomisation creates within-user contamination (a user experiences both models) that violates the SUTVA assumption.

Holdout period. Run the experiment long enough to capture weekly cycles (at least 2 weeks) and wait for novelty effects to decay (typically 3-5 days). Do not peek at interim results and stop early; this inflates Type I error.

Primary vs guardrail metrics. Define one primary metric (CTR, revenue, engagement) and several guardrail metrics (latency P99, error rate, user complaints). The experiment succeeds only if the primary metric improves AND all guardrail metrics stay within acceptable bounds.

Analysis. Use a two-tailed t-test or Mann-Whitney U for the primary metric. Apply Bonferroni correction if testing multiple metrics. Report the confidence interval on the effect size, not just the p-value.

Avoid. Do not run simultaneous A/B tests on overlapping user populations without orthogonal allocation. Do not change the model during the experiment. Do not stop early on a positive result.

References

Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., and Dennison, D. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS 2015. https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
Breck, E., Cai, S., Nielsen, E., Salib, M., and Sculley, D. (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. IEEE BigData 2017. doi:10.1109/BigData.2017.8258038
Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press. https://www.deeplearningbook.org

LLM Serving & Inference Evaluation & Benchmarks