Why ML differs from software engineering

Background

Traditional software encodes rules explicitly; ML infers rules from data
This inversion changes every phase of development: requirements, testing, debugging, and deployment
Popularized as a distinct discipline by Pedro Domingos (The Master Algorithm, 2015) and Andrew Ng

The Concept

In traditional software, developers write the logic. In ML, developers write the training procedure and the model learns the logic from examples. This makes ML outputs probabilistic, data-dependent, and hard to debug with conventional software tools.

How it Links

Motivates the entire ML workflow as distinct from software engineering
Explains why data quality matters more than code quality in many ML projects
Connects to MLOps, model monitoring, and the need for evaluation metrics beyond unit tests

Concrete Example

A sorting algorithm is deterministic: given the same input, it always produces the same sorted output. A spam classifier is probabilistic: its behavior depends on what training emails it saw, and it can be silently wrong on novel phishing styles it never encountered during training.

Prerequisites

History of AI & ML (recommended context)

Learning outcomes

Articulate the three axes (specification, testing, lifecycle) on which ML systems differ from traditional software.
Define CACE (Changing Anything Changes Everything) and give a concrete example.
List the categories of hidden technical debt unique to ML, per Sculley et al. 2015.
Decide, for a given system, whether the production-readiness checklist of Breck et al. 2017 applies.

TL;DR

Traditional software is specified, tested, and versioned at the code level; ML systems are specified by data and behaviour over a distribution, tested statistically, and versioned across code + data + model + config. The most expensive consequence is silent failure: an ML system can keep returning answers long after the world has shifted out from under its training distribution.

Intuition

The single-sentence framing

In software, you write the rules and the data flows through them. In ML, you specify the goal and the data writes the rules.

Karpathy [4] called this “Software 2.0”: instead of programming the function, you program the search space of functions and the objective, then let an optimiser pick a function out of that space. The implications cascade through every part of the engineering process.

Formal contrast

What changes when the rules are learned

Specification

In a payroll system, the spec is “compute gross pay = hours x rate, deduct taxes per Schedule B”. You can write a unit test for any single payroll example and a property test for invariants (net pay never exceeds gross pay). The spec is closed-form.

In a fraud detector, the spec is “flag transactions a human reviewer would call fraudulent”. The spec is a distribution, defined by examples a labeller has agreed are positive or negative. There is no closed form, only an empirical sample. Two consequences:

You cannot prove correctness; you can only estimate generalisation error on held-out samples.
The spec drifts because labellers, fraudsters, and customer behaviour all evolve. There is no single fixed truth.

Testing

Software tests are deterministic: same input, same expected output. ML tests are statistical performance over a distribution: precision >= 0.85 at recall >= 0.6 on the held-out evaluation set. A single failing example is not a bug; a 5-point drop in average AUC is.

Breck et al. [2] propose 28 specific tests in four categories: data, model, ML infrastructure, and monitoring. Their argument is that ML production readiness is a checklist, not a unit-test suite.

Versioning

Traditional software has one moving part: code. ML systems have at least four:

Code (training script, serving stack)
Data (training set, features, labels - the data is the spec)
Model (weights, architecture, hyperparameters)
Config (preprocessing parameters, decision thresholds, feature flags)

Reproducing yesterday’s prediction means snapshotting all four, not just the git SHA. Tools such as DVC, MLflow, and Weights & Biases exist precisely to fill this gap.

Failure modes

Traditional software fails loudly: a null pointer, an HTTP 500, a stack trace. ML fails silently. A model whose training distribution no longer matches production keeps returning predictions; only metrics monitored over time will catch the drift. Amodei et al. [3] formalise several silent failure modes (negative side effects, reward hacking, distributional shift) as “concrete problems in AI safety”.

Iteration loop length

A traditional code change is: write, run unit tests, deploy. Loop time: minutes.

An ML change can be: change a feature, retrain (hours to days), evaluate on held-out (minutes), shadow-deploy (days to weeks), A/B test against control (weeks). Loop time: weeks to months. This is why teams that drag tabular-software discipline into ML systems get blocked: the compounding cost of iteration is much higher.

Debuggability

When an ML model is wrong, the question “why did it predict this?” has no unique answer. Tools like SHAP and LIME approximate the answer, but they describe correlation in the model, not causation in reality. The honest debugging move is often “look at 50 misclassified examples” rather than “step through the code”.

Reproducibility

Floating-point summation order changes results across GPUs. Random seeds rarely fully control PyTorch behaviour because cuDNN kernels are non-deterministic by default. Re-running yesterday’s pipeline on today’s data with today’s library versions can produce a numerically different model. Best teams pin everything (data hash, library versions, hardware) and accept residual non-determinism as part of the spec.

Side-by-side

	Traditional SW	Classical ML	Modern AI / LLM
Specification	Closed-form: explicit rules in code.	Distribution: input/output pairs from a sample.	Open-ended: prompts, instruction tuning, preferences (RLHF).
Testing	Deterministic unit + integration tests.	Held-out metrics: accuracy, AUC, F1. Statistical regression checks.	Eval suites (MMLU, HumanEval), red-team prompts, hallucination rate. No ground truth for many tasks.
Versioning	Code only (git SHA).	Code + data + model + config + preprocessing.	All of the above + base-model checkpoint + system prompt + tokenizer + RLHF policy.
Failure mode	Crash, exception, wrong output (loud).	Silent: predictions stay valid-shaped while distribution drifts.	Silent + plausible: hallucination, prompt injection, jailbreak.
Determinism	Bit-exact under same inputs and version.	Often non-deterministic across hardware due to floating-point order and cuDNN.	Sampling-based (temperature > 0) by design; even greedy decoding can drift across GPU types.
Iteration loop	Seconds to minutes (test + deploy).	Hours to days (retrain + evaluate).	Days to weeks for fine-tuning; minutes for prompt iteration.
Debuggability	Stack traces, breakpoints, time-travel debugging.	Feature attributions (SHAP, LIME), error analysis, slice-based metrics.	Activation patching, prompt ablations, mech-interp - mostly research-only.
Reproducibility	Bit-exact across machines and time.	Approximate; pin data, code, env, seeds.	Hard: API providers update models silently; sampling is stochastic; weights drift with training.

The five hidden-debt patterns from Sculley et al. (2015)

Sculley and the Google ML team [1] argue ML systems carry forms of technical debt that have no analogue in traditional software. Five are worth memorising.

CACE: Changing Anything Changes Everything

Add a new feature, even a useful one, and the meaning of every weight in the model shifts. Tune one hyperparameter and your loss landscape moves. Drop one preprocessing step and your eval drops two points. ML modules cannot be encapsulated the way software modules can: the weights of a logistic regression depend on every input feature jointly, so any local change is global.

The practical consequence: A/B-test every change individually, even ones that “should be” trivial.

Entanglement

Features that look independent are not. Two correlated features (say, page-load-time and bounce-rate) train weights that depend on each other; remove one and the other’s weight is wrong. The Sculley paper recommends either accepting entanglement (treat the whole feature set as one indivisible unit) or paying the cost of full retraining whenever a single feature is added or removed.

Glue code

In production ML systems, the “real” ML code (model fit, predict) is typically less than 5% of the codebase. The other 95% is glue: feature pipelines, data validation, monitoring, deployment, A/B harnesses, model registries. Underestimating glue cost is the most common reason “we’ll just train a model” projects fail.

Pipeline jungles

Feature engineering in production tends to grow as a directed graph of “scrapes, joins, filters, aggregates” that nobody fully understands. Each new feature plugs into an existing pipeline; removing any one of them risks breaking a downstream consumer you do not know about. Sculley et al.’s advice: invest in clean feature stores and documented contracts before pipelines metastasise.

Dead experimental code paths

Every “let’s try this gating function” branch left in the codebase is a foot-gun. The Knight Capital incident (2012) - a $440M loss in 45 minutes from re-activating a dead code path - is the canonical example. ML systems are full of dead branches because experimentation is the work; periodic dead-code audits are a hygiene practice, not a luxury.

What this means in practice

Production-readiness checklist (paraphrased from Breck et al. 2017)

Before deploying an ML model, you should have positive answers to at least:

Data: feature schema validated; expected-vs-actual distribution checked nightly; labelling SLA known.
Model: metrics tracked per slice (not just globally); calibration measured; baselines documented.
Infrastructure: training and serving paths tested independently; rollback plan defined.
Monitoring: prediction distribution alarmed; data-freshness alarmed; canary requests automated.

The Breck rubric [2] grades each system on a 0-3 scale across 28 items.

When traditional-SW thinking applies

Many ML pipeline components are still ordinary software: REST endpoints, feature pipelines, schedulers, dashboards. Apply traditional software hygiene there - unit tests, type checks, deterministic behaviour. The point is not that ML is different on every line, but that the model layer has different rules and expects different discipline.

Pop quiz

Check yourself

Score: 0 / 4(0 answered)

1. What does CACE stand for, in the context of ML technical debt?
2. Which is the LEAST appropriate test to gate an ML model deployment?
3. A model has been in production for six months without code changes; AUC drops 4 points overnight. What is the most likely cause?
4. Approximately what fraction of a production ML codebase is the "actual" model fit/predict code, per Sculley et al. 2015?

Interview Q&A

A senior backend engineer joins your ML team and says testing should be deterministic. How do you respond?onsite

I would agree on the infrastructure layer (feature pipelines, serving, dashboards) and disagree on the model layer. Concretely:

Deterministic unit tests are appropriate for any code that does not depend on the trained weights: schema validation, feature transforms, business rules around the prediction.
The model itself cannot be unit-tested in the traditional sense because its outputs are statistical. Instead we need:
- Held-out evaluation with calibrated thresholds (AUC, precision/recall at operating points, calibration error).
- Distributional checks (feature drift, prediction drift) that fire on shift.
- Slice-based metrics (subgroup performance) so a global average does not mask localized failures.
The ML Test Score rubric [2] codifies this: 28 tests across data, model, infrastructure, monitoring. It explicitly does not assume bit-exact reproducibility.

The general rule: deterministic testing where it costs nothing, statistical testing where it does.

What versioning artifacts must a team snapshot to reproduce yesterday's prediction?phone

At least four:

Code (training script + serving binary, pinned by git SHA).
Data (training set hash, schema, feature store version).
Model (weights, architecture, hyperparameters).
Config (preprocessing params, decision threshold, feature flags, A/B assignment).

For LLMs add: base-model checkpoint, tokenizer, system prompt, RLHF policy. For systems calling external APIs, add the API version because providers can silently update models.

Without all of these, you can reproduce a similar prediction; you cannot reproduce that prediction.

Pick a real or hypothetical ML system and describe one concrete failure mode that would never appear in a traditional service.research

A search-relevance model trained on click-through data. The system has been in production for six months and AUC on weekly holdout has been stable. Then a marketing campaign drives a new user demographic to the site whose query distribution differs from the training data. The model’s AUC stays the same on its weekly holdout (because the holdout is sampled from the same training distribution!) but online relevance metrics tank.

This is a distributional shift failure that has no analogue in traditional software. The system did not crash, did not return malformed output, and even passed every offline test. What broke is the assumption that today’s traffic matches yesterday’s labels. Detecting it requires monitoring the input distribution and the prediction distribution, not just the output correctness on a fixed test set [3].

The fix is structural: alarm on input drift, refresh the holdout regularly, and refit on a sliding window. None of these are software-engineering practices in the traditional sense; they are systems-engineering practices invented for ML [5].

References

Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., and Dennison, D. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS 28. https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
Breck, E., Cai, S., Nielsen, E., Salib, M., and Sculley, D. (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. IEEE Big Data. doi:10.1109/BigData.2017.8258038
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). Concrete Problems in AI Safety. arXiv:1606.06565
Karpathy, A. (2017). Software 2.0. https://karpathy.medium.com/software-2-0-a64152b37c35
Huyen, C. (2022). Designing Machine Learning Systems. O'Reilly. ISBN 978-1-098-10796-3

History of AI & ML AI / ML / DL / GenAI map