History of AI & ML
- AI has gone through several boom-bust cycles since the 1950s, each shaped by a new class of algorithms
- Understanding the history explains why current architectures look the way they do
- Key eras: symbolic AI, connectionism, statistical ML, deep learning, foundation models
The history of AI/ML is the story of competing paradigms: rule-based reasoning vs learning from data, shallow vs deep representations, narrow vs general intelligence. Each wave added tools and exposed new limits.
- Explains why deep learning dominates today - scale + data + GPU compute converged around 2012
- Transformer architecture (2017) and RLHF (2022) are the latest chapters in this arc
- Historical failures (AI winters) caution against hype cycles and highlight which claims are well-founded
- Trace the line from Turing 1950 to GPT-4 in fewer than ten lemmas.
- Name the two AI winters and the specific result that caused each.
- Identify the three pillars (data, compute, algorithm) that enabled the deep-learning revival of the 2010s.
- Place any modern technique inside the symbolic / statistical / deep / foundation-model timeline.
TL;DR
The field oscillated between symbolic and connectionist research for half a century. The deep-learning era (2012-) and the foundation-model era (2017-) were enabled less by new ideas than by three quantitative changes: GPU compute, web-scale data, and the transformer [10].
Intuition
It is tempting to read the history of AI as a single line of progress. It is closer to the truth that the field has rediscovered the same handful of ideas under different names every twenty years, each time pushed forward by a quantitative change in compute or data rather than a qualitative leap in theory.
Three threads run through the timeline:
- Symbolic AI asks: how do we encode knowledge? It dominates 1956 to ~1990 and resurfaces in modern hybrid systems like retrieval-augmented generation.
- Statistical and shallow ML asks: how do we learn from data? It dominates ~1990 to ~2012 and is still the workhorse for tabular problems.
- Deep learning and foundation models ask: how do we learn representations? It dominates 2012 to today.
The two AI winters are the gaps between these waves: 1974-1980 (after symbolic AI hit combinatorial walls and the Lighthill report cut UK funding) and 1987-1993 (after expert systems collapsed under the maintenance burden of hand-written rules).
Timeline
- 1943McCulloch-Pitts neuronpaper
A logical model of a neuron as a threshold unit, kicking off the formal study of neural computation.
- 1950Turing testpaper
Turing reframes the question "can machines think?" as an imitation game.
- 1956Dartmouth workshopmilestone
McCarthy, Minsky, Rochester, Shannon coin "artificial intelligence" and launch the field.
- 1958Perceptronpaper
Rosenblatt builds the first trainable linear classifier in software and hardware.
- 1965ELIZA
Weizenbaum's chatbot at MIT mimics a Rogerian psychotherapist with simple pattern matching.
- 1969Perceptrons bookcritique
Minsky and Papert prove single-layer perceptrons cannot learn XOR; momentum stalls.
- 1973Lighthill reportwinter
UK government report criticises AI for failing to scale, triggering the first AI winter.
- 1974First AI winter beginswinter
Funding for symbolic AI dries up across DARPA and the UK research councils.
- 1980Expert systems boom
XCON at DEC saves an estimated $40M/year, kicking off a commercial expert-systems wave.
- 1986Backpropagation popularisedpaper
Rumelhart, Hinton, Williams in Nature show multi-layer nets can be trained end-to-end.
- 1987Second AI winter beginswinter
Lisp machine market collapses; expert systems prove too brittle to maintain.
- 1989LeNet / convnets on ZIP codespaper
LeCun et al. apply backprop to handwritten digit recognition for the US Postal Service.
- 1995Support vector machinespaper
Cortes and Vapnik formalise SVMs; kernel methods dominate the late 1990s.
- 1997LSTMpaper
Hochreiter and Schmidhuber introduce gated recurrent units to overcome vanishing gradients.
- 1997Deep Blue beats Kasparovmilestone
IBM's search-based chess engine defeats the world champion: a symbolic-AI peak.
- 2006Deep belief netspaper
Hinton and Salakhutdinov re-ignite deep nets with layer-wise unsupervised pretraining.
- 2009ImageNet releaseddata
Fei-Fei Li releases a 14M-image labeled dataset that becomes the deep-learning testbed.
- 2012AlexNetmilestone
Krizhevsky, Sutskever, Hinton win ILSVRC by 11 points; the deep-learning era begins in earnest.
- 2014Seq2seq + GANpaper
Sutskever et al. propose seq2seq learning; Goodfellow et al. propose generative adversarial nets.
- 2015ResNetpaper
He et al. show 152-layer residual networks reach superhuman ImageNet accuracy.
- 2016AlphaGomilestone
DeepMind defeats Lee Sedol 4-1 with deep RL + Monte Carlo tree search.
- 2017Transformerpaper
Vaswani et al. replace recurrence with attention; sequence modelling is rewritten.
- 2018BERT and GPT-1paper
Pre-training + fine-tuning becomes the standard NLP recipe.
- 2020GPT-3paper
Brown et al. show 175B-parameter language models do in-context learning at scale.
- 2022ChatGPTmilestone
RLHF-tuned GPT-3.5 reaches 100M monthly users in two months; AI enters the mainstream.
- 2023GPT-4 and diffusion-everywheremilestone
Multimodal foundation models, Stable Diffusion, and open-weights LLMs (Llama 2) define the year.
Eras compared
| Symbolic AI | Statistical ML | Deep Learning | Foundation models / LLMs | |
|---|---|---|---|---|
| Time | 1956-1990 | 1990-2012 | 2012-2017 | 2017-today |
| Representative method | expert systems, SAT solvers, planning | SVMs, random forests, boosting | CNNs, RNNs, ResNets | transformers, RLHF, diffusion |
| Compute regime | CPU, kilobytes | CPU clusters, gigabytes | GPU, terabytes | GPU/TPU pods, petabytes |
| Data regime | hand-curated rules | thousands to millions of labels | millions of labels | web-scale unlabeled corpora |
| Killer application | Deep Blue, XCON, MYCIN | spam filters, ad CTR, PageRank | ImageNet, speech recognition | ChatGPT, Copilot, Stable Diffusion |
| Limit that ended the era | combinatorial blow-up, brittleness | feature engineering ceiling | architecture proliferation | hallucinations, alignment, cost (open question) |
The 1958 hype cycle
The day after Rosenblatt’s demo at the Office of Naval Research, the New York Times reported:
“The Navy revealed the embryo of an electronic computer today that it expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence… Dr. Frank Rosenblatt, a research psychologist at the Cornell Aeronautical Laboratory, Buffalo, said Perceptrons might be fired to the planets as mechanical space explorers.”
The actual Mark I Perceptron [4] was a 400-photocell letter classifier. The gap between what was demonstrated and what was promised in the press is a pattern that recurs every fifteen years (1958, 1973, 1987, and arguably 2023).
The two AI winters
Why each winter happened
First winter: 1974-1980
Three forces converged.
- Combinatorial explosion. Symbolic-AI methods scaled poorly. SHRDLU (Winograd 1971) handled 50 objects in a blocks-world; extending it to a real room would have multiplied the search space by orders of magnitude.
- The Minsky-Papert critique. The 1969 Perceptrons book [5] demonstrated that single-layer perceptrons could not represent XOR, and the (then) folkloric extension to multi-layer nets was that nobody knew how to train them. Connectionism stalled for 17 years until backpropagation was popularised [6].
- The Lighthill report. James Lighthill’s 1973 report to the UK Science Research Council concluded AI had not delivered on its promises and recommended cutting funding. DARPA followed with sharp cuts in the United States.
Second winter: 1987-1993
Expert systems looked like they had escaped the first winter, but two things broke them.
- Maintenance cost. A production expert system needed thousands of hand-written rules; rule sets accumulated contradictions and required full-time knowledge engineers. XCON, the poster child, eventually contained ~10,000 rules and could not be ported off DEC hardware.
- The Lisp machine collapse. Symbolics, Lisp Machines Inc., and Texas Instruments had built specialised hardware for symbolic AI. By 1987, commodity workstations from Sun and others ran Lisp at comparable speeds for an order of magnitude less money, and the specialised market evaporated almost overnight.
Why we are not in a third winter (yet)
Today’s models are wrong in well-known and serious ways: hallucinations, prompt injection, alignment, cost. But they pass the bar exam, write working code, and translate between 100 languages. The economic floor is much higher than after either previous winter, even if the ceiling proves lower than current excitement suggests.
The three pillars of the deep-learning revival
Backpropagation was published in 1986. Convolutional networks were published in 1989. LSTMs were published in 1997 [8]. Why did deep learning erupt in 2012? Three quantitative changes lined up:
- Compute. A consumer GPU (GTX 580) gave Krizhevsky et al. [9] ~3 TFLOPS for a few hundred dollars - roughly 100x what a CPU at the same price provided.
- Data. ImageNet released 1.4M labeled images in 2009; before that, vision benchmarks topped out at tens of thousands.
- Algorithms. ReLU, dropout, batch normalisation, and the realisation that depth without vanishing gradients was achievable - each shaved an order of magnitude off training cost.
LeCun, Bengio, and Hinton [11] made this argument explicitly in their Nature review: the science was in place by the late 1990s; the engineering caught up around 2012.
Why this matters for practice
Every modern technique is a descendant of an older one.
- Transformers [10] generalize the attention mechanism Bahdanau introduced in 2014, which itself was a learned alternative to the alignment used in IBM Model 1 from the 1990s.
- In-context learning rediscovers, in a soft form, the case-based reasoning of 1980s symbolic AI.
- RAG and tool use revive symbolic AI’s commitment to external knowledge bases inside a connectionist substrate.
- Diffusion models generalise denoising autoencoders (Vincent 2008) into a multi-step denoising process.
Reading the literature with the timeline in mind makes “new” papers easier to triage: most variations are quantitative, not qualitative.
Pop quiz
- 1. Which event is most directly credited with triggering the first AI winter?
- 2. What three quantitative changes enabled the deep-learning revival of the 2010s?
- 3. Which paper, in retrospect, marks the beginning of the foundation-model era?
- 4. Why did expert systems fail in the late 1980s?
Interview Q&A
Walk me from McCulloch-Pitts (1943) to GPT-4 (2023) in under three minutes.onsite
A reasonable thirty-second-per-decade outline:
- 1943-1958. McCulloch-Pitts [1] formalises the neuron as a threshold unit. Turing [2] reframes “can machines think” as an imitation game. Dartmouth 1956 [3] coins “AI”. Rosenblatt builds the perceptron [4].
- 1969-1986. Minsky-Papert [5] stall connectionism. Symbolic AI dominates until expert-systems brittleness shows. Backprop is published [6], but compute and data are still tiny.
- 1989-2006. LeNet [7] shows convnets work on real data. SVMs, random forests, and boosting take over the 1990s. LSTMs [8] arrive. Hinton’s 2006 deep belief nets restart neural research.
- 2012-2017. AlexNet [9] wins ImageNet by 11 points and the deep-learning era starts in earnest. ResNet, GANs, seq2seq follow.
- 2017-today. Transformers [10] replace recurrence; BERT and GPT scale them. GPT-3 (2020), ChatGPT (2022), and GPT-4/multimodal foundation models (2023) define the present.
The honest summary: most modern ideas were known by 1997. The compute, data, and a few engineering tricks made them work twenty years later [11].
What were the two AI winters and what specifically caused each?phone
First AI winter (1974-1980). Caused by (a) combinatorial explosion in symbolic search, (b) the Minsky-Papert proof that single-layer perceptrons cannot learn XOR, which discouraged neural-net research for 17 years, and (c) the Lighthill report cutting UK funding, with DARPA following in the US.
Second AI winter (1987-1993). Caused by (a) maintenance cost of expert systems - rule sets bloated to thousands of brittle, contradictory entries, and (b) the collapse of the Lisp machine market when commodity workstations caught up.
The general lesson is that AI winters follow gaps between hype and delivery, not failures of any single technique.
Which historical idea most influenced modern transformers, and how?research
The most direct ancestor of attention is the alignment model in statistical machine translation (IBM Model 1, 1990s), which assigned probabilities of source words generating target words. Bahdanau, Cho, and Bengio (2014) replaced the discrete alignment with a learned soft attention over an encoder’s hidden states. Vaswani et al. [10] generalised this: removed the encoder-decoder split, used self-attention everywhere, added positional encodings to recover order, and scaled.
The deeper lineage:
- McCulloch-Pitts [1] gave us the unit.
- Backprop [6] gave us the training rule.
- LSTMs [8] gave us long-range dependencies via gating.
- Attention removed the bottleneck of fixed-size hidden state in seq2seq.
- Transformers removed recurrence entirely.
So the transformer is connectionism + soft alignment + a willingness to throw recurrence away.
References
- McCulloch, W. S. and Pitts, W. (1943). A Logical Calculus of the Ideas Immanent in Nervous Activity. Bulletin of Mathematical Biophysics 5, 115-133.
- Turing, A. M. (1950). Computing Machinery and Intelligence. Mind LIX(236), 433-460. doi:10.1093/mind/LIX.236.433
- McCarthy, J., Minsky, M. L., Rochester, N., and Shannon, C. E. (1955). A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence. Dartmouth College. http://jmc.stanford.edu/articles/dartmouth/dartmouth.pdf
- Rosenblatt, F. (1958). The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain. Psychological Review 65(6), 386-408. doi:10.1037/h0042519
- Minsky, M. and Papert, S. (1969). Perceptrons: An Introduction to Computational Geometry. MIT Press. ISBN 0-262-13043-2
- Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature 323, 533-536. doi:10.1038/323533a0
- LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. (1989). Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation 1(4), 541-551. doi:10.1162/neco.1989.1.4.541
- Hochreiter, S. and Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation 9(8), 1735-1780. doi:10.1162/neco.1997.9.8.1735
- Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. NeurIPS 25. https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. NeurIPS 30. arXiv:1706.03762
- LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. Nature 521, 436-444. doi:10.1038/nature14539