• The pre-history of AI
• Birth of AI: Dartmouth 1956
• Symbolic AI and expert systems
• AI winters and why they happened
• The neural network comeback
• The data revolution
• Transformers explained simply
• From GPT to ChatGPT

• Humans have long dreamed of creating artificial life
• Ancient myths featured mechanical beings:
  • Talos (Greek): Bronze giant protecting Crete
  • Golem (Jewish folklore): Clay figure brought to life
  • Frankenstein (1818): Mary Shelley’s novel about creating life
• We’ve always been fascinated with replicating intelligence
• Can machines truly think, or only simulate thinking?

• Can intelligence be reduced to rules and symbols, or does it emerge from interconnected units?
• 1642: Blaise Pascal builds the Pascaline, a mechanical calculator for addition and subtraction
• 1673: Gottfried Leibniz creates a machine that can multiply and divide
• 1837: Charles Babbage designs the Analytical Engine, a programmable, general-purpose computer (never completed)
• Ada Lovelace writes what many consider the first computer program for Babbage’s machine
• Calculation could be separated from human minds

• For most of history, the word “computer” referred to a person, not a machine!
• Large teams of human computers performed complex calculations:
  • Astronomical tables
  • Ballistics trajectories
  • Census data
• Often women, who were paid less than male mathematicians
• This division of mental labour (Adam Smith!) showed that complex calculations could be broken into simple, mechanical steps
• Historian Lorraine Daston: Calculation shifted from “genius” to “merely mechanical”

• 1936: Alan Turing publishes “On Computable Numbers, with an Application to the Entscheidungsproblem” (decision problem)
• Introduces the concept of a universal machine that can compute anything computable
• 1950: Publishes “Computing Machinery and Intelligence” in Mind (philosophy journal)
  • Proposes the famous Turing Test: Can a machine fool a human into thinking it’s human?
  • Asks: “Can machines think?” and reframes it as a practical question (“Can machines do what we (as thinking entities) can do?”)
• Turing’s ideas laid the theoretical foundation for both computing and AI
• He suggested machines could learn, not just follow fixed rules

• Summer 1956: A workshop at Dartmouth College marks the birth of AI as a field
• Organised by John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon
• First use of the term “Artificial Intelligence”
• The founding conjecture:

“Every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it.”

• Attendees included Herbert Simon and Allen Newell
• Optimism was high: Many believed human-level AI was just decades away

Symbolic AI, 1956–1980s

• Intelligence = manipulating symbols according to rules
• Program the formal rules behind intelligent behaviour into machines
• Early successes: ELIZA (therapist, video here), SHRDLU (block world, video here)
  • Try ELIZA here: https://anthay.github.io/eliza.html (by the ELIZA Archaeology Project)
• Bold but overconfident predictions: Minsky and Simon expected human-level AI within a generation

Expert systems, 1970s–1980s

• Shift to narrow domains with expert knowledge

• Encode human expertise as if-then rules

• Examples: MYCIN (bacterial infections, video here), DENDRAL (chemical analysis, video here)

• Edward Feigenbaum: “The problem-solving power… is primarily a consequence of the specialist’s knowledge employed”

• Both approaches relied on hand-crafted knowledge, not learning from data

• Neither could learn or improve automatically

• 1943: McCulloch and Pitts propose artificial neurons
  • Simple binary units that mimic biological neurons
• 1958: Frank Rosenblatt invents the Perceptron
  • A simple neural network that could learn to classify patterns
  • Trained by adjusting weights based on errors
• The New York Times (1958) predicted it would “walk, talk, see, write, reproduce itself”
• Instead of programming rules, let the machine learn from examples
• However, single-layer perceptrons couldn’t solve certain simple problems, so interest waned
• Multi-layer networks could solve these problems, but training them was hard
• It took nearly 20 years for neural networks to make their comeback

• How does an artificial neuron work?
• Each input (\(x_i\)) (e.g., pixel brightness) is multiplied by a weight (\(w_i\)) to increase or decrease the importance of that signal, then summed with a bias (\(b\)) (a threshold that determines when the neuron “fires”): \(z = \sum x_i \cdot w_i + b\)
• The result passes through an activation function (non-linear transformation) to produce the output: \(y = f(z)\)
• Without the activation function, the network would be just a linear model!

• 1986: Rumelhart, Hinton, and Williams popularise backpropagation
• You can train multi-layer networks by propagating errors backwards through the layers
• It finds functions that are highly non-linear with very precise adjustments to many weights
• Finally, neural networks could learn complex patterns!
• Renewed interest in connectionism
• But challenges remained:
  • Limited computing power
  • Not enough training data
• Neural networks showed promise but couldn’t yet compete with other methods

• 2009: Google researchers (Halevy, Norvig, Pereira) publish a landmark paper called “The Unreasonable Effectiveness of Data”
• Simple models with lots of data beat complex models with less data
• Traditional AI: Design clever algorithms
• New AI: Feed massive amounts of data to simple learning algorithms

“Now go out and gather some data, and see what it can do.”

• This insight transformed the field
• Data became the new oil, together with computing power and better algorithms
• Any one of them alone wasn’t enough!

• ImageNet: A dataset of 14 million labeled images
• Annual competition: Classify images into 1,000 categories
• 2012: AlexNet (Krizhevsky, Sutskever, Hinton) wins by a huge margin
  • Error rate: 15.3% (next best: 26.2%)
  • Used deep neural networks
  • Trained on Graphics Processing Units (GPUs), which are better than Central Processing Units for parallel processing
• This was a great moment for deep learning!
• Within years, deep learning dominated computer vision
• Proved that neural networks + big data + GPUs = breakthrough

• 2017: Google researchers publish “Attention Is All You Need” (Vaswani et al.)
• Self-attention mechanism
• Instead of processing sequentially:
  • Each word can “look at” every other word directly
  • Computes relevance scores between all pairs
  • Parallel processing: Much faster to train
• This architecture powers GPT (Generative Pre-trained Transformer), BERT, and pretty much all modern AI
• Beyond text: Now used in images, audio, video, proteins, games…

• Text must be converted to numbers
• Tokenisation: Split text into tokens
  • Usually words, letters, or word pieces
  • “unhappiness” → “un”, “happiness”
  • GPT-2 has a vocabulary of 50,257 tokens
• Each token gets a unique ID number
• Think of it as creating a dictionary where each word/piece has a code

• Each token becomes a vector (a list of numbers)
• GPT-2: Each token → 768 numbers
• These vectors capture meaning:
  • Similar words are close together
  • “king” - “man” + “woman” ≈ “queen”
• Also add position information:
  • Word order matters in language!
  • “Dog bites man” ≠ “Man bites dog”
• These embeddings are learned during training

• The core mechanism that makes transformers powerful
• For each word, ask: “Which other words are relevant to understanding me?”
• Creates three vectors for each token:
  • Query (Q): “What am I looking for?” (does “it” refer to “animal” or “street”?)
  • Key (K): “What do I contain?” (check the characteristics of “it” as a token)
  • Value (V): “What information can I provide?” (provide the information: “ah, it’s animal!”)
• Attention score = how well Query matches Key
• Like a search engine where each word searches for relevant context

• After attention, each token passes through a feed-forward network (FFN, Multi-Layer Perceptron, MLP)
• Attention is about “looking around” to find connections between words
• FFN is about “thinking” about what those connections actually mean for each individual word
• This refines and transforms the representation
• Models stack many blocks:
  • GPT-2 (small): 12 blocks
  • GPT-3: 96 blocks
  • GPT-4: Rumoured to be much larger
• Each block builds more abstract understanding
  • Early layers: Grammar, syntax
  • Later layers: Meaning, context, reasoning

• Modern AI isn’t just about text anymore
• Multimodal models can process:
  • Text ↔︎ Images (DALL-E, Midjourney)
  • Text ↔︎ Audio (Whisper, ElevenLabs)
  • Text ↔︎ Video (Sora, Runway)
  • Text ↔︎ Code (Codex, Copilot)
• Same transformer architecture, different inputs/outputs
• Vision Transformers (ViT): Treat images as sequences of patches
• The boundaries between modalities are blurring

• Georgia Tech’s Transformer Explainer
• Interactive visualisation of how transformers work
• See attention patterns in real time
• Experiment with temperature and sampling
• Runs GPT-2 directly in your browser!
• Great for building intuition

https://poloclub.github.io/transformer-explainer