• Last time we explored data documentation and governance
• Datasheets for datasets, model cards for models, system cards for AI systems
• Data lineage: tracking where data comes from
• The consent problem in AI training data
• Today: Types of bias and how they arise
• This topic affects real people in real ways, as we’ll see
• Whose values does AI encode when it makes decisions?

Part 1: The big picture

• What is bias? (It’s complicated!)
• The mirror problem: AI reflects us
• Real-world harms: who gets hurt?

Part 2: Types of bias

• Historical bias: the past baked in
• Representation bias: who’s missing?
• Measurement bias: bad proxies
• Aggregation and evaluation bias

Part 3: Applying the framework

• Activity: Spot the bias!
• Where bias shows up (preview)

Part 4: The hard questions

• The impossibility theorem: can we be fair?
• Fairness tradeoffs: uncomfortable choices
• Is AI bias fixable?

https://gizmodo.com/a-dish-of-neurons-playing-doom-is-the-wildest-thing-ive-seen-in-ages-2000727674

Robert Julian-Borchak Williams, Detroit, 2020

• Arrested in front of his family
• Accused of stealing watches
• Held for 30 hours in jail
• Later released: wrong person

What happened?

• A facial recognition system matched his driver’s licence photo to grainy surveillance footage
• The algorithm was wrong
• Robert is Black. Research shows facial recognition has higher error rates for darker-skinned faces

This wasn’t a software bug. It was how the system was built

“Bias” means different things:

In AI, bias typically means:

A system that produces systematically unfair outcomes for certain groups of people.

But who defines “unfair”? That’s the hard part.

Is AI biased, or is it just showing us what we already are?

• AI learns from human-generated data
• Historical data contains historical discrimination
• If humans made biased decisions, AI learns those patterns
• And AI can amplify existing biases at scale

Example: Word embeddings

• “Man” is to “Doctor” as “Woman” is to… “Nurse”
• The AI learned this from millions of human texts
• It picked up our own sexism from the text

The scenario:

You’re applying to university. You get to choose who evaluates your application:

Option A: A human admissions officer

• Might favour applicants who remind them of themselves
• Could be swayed by mood, time of day, or fatigue
• You can appeal and read social cues
• Bias is inconsistent and unpredictable

Option B: An AI admissions tool

• Might penalise your ZIP code, name, or school
• Applies the same criteria to everyone, every time
• You cannot argue with it or explain context
• Bias is systematic and invisible

⏱️ 2 minutes

AI systems are making decisions about:

The scale problem:

• Algorithms decide faster than humans can review
• One biased system can affect millions of people
• Errors compound: a bad decision today feeds worse data into tomorrow’s model

This happens when past discrimination is baked into the training data, even if the data accurately reflects the real world at the time

The Amazon hiring case (2018):

• Amazon built an AI to screen CVs
• Trained on 10 years of past hiring decisions
• Past hiring was male-dominated
• AI learned: penalise CVs mentioning “women’s”
  • “Women’s chess club captain” → downgraded
  • All-women’s college → downgraded
• Amazon scrapped the tool

The data was “accurate”: it did reflect Amazon’s actual hiring. But that hiring was biased.

When certain groups are underrepresented in training data, the model performs poorly for them

Example: Voice assistants

• Voice recognition systems trained primarily on:
  • American and British accents
  • Male voices (often from tech employees)
  • Native speakers
• Result: Higher error rates for:
  • Non-native English speakers
  • Regional accents (Scottish, Indian, Nigerian)
  • Women’s voices in some systems
  • Children and elderly users

Why?

• Whoever collects the data determines who’s in it
• Convenience samples from developers themselves
• “Edge cases” are actually most of the world

Using a measurable proxy for something you actually care about… but the proxy doesn’t work equally for everyone

Example: ZIP code as a credit proxy

• Banks can’t legally use race to make loan decisions
• But they can use ZIP codes
• ZIP codes correlate strongly with race due to housing discrimination
• Result: A “race-neutral” variable that encodes race

Other problematic proxies:

Treating diverse populations as if they’re all the same, when the underlying relationships differ across groups

Example: Diabetes prediction

• A single model is trained on data from all patients
• But diabetes manifests differently across ethnicities:
  • Different genetic risk factors
  • Different dietary patterns
  • Different symptom presentations
• A single model may work well on average but poorly for specific groups
• Simpson’s paradox

The maths:

When the benchmark dataset used to test a model doesn’t represent the population it will be used on

The benchmark problem:

• Standard benchmarks become industry standards
• Everyone optimises for the same tests
• If the test is biased, success on the test means nothing

Example: ImageNet

• For years, the gold standard in computer vision
• But images were predominantly from the US and Europe
• Models trained and tested on it worked great… in the US
• Deployed globally: failures on everyday objects from other cultures

Sometimes a model works fine where it was built but fails when used in a different context.

Example: US model in India

• Credit scoring model trained on US financial data
• Deployed in India to assess loan applications
• Problem: Different banking systems, income patterns, credit histories
• Result: Inappropriate decisions for the new context

Example: COVID-19 detection

• Models trained on hospital data from one region
• Deployed in different regions with different equipment
• X-ray machines, demographics, disease prevalence all differed
• Performance dropped significantly

Tradeoff 1: Accuracy vs. Fairness

• Making predictions equally accurate across groups may reduce overall accuracy
• Who pays the cost?

Tradeoff 2: Individual vs. Group Fairness

• Treating individuals identically (blind to group) may produce unequal group outcomes
• Treating groups equally may disadvantage qualified individuals

Tradeoff 3: Transparency vs. Gaming

• Revealing how the algorithm works allows people to game it
• Keeping it secret prevents accountability

Tradeoff 4: Short-term vs. Long-term

• Using current data perpetuates historical inequalities
• Ignoring current data may reduce accuracy today

The optimists

• AI bias is a technical problem with technical solutions
• Better data, better algorithms, better audits
• AI might be less biased than humans
  • Humans are inconsistent; AI is at least consistent
  • AI decisions can be audited; human gut feelings cannot
• Datasets are improving, regulations are catching up

“A biased algorithm can be retrained. A biased hiring manager is harder to fix.”

The sceptics

• AI bias reflects societal problems that can’t be coded away
• “Fair AI” talk distracts from the root causes
• Who defines fairness? Usually those already in power
• AI obscures human accountability
• Some decisions shouldn’t be automated at all

“You can’t fix a discriminatory housing market by tweaking a credit-scoring model.”