• Last time we explored data documentation and governance
• Datasheets for datasets, model cards for models, system cards for AI systems
• Data lineage: tracking where data comes from
• The consent problem in AI training data
• Today: Types of bias and how they arise
• Perhaps one of the most important topics in this course!
• AI is reshaping society, but whose values does it encode?

Part 1: The Big Picture

• What is bias? (It’s complicated!)
• The mirror problem: AI reflects us
• Real-world harms: Lives affected

Part 2: Types of Bias

• Historical bias: The past encoded
• Representation bias: Who’s missing?
• Measurement bias: Proxies gone wrong
• Aggregation and evaluation bias

Part 3: Applying the Framework

• Activity: Spot the bias!
• Where bias shows up (preview)

Part 4: The Hard Questions

• The impossibility theorem: Can we be fair?
• Fairness tradeoffs: Uncomfortable choices
• Is AI fixable, or do we need new approaches?

Robert Julian-Borchak Williams, Detroit, 2020

• Arrested in front of his family
• Accused of stealing watches
• Held for 30 hours in jail
• Later released: wrong person

What happened?

• A facial recognition system matched his driver’s licence photo to grainy surveillance footage
• The algorithm was wrong
• Robert is Black. Research shows facial recognition has higher error rates for darker-skinned faces

This wasn’t an AI bug per se, it was how the system was built

“Bias” means different things:

In AI, bias typically means:

A system that produces systematically unfair outcomes for certain groups of people.

But who defines “unfair”? That’s where it gets hard…

A provocative question:

Is AI biased… or is it just showing us what we already are?

The uncomfortable truth:

• AI learns from human-generated data
• Historical data contains historical discrimination
• If humans made biased decisions, AI learns those patterns
• AI can amplify existing biases at scale

Example: Word embeddings

• “Man” is to “Doctor” as “Woman” is to… “Nurse”
• The AI learned this from millions of human texts
• It’s reflecting our own sexism

Take a position:

If an AI hiring tool discriminates against women…

Who is responsible?

A. The company using the tool

B. The company that built the tool

C. The data scientists who trained it

D. The society that created biased data

E. All of the above—but in what proportion?

Think about:

• Who had the power to prevent this?
• Who benefits from using AI?
• Who bears the harm?

AI systems are making decisions about:

The scale is unprecedented:

• Algorithms make decisions faster than humans can review
• A single biased system can affect millions of people
• Errors compound: one bad decision leads to worse data for future decisions

What is it?

Historical bias occurs when past discrimination is baked into the training data, even if the data accurately reflects the real world at the time

The Amazon hiring case (2018):

• Amazon built an AI to screen CVs
• Trained on 10 years of past hiring decisions
• Past hiring was male-dominated
• AI learned: penalise CVs mentioning “women’s”
  • “Women’s chess club captain” → downgraded
  • All-women’s college → downgraded
• Amazon scrapped the tool

The irony: The data was “accurate”, it did reflect Amazon’s historical hiring. But that history was biased!

When certain groups are underrepresented in training data, the model performs poorly for them

Example: Voice assistants

• Voice recognition systems trained primarily on:
  • American and British accents
  • Male voices (often from tech employees)
  • Native speakers
• Result: Higher error rates for:
  • Non-native English speakers
  • Regional accents (Scottish, Indian, Nigerian)
  • Women’s voices in some systems
  • Children and elderly users

Why?

• Whoever collects the data determines who’s in it
• Convenience samples from developers themselves
• “Edge cases” are actually most of the world

Using a measurable proxy for something you actually care about… but the proxy doesn’t work equally for everyone

Example: ZIP code as a credit proxy

• Banks can’t legally use race to make loan decisions
• But they can use ZIP codes
• ZIP codes correlate strongly with race due to housing discrimination
• Result: A “race-neutral” variable that encodes race

Other problematic proxies:

Treating diverse populations as homogeneous when the underlying relationships differ across groups

Example: Diabetes prediction

• A single model is trained on data from all patients
• But diabetes manifests differently across ethnicities:
  • Different genetic risk factors
  • Different dietary patterns
  • Different symptom presentations
• A single model may work well on average but poorly for specific groups
• Simpson’s paradox

The maths:

When the benchmark dataset used to test a model doesn’t represent the population it will be used on

The benchmark problem:

• Standard benchmarks become industry standards
• Everyone optimises for the same tests
• If the test is biased, success on the test means nothing

Example: ImageNet

• For years, the gold standard in computer vision
• But images were predominantly from the US and Europe
• Models trained and tested on it worked great… in the US
• Deployed globally: failures on everyday objects from other cultures

What is it?

When a model is used in a different context than it was designed for.

Example: US model in India

• Credit scoring model trained on US financial data
• Deployed in India to assess loan applications
• Problem: Different banking systems, income patterns, credit histories
• Result: Inappropriate decisions for the new context

Example: COVID-19 detection

• Models trained on hospital data from one region
• Deployed in different regions with different equipment
• X-ray machines, demographics, disease prevalence all differed
• Performance dropped significantly

Tradeoff 1: Accuracy vs. Fairness

• Making predictions equally accurate across groups may reduce overall accuracy
• Who pays the cost?

Tradeoff 2: Individual vs. Group Fairness

• Treating individuals identically (blind to group) may produce unequal group outcomes
• Treating groups equally may disadvantage qualified individuals

Tradeoff 3: Transparency vs. Gaming

• Revealing how the algorithm works allows people to game it
• Keeping it secret prevents accountability

Tradeoff 4: Short-term vs. Long-term

• Using current data perpetuates historical inequalities
• Ignoring current data may reduce accuracy today

View 1: Optimists 🌟

• AI bias is a technical problem with technical solutions
• Better data, better algorithms, better audits
• AI might be less biased than humans
  • Humans are inconsistent, AI is consistent
  • AI can be audited, humans are opaque
• Progress is being made: datasets improving, regulations emerging

Key argument:

“A biased AI can be fixed. A biased human is much harder to change.”

View 2: Sceptics 🤔

• AI bias reflects societal problems that can’t be coded away
• “Fair AI” is a distraction from addressing root causes
• Who defines fairness? (Usually those in power)
• AI obscures human accountability
• Some decisions shouldn’t be automated at all

Key argument:

“Fixing AI bias without fixing society is like putting a bandage on a broken bone.”