• Last time we covered types of bias in AI systems
• Bias can enter at every stage: data collection, labelling, model design, deployment
• Historical bias: past discrimination baked into training data
• Representation bias: some groups underrepresented
• Measurement bias: proxies that correlate with protected attributes
• Fairness has multiple definitions that can conflict
• Today: how AI is used in healthcare and finance, and where bias shows up

Part 1: AI in healthcare

• Traditional ML vs LLMs in medicine
• Real deployments and RAG in clinical settings

Part 2: AI in finance

• ML for trading, credit scoring, and fraud
• LLMs, RAG, and agentic workflows

Part 3: When bias creeps in

• The Optum algorithm and credit scoring
• RAG-specific risks in both fields
• Patterns that repeat across domains

Part 4: What can we learn?

• Questions to ask about any AI system
• Group discussion

You will work in groups of 4-5 students to explore how AI tools perform in a real-world domain

Choose one of seven domains:

• Healthcare (diagnosis, treatment, patient care)
• Education (tutoring, assessment, feedback)
• Finance (investing, fraud, credit)
• Law (legal research, contracts, compliance)
• Creative industries (writing, art, music)
• Scientific research (literature review, hypothesis generation)
• Customer service (support bots, complaint handling)

No programming required: the focus is on evaluation and design

More information here: https://github.com/danilofreire/datasci185/blob/main/project/project-instructions.pdf

ML has been used in medicine for years, mostly for tasks with labelled data and a clear outcome (supervised learning):

• Medical imaging: CNNs detect tumours in X-rays, mammograms, retinal scans. Some match or beat radiologists
• Risk scoring: ML models predict readmission, sepsis, mortality. Hospitals use these to allocate resources
• Drug discovery: ML screens millions of molecules against protein targets, saving years in early development
• Electrocardiogram analysis: Deep learning flags arrhythmias from wearables in real time

Give the model structured input, get a classification or a number out

LLMs handle tasks that involve reading, writing, and reasoning about text:

• Clinical note summarisation: Pull out key medications, diagnoses, and follow-up instructions in seconds
• Patient-facing chatbots: Explain diagnoses in plain language, answer follow-up questions, triage symptoms
• Literature synthesis: Summarise findings across dozens of papers on a condition
• Clinical trial matching: Match unstructured patient records against trial eligibility criteria (previously done by hand)
• My suggestion: DeepMind’s MedGemma, a free and open-source LLM for medical tasks. Optimised for medical text and image comprehension

The key difference from traditional ML: these tasks need language comprehension, not number-crunching

Over 150 US health systems use AI-drafted messages via Epic’s MyChart (GPT-4):

• Patient sends a question about medication side effects
• System reads the patient’s record and drafts a response
• Clinician reviews and sends. Saves ~30 seconds per message

Google’s Med-PaLM 2 scored 86.5% on US Medical Licensing Exam questions. Physicians preferred its open-ended answers over those from other doctors

But passing exams ≠ treating patients. The model cannot examine the patient, read body language, or pick up on context outside the charts

LLMs are already used to talk to patients:

• Multilingual support: Translate discharge instructions into a patient’s language without a dedicated translator
• After-visit summaries: Plain-language version of what happened and what to do next
• Symptom triage: Chatbots recommend whether to see a doctor, go to A&E, or stay home

John Muir Health: clinicians using AI-assisted charting saved 34 min/day on documentation. Physician turnover dropped 44%. More here

Liability remains unresolved. If an AI chatbot gives bad advice and a patient is harmed, who is responsible?

Finance adopted ML early, and for good reason: there is a lot of structured data and the payoffs are measurable.

• Price forecasting: Neural networks on time-series data. Results are mixed; markets are noisy
• Credit scoring: Logistic regression estimate default probability. Backbone of consumer lending
• Fraud detection: Deep learning flags suspicious transactions in real time. Outperforms rule-based systems (McKinsey, 2023)
• Algorithmic trading: RL agents learn strategies by trial and error in simulated markets

These tasks share a common trait: the inputs are numbers, the outputs are numbers, and you can tell quickly whether the model is working

LLMs handle tasks that require reading and interpreting text. Much of the information that moves markets is in natural language:

• Sentiment analysis: Classify financial news and earnings calls as positive, negative, or neutral
• Document summarisation: Condense 10-K filings and earnings reports into short summaries
• Numerical reasoning: Models like FinQA read financial tables and do multi-step calculations
• Advisory chatbots: Answer portfolio questions, explain products, guide users through tax season

See the Chartered Financial Analyst Institute practical guide (2024) for more detail.

Several LLMs have been fine-tuned for finance:

• BloombergGPT: 363B tokens of financial data + 345B general. Beats general models on financial benchmarks
• FinGPT: Open-source, fine-tunes LLaMA via LoRA. Under $300 per run
• FinMA (PIXIU): 136K finance-specific instructions. Beats general LLMs on sentiment

Benchmarks (CFA Institute; Li et al., 2023):

• Finance-tuned LLMs beat general models on sentiment and classification
• GPT-4 still wins on numerical reasoning (more maths in pre-training)
• For stock prediction, no model is reliable. ARIMA/LSTM still more practical

In healthcare:

• Clinicians query drug interaction databases and clinical guidelines through natural language
• Systems retrieve relevant papers and generate evidence-backed answers
• Medical knowledge bases can be updated without retraining the model

In finance:

• Compliance officers ask questions about new regulations. The system retrieves the actual rule text and generates grounded answers
• Analysts query internal research repositories
• Knowledge bases stay current as regulations change

RAG is popular in both fields for one reason: it lets the model cite actual documents instead of making things up.

RAG reduces hallucination but does not eliminate risk. Both domains face domain-specific problems:

Retrieval quality:

• If the knowledge base is incomplete or outdated, the retrieved documents will be wrong. In medicine, outdated guidelines can be dangerous
• In finance, retrieving superseded regulations may lead to compliance failures

Bias in the knowledge base:

• Medical literature underrepresents certain populations. RAG retrieves what exists, so gaps in research translate into gaps in answers
• Financial knowledge bases skew toward English-language and Western-market sources

False confidence:

• RAG answers look authoritative because they cite documents. Users may trust them more than they should
• A clinician might skip verification if the answer comes with citations. A compliance officer might not read the full regulation

Obermeyer et al. (2019), published in Science, exposed a widely-used Optum algorithm

What it did: Identified patients needing extra care. Getting flagged = getting help

The problem: Used healthcare costs as a proxy for needs. But Black patients spend less even when equally sick (access barriers, insurance gaps, income inequality)

Result: At the same risk score, Black patients had 26.3% more chronic conditions. Unbiased referral would have raised the Black patient share from 17.7% to 46.5%

Race was not an input. But costs carried racial information because healthcare access itself was unequal

What happened next:

• Researchers contacted Optum before publication
• Switched from predicting costs to predicting health outcomes
• Racial bias reduced by ~84%

Lessons:

• The fix was simple. The hard part was recognising the problem
• Nobody had checked if the proxy worked equally for all groups
• Cost seemed reasonable, until someone disaggregated by race

Credit scoring decides who gets loans and at what rate. High-stakes ML

Proxy problems:

• Postcode correlates with race → indirect racial encoding
• Purchase patterns correlate with gender
• Career breaks penalise women disproportionately

Apple Card (2019): David Heinemeier Hansson’s credit limit was 20× his wife’s despite shared finances. NY regulators investigated Goldman Sachs. Found no intentional discrimination, but the inability to explain the outcomes damaged trust

LLMs in finance carry their own bias risks:

Sentiment analysis:

• Training data skews toward English-language media (Bloomberg, Reuters, FT)
• Less accurate for emerging markets, smaller firms, non-English sources
• Biased sentiment → biased trading signals → biased capital flows

Advisory chatbots:

• Trained on advice for wealthy clients → may not serve lower-income users
• Better in English than other languages

Summarisation:

• Stronger on large US corporations than international firms → unequal service

Healthcare AI

• Traditional ML: imaging, risk scoring, drug discovery
• LLMs: note summarisation, patient communication, literature synthesis
• RAG: evidence-backed clinical Q&A

Finance AI

• ML: trading, credit scoring, fraud detection
• LLMs: sentiment, summarisation, agentic workflows
• RAG: compliance and regulatory Q&A

Bias and RAG risks

• Proxy variables carry hidden assumptions
• Historical data encodes inequality
• RAG reduces hallucination but inherits knowledge base biases
• Disaggregated analysis is the only way to see the problem