• Last time we covered types of bias in AI systems
• Bias can enter at every stage: data collection, labelling, model design, deployment
• Historical bias: past discrimination baked into training data
• Representation bias: some groups underrepresented
• Measurement bias: proxies that correlate with protected attributes
• Fairness has multiple definitions that can conflict
• Today: how AI is used in healthcare and finance, and where bias shows up

Part 1: AI in healthcare

• Traditional ML vs LLMs in medicine
• Real deployments and RAG in clinical settings

Part 2: AI in finance

• ML for trading, credit scoring, and fraud
• LLMs, RAG, and agentic workflows

Part 3: When bias creeps in

• The Optum algorithm and credit scoring
• RAG-specific risks in both fields
• Patterns that repeat across domains

Part 4: What can we learn?

• Questions to ask about any AI system
• Group discussion

ML has been used in medicine for years, mostly for tasks with labelled data and a clear outcome:

• Medical imaging: CNNs detect tumours in X-rays, mammograms, retinal scans. Some match or beat radiologists
• Risk scoring: ML models predict readmission, sepsis, mortality. Hospitals use these to allocate resources
• Drug discovery: ML screens millions of molecules against protein targets, saving years in early development
• ECG analysis: Deep learning flags arrhythmias from wearables in real time

All well-defined tasks: structured input in, classification or number out.

LLMs handle tasks that involve reading, writing, and reasoning about text:

• Clinical note summarisation: Pull out key medications, diagnoses, and follow-up instructions in seconds
• Patient-facing chatbots: Explain diagnoses in plain language, answer follow-up questions, triage symptoms
• Literature synthesis: Summarise findings across dozens of papers on a condition
• Clinical trial matching: Match unstructured patient records against trial eligibility criteria (previously done by hand)
• My suggestion: DeepMind’s MedGemma, a free and open-source LLM for medical tasks. Optimised for medical text and image comprehension

All require understanding language, not crunching numbers.

Over 150 US health systems use AI-drafted messages via Epic’s MyChart (GPT-4):

• Patient sends a question about medication side effects
• System reads the patient’s record and drafts a response
• Clinician reviews and sends. Saves ~30 seconds per message

Google’s Med-PaLM 2 scored 86.5% on US Medical Licensing Exam questions. Physicians preferred its open-ended answers over those from other doctors.

But passing exams ≠ treating patients. The model cannot examine the patient, read body language, or pick up on context outside the chart.

LLMs are already used to talk to patients:

• Multilingual support: Translate discharge instructions into a patient’s language without a dedicated translator
• After-visit summaries: Plain-language version of what happened and what to do next
• Symptom triage: Chatbots recommend whether to see a doctor, go to A&E, or stay home

John Muir Health: clinicians using AI-assisted charting saved 34 min/day on documentation. Physician turnover dropped 44%.

Liability remains unresolved. If an AI chatbot gives bad advice and a patient is harmed, who is responsible?

Finance was one of the earliest adopters of ML. Tasks are well-defined and data is plentiful:

• Price forecasting: Neural networks on time-series data. Results are mixed; markets are noisy
• Credit scoring: Logistic regression estimate default probability. Backbone of consumer lending
• Fraud detection: Deep learning flags suspicious transactions in real time. Outperforms rule-based systems (McKinsey, 2023)
• Algorithmic trading: RL agents learn strategies by trial and error in simulated markets

All involve numerical data, clear targets, and plenty of training examples.

LLMs handle tasks that require reading and interpreting text. Much of the information that moves markets is in natural language:

• Sentiment analysis: Classify financial news and earnings calls as positive, negative, or neutral
• Document summarisation: Condense 10-K filings and earnings reports into short summaries
• Numerical reasoning: Models like FinQA read financial tables and do multi-step calculations
• Advisory chatbots: Answer portfolio questions, explain products, guide users through tax season

See the Chartered Financial Analyst Institute practical guide (2024) for more detail.

Several LLMs have been fine-tuned for finance:

• BloombergGPT: 363B tokens of financial data + 345B general. Beats general models on financial benchmarks
• FinGPT: Open-source, fine-tunes LLaMA via LoRA. Under $300 per run
• FinMA (PIXIU): 136K finance-specific instructions. Beats general LLMs on sentiment

Benchmarks (CFA Institute; Li et al., 2023):

• Finance-tuned LLMs beat general models on sentiment and classification
• GPT-4 still wins on numerical reasoning (more maths in pre-training)
• For stock prediction, no model is reliable. ARIMA/LSTM still more practical

In healthcare:

• Clinicians query drug interaction databases and clinical guidelines through natural language
• Systems retrieve relevant papers and generate evidence-backed answers
• Medical knowledge bases can be updated without retraining the model

In finance:

• Compliance officers ask questions about new regulations. The system retrieves the actual rule text and generates grounded answers
• Analysts query internal research repositories
• Knowledge bases stay current as regulations change

RAG is one of the most practical near-term LLM applications in both fields because it addresses the hallucination problem directly.

RAG reduces hallucination but does not eliminate risk. Both domains face domain-specific problems:

Retrieval quality:

• If the knowledge base is incomplete or outdated, the retrieved documents will be wrong. In medicine, outdated guidelines can be dangerous
• In finance, retrieving superseded regulations may lead to compliance failures

Bias in the knowledge base:

• Medical literature underrepresents certain populations. RAG retrieves what exists, so gaps in research translate into gaps in answers
• Financial knowledge bases skew toward English-language and Western-market sources

False confidence:

• RAG answers look authoritative because they cite documents. Users may trust them more than they should
• A clinician might skip verification if the answer comes with citations. A compliance officer might not read the full regulation

Every application we discussed is vulnerable to the same biases from last lecture:

• Historical bias: training data reflects past discrimination
• Representation bias: some groups underrepresented
• Measurement bias: proxies that work differently across groups
• Feedback loops: predictions reinforce themselves over time

The next slides cover real cases in healthcare and finance.

Obermeyer et al. (2019), published in Science, exposed a widely-used Optum algorithm.

What it did: Identified patients needing extra care. Getting flagged = getting help.

The problem: Used healthcare costs as a proxy for needs. But Black patients spend less even when equally sick (access barriers, insurance gaps, income inequality).

Result: At the same risk score, Black patients had 26.3% more chronic conditions. Unbiased referral would have raised the Black patient share from 17.7% to 46.5%.

Race was not an input. But costs carried racial information because healthcare access itself was unequal.

What happened next:

• Researchers contacted Optum before publication
• Switched from predicting costs to predicting health outcomes
• Racial bias reduced by ~84%

Lessons:

• The fix was simple. The hard part was recognising the problem
• Nobody had checked if the proxy worked equally for all groups
• Cost seemed reasonable, until someone disaggregated by race

Credit scoring decides who gets loans and at what rate. High-stakes ML.

Proxy problems:

• Postcode correlates with race → indirect racial encoding
• Purchase patterns correlate with gender
• Career breaks penalise women disproportionately

Apple Card (2019): David Heinemeier Hansson’s credit limit was 20× his wife’s despite shared finances. NY regulators investigated Goldman Sachs. Found no intentional discrimination, but the inability to explain the outcomes damaged trust.

LLMs in finance carry their own bias risks:

Sentiment analysis:

• Training data skews toward English-language media (Bloomberg, Reuters, FT)
• Less accurate for emerging markets, smaller firms, non-English sources
• Biased sentiment → biased trading signals → biased capital flows

Advisory chatbots:

• Trained on advice for wealthy clients → may not serve lower-income users
• Better in English than other languages

Summarisation:

• Stronger on large US corporations than international firms → unequal service

Healthcare AI

• Traditional ML: imaging, risk scoring, drug discovery
• LLMs: note summarisation, patient communication, literature synthesis
• RAG: evidence-backed clinical Q&A

Finance AI

• ML: trading, credit scoring, fraud detection
• LLMs: sentiment, summarisation, agentic workflows
• RAG: compliance and regulatory Q&A

Bias and RAG risks

• Proxy variables carry hidden assumptions
• Historical data encodes inequality
• RAG reduces hallucination but inherits knowledge base biases
• Disaggregated analysis is the only way to see the problem

Required readings:

• Obermeyer, Z. et al. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science.
• CFA Institute (2024). Practical guide for LLMs in the financial industry. RPC Research.

Recommended:

• Li, Y. et al. (2023). Large language models in finance: a survey. ICAIF-23.
• Singhal, K. et al. (2023). Towards expert-level medical question answering with large language models. Nature.
• O’Neil, C. (2016). Weapons of math destruction. Crown.
• George Zouq (2026). Awesome AI in Finance.
• AI4Finance Foundation (2026). AI4Finance.

Watch:

• Coded Bias (2020). Netflix documentary on AI discrimination (90 min)