QTM 385 - Experimental Methods

Lecture 15 - Natural and Quasi-Experiments

Danilo Freire
danilo.freire@emory.edu

Emory University

Hello, everyone! 😊

Brief recap 📚

Pre-analysis plans

Core Components

  • 7-section structure: Introduction, Hypotheses, Population, Intervention, Outcomes, Analysis, Implementation (EGAP template)
  • Key historical roots: FDA Modernization Act 1997, reproducibility crisis
  • Critical elements:
    • Clear primary/secondary hypotheses with theoretical basis
    • Power analysis for sample size determination
    • Ethical safeguards for vulnerable populations
  • PAPs increase transparency and reduce publication bias
  • PAPs can (and should!) be flexible
  • Some scholars say PAPs are overrated, but they are here to stay
  • Best practices:
    • Use OSF/AEA/EGAP registries to host your PAP
    • Include randomisation code snippets
    • Pre-specify attrition handling methods
    • You can use DeclareDesign to do all that!
    • Share your PAP with collaborators and reviewers!

Implementation challenges

  • PAPs are not perfect: 50% of researchers spend more than 2 weeks writing them
  • Common pitfalls:
    • Over-specification of analysis paths
    • Underpowered designs due to sample constraints
    • Non-compliance with treatment protocols
  • Solutions:
    • Pre-registered flexibility for exploratory analysis
    • Sensitivity analyses for robustness checks
    • Clear attrition management strategies

Today’s plan 📅

When experiments are not possible

  • The experimental ideal is not always feasible
    • Ethical constraints, logistical challenges, political barriers
  • Natural experiments: When “nature” (broadly speaking) does the randomisation
    • Examples: Earthquakes, hurricanes, pandemics, lottery wins
  • Quasi-experiments: Broader term than natural experiments, sometimes we can control the treatment but not the assignment
    • Examples: School vouchers, minimum wage laws, health interventions, politicy changes
    • Methods to estimate causal effects: Difference-in-differences, regression discontinuity, instrumental variables
  • They are somewhere between observational and experimental studies, and have received increased attention in the last decades

The experimental ideal and its limits 🌟

Why we should think about observational studies with RCTs in mind

  • As you already know by now, RCTs have the highest internal validity among all research designs
  • Since the treatment is isolated from other factors, we can be more confident about the causal effect
  • But RCTs are not always feasible, and sometimes they are not ethical or practical
    • Example: Testing the effects of smoking on health outcomes, or of poverty on child development
  • In these cases, we need to think creatively about how to estimate causal effects
  • Yet, the logic of RCTs should guide our thinking in observational studies
    • Random assignment, control groups, pre-treatment equivalence, selection bias
  • Natural and quasi-experiments are attempts to approximate the conditions of an RCT when true randomisation is not possible
  • They are “second-best” in terms of internal validity compared to a well-executed RCT
  • But often the best available or only feasible options
  • Both RCTs and natural/quasi-experiments are driven by the same goal: to establish causal relationships
  • The core logic of experiments, that is, of counterfactuals and potential outcomes, is the same
  • Natural and quasi-experiments often have higher external validity than RCTs, as they are closer to real-world conditions
  • But we only use observational data

Natural experiments 🌍

Definition

  • Natural experiments are situations where the assignment of treatment is determined by nature or other factors outside the control of the researcher
  • They are quasi-experimental designs that mimic the random assignment of treatments in experimental studies
  • In many ways, they are the “gold standard” (so to speak!) for causal inference in observational studies
  • There are mainly two kinds of natural experiments:
    • True natural experiments: When the treatment is indeed randomly assigned
    • As-if natural experiments: When the treatment is assigned by a procedure that we claim is as good as random for practical purposes
    • Examples of true natural experiments: Natural disasters, weather changes, lottery wins
    • Examples of as-if natural experiments: Eligibility rules, policy changes, roll-out of new technologies

True natural experiments

  • True natural experiments are rare but great opportunities for causal inference
  • They can be analysed in the same way as RCTs, with treatment and control groups and pre-treatment equivalence
  • The same concerns about internal validity apply to true natural experiments
    • Attrition
    • Non-compliance
    • Contamination
  • However, the researcher needs to make stronger assumptions about the assignment mechanism in true natural experiments
  • Yet often it is indeed the case that the assignment is random

Example 01: Charter schools

Angrist et al. (2013)

Explaining charter school effectiveness

Source: https://www.aeaweb.org/articles?id=10.1257/app.5.4.1

Research question and challenge

  • Research question: What is the causal effect of attending charter schools in Massachusetts on student test scores?
  • The challenge: Students who choose to attend charter schools are likely different from those who don’t
    • Maybe more motivated families, different prior academic preparation, etc
    • This is selection bias!
  • Need a way to isolate the causal effect of charter schools, not just correlation
  • The clever idea: Many charter schools are oversubscribed and use lotteries to decide who gets in
  • Instrumental Variable (IV) Approach: Use lottery win as an instrument to estimate the causal effect of charter attendance.
    • Lottery win \(\rightarrow\) Charter Attendance \(\rightarrow\) Test Scores
    • Lottery win is related to charter attendance, but ideally only affects test scores through charter attendance.

Estimation strategy

  • Equation (1): Outcome equation (What we want to explain)

\(Y_{igt} = \alpha_{2t} + \beta_{2g} + \sum_{j} \delta_{j} d_{ij} + X_{i}' \theta + \tau S_{igt} + \epsilon_{igt}\)

  • \(Y_{igt}\): Test score for student i, grade g, year t
  • \(S_{igt}\): Years spent in charter school (our treatment)
  • \(\tau\): Causal effect of charter attendance (what we want to find!)
  • Lots of control variables (\(X_i\), year and grade fixed effects, etc) to account for other factors
  • Equation (2): First Stage (How lottery affects attendance)

\(S_{igt} = \alpha_{1t} + \beta_{1g} + \sum_{j} \kappa_{ij} d_{ij} + X_{i}' \mu + \pi Z_{i} + \eta_{igt}\)

  • \(Z_i\): Dummy variable = 1 if student i won the charter lottery (our instrument)

  • \(\pi\): Effect of lottery win on charter attendance (should be positive and strong)

  • In Plain English:

  1. First Stage: Check if winning the lottery actually increases charter school attendance (it should!)
  2. Second Stage: Use the lottery win (which is random) to predict charter school attendance, and then see how this predicted attendance affects test scores

Results

Results

“As-If” randomness: The core assumption of natural experiments

  • The validity of most natural/quasi experiments hinges on the assumption of “as-if” randomness
  • We need to argue for the plausibility that the natural assignment mechanism is exogenous
    • Independent of other factors that influence the outcome
  • Consider:
    • Is the “natural” assignment process truly independent of confounding variables?
    • Could individuals or groups have influenced the “assignment” to treatment or control?
    • Is there evidence of pre-existing differences between the groups “naturally” assigned to different conditions?
  • Requires careful justification and scrutiny of the assignment process
  • Not always perfect randomness, but sufficient approximation for causal inference in many cases

Example 02: Ferwerda and Miller (2014)

Political Devolution and Resistance to Foreign Rule: A Natural Experiment

Source: Ferwerda, J., & Miller, M. A. (2014). Political devolution and resistance to foreign rule: A natural experiment. American Political Science Review, 108(3), 642–660. doi:10.1017/S0003055414000240

Theory and “as-if” randomness

Results

Criticism of the natural experiment approach

Kocher and Monteiro (2016)

Source: https://doi.org/10.1017/S1537592716002863

Kocher and Monteiro (2016)

Quasi-experiments 📊

Quasi-experiments: A broader toolkit for approximation

  • Quasi-experiments encompass a wider range of designs that lack full random assignment by the researcher
  • Often involve some researcher manipulation of the research design, but not direct treatment randomisation
  • Includes natural experiments as a subset, but also encompasses designs where researchers actively construct comparison groups
  • Key feature: strategic use of design and statistical techniques to address confounding and selection bias
  • Examples of Quasi-Experimental Designs:
    • Regression Discontinuity Design (RDD)
    • Difference-in-Differences (DID)
    • Instrumental Variables (IV)
    • Matching Methods

Regression discontinuity design (RDD)

  • RDD exploits sharp discontinuities in treatment assignment based on a threshold or cutoff score
  • Units are assigned to treatment or control based on whether they fall above or below a specific cutoff point on a continuous assignment variable (running variable)
  • Compares units just above the threshold to units just below
    • For the linear case, \(Y = \beta_{0} + \beta_{1}X + \beta_{2}(X > X_{0}) + \epsilon\)
  • Logic: Units very close to the cutoff are likely to be very similar in all other respects, except for treatment status
  • Cutoff creates a local “as-if” randomisation around the threshold
  • RDD estimates the treatment effect at the discontinuity point
  • Key Assumption: Continuity
    • Potential outcomes are continuous functions of the assignment variable at the cutoff
    • No other discontinuous changes at the cutoff point besides the treatment

RDD: Visualising the discontinuity

  • X-axis: Assignment/running variable
  • Y-axis: Outcome variable
  • Vertical dashed line: Cutoff threshold
  • Treatment assigned for D >= Cutoff
  • Continuity assumption: Smooth relationship between X and Y, except for the treatment effect at the cutoff
  • Fuzzy RDD: When treatment assignment is not perfectly determined by the cutoff (some non-compliance)
  • Manipulation of assignment variable can invalidate RDD
  • Local Average Treatment Effect (LATE): RDD estimates effect for those near the cutoff
  • Extrapolation beyond the cutoff is generally not recommended

Example 03: Mignozzetti et al. (2024)

Source: Mignozzetti, U., & al. (2024). Legislature size and welfare: Evidence from Brazil, American Journal of Political Science

Mignozzetti et al. (2024)

Mignozzetti et al. (2024)

Difference-in-Differences (DID): Leveraging policy changes

  • Core Idea: Compare changes in outcomes over time between a treatment group and a control group
  • Exploits naturally occurring events or policy changes that affect one group but not another
  • Relies on comparing the difference in changes (hence “difference-in-differences”)
    • Change in outcome for treatment group - Change in outcome for control group
  • Aims to isolate the treatment effect from:
    • Pre-existing differences between groups (captured by baseline differences)
    • Common trends over time (assumed to affect both groups similarly)
  • Key Assumption: Parallel Trends
    • In the absence of treatment, the treatment and control groups would have followed similar trends in outcomes

Example 04: Card and Krueger (1994)

Source: Card, D., & Krueger, A. B. (1994). Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania. The American Economic Review, 84(4), 772-793.

Card and Krueger (1994)

Threats to validity and mitigation strategies

Major threats to validity in quasi-experiments

  • Confounding (selection bias):
    • Pre-existing differences between treatment and control groups that are related to both treatment and outcome
    • Can bias estimates in any quasi-experimental design
  • Violation of key assumptions:
    • Parallel trends assumption in DID may not hold
    • Continuity assumption in RDD may be violated
    • Instrument validity assumptions (relevance and exclusion) in IV may be questionable
    • Selection on observables assumption in Matching may be false
  • Measurement error: Can attenuate or bias estimates in any design
  • Spillover effects/interference: Treatment effects in one group can affect outcomes in the control group, violating stable unit treatment value assumption (SUTVA)
  • Attrition and missing data: Differential attrition between treatment and control groups can introduce bias

Strategies for strengthening validity

  • Careful design choice: Select the most appropriate quasi-experimental design given the research question and context
  • Robustness checks and sensitivity analyses:
    • Test the sensitivity of findings to violations of key assumptions
    • Conduct placebo tests, falsification tests, and alternative specifications
  • Control variables: Include relevant covariates to control for observable confounding
  • Assumption checks and justification:
    • Clearly state and justify the assumptions of the chosen design
    • Provide evidence to support the plausibility of assumptions (e.g., pre-treatment trends for DID, density tests for RDD, instrument relevance tests for IV)
  • Triangulation: Use multiple methods and data sources to corroborate findings
  • Transparency: Be transparent about design limitations and assumptions in reporting results

Ethics beyond randomisation

  • While quasi-experiments often arise from ethical constraints on randomisation, ethical considerations remain important
  • Be transparent about the limitations of the design and assumptions being made
  • Quasi-experimental findings may be more easily misinterpreted than RCT results - emphasize responsible communication
  • Consequences: Consider potential unintended consequences of studying natural events or policy changes, especially if findings inform future interventions
  • Maintain ethical standards in data collection, storage, and analysis, especially when using sensitive data
  • Even in observational settings, consider ethical guidelines for data collection and participant interaction
  • Ethical review boards may still be involved in quasi-experimental research, especially if it involves human subjects data

Conclusion 🌟

Key takeaways

  • Natural and quasi-experiments are valuable tools for causal inference when RCTs are not feasible
  • Natural experiments use natural assignment mechanisms to mimic randomisation
  • Quasi-experiments encompass a broader range of designs that lack full random assignment
  • RDD, DID, IV, and Matching are common quasi-experimental designs
  • Validity threats in quasi-experiments include confounding, assumption violations, measurement error, and attrition
  • Strategies for strengthening validity include careful design choice, robustness checks, control variables, assumption checks, triangulation, and transparency
  • Ethical considerations remain important in quasi-experimental research

Thanks very much! 😊

See you next time! 👋