• To this point, we’ve focused mainly on estimating the Average Treatment Effect (ATE)
• Assumption: The treatment effect might be similar across individuals
• Reality: Interventions rarely affect everyone identically. Effects often vary across individuals, groups, or contexts
• Practical Value: Policymakers want to know who benefits most (or least) from an intervention and under what conditions
  • Targeting resources effectively
  • Tailoring programmes
• Scientific Value: Understanding why effects vary helps uncover causal mechanisms
  • What makes an intervention work (or not)?

• School Choice: Policies allowing school choice might benefit children whose parents prioritise academic quality more than those whose parents prioritise proximity
• Job Training: Programmes might increase earnings for voluntary participants but have no effect on those required to attend for public assistance eligibility
• Tax Compliance: Efforts to increase compliance might work differently depending on whether taxpayers can easily conceal income
• Advertising: A campaign targeting teenagers might alienate older consumers

• Individual treatment effect: \(\tau_i = Y_i(1) - Y_i(0)\)
• Treatment effect heterogeneity refers to the variance of individual treatment effects across subjects: \(Var(\tau)\)
• We want to estimate \(Var(\tau)\) and test if \(Var(\tau) > 0\) (indicating heterogeneity)
• Recall the relationship:

\[Var(\tau) = Var(Y_i(1) - Y_i(0))\] \[Var(\tau) = Var(Y_i(1)) + Var(Y_i(0)) - 2Cov(Y_i(1), Y_i(0))\]

• This equation highlights the challenge! Let’s see why… 🤓

• \(Var(Y(1))\) + \(Var(Y(0))\): Total variance in outcomes if we could see both worlds
• \(Cov(Y_i(1), Y_i(0))\): The covariance term. This is the key!
  • It asks: Do people who would have high outcomes without the treatment also have high outcomes with it?
• Why subtract the covariance? It accounts for stable differences between subjects
• Scenario: high positive covariance (common case)
  • People who are “high-achievers” (\(Y_i(0)\) is high) may also be “high-achievers” after treatment (\(Y_i(1)\) is high)
  • This stable difference makes \(Var(Y(1))\) and \(Var(Y(0))\) large, but there’s no heterogeneity in the effect
  • Subtracting \(2Cov(\cdot)\) corrects for this stable variance, isolating the true variance of the effect, \(Var(\tau)\)
• Example: If the effect is constant (\(\tau_i = \tau\) for all \(i\)), then \(Var(\tau) = 0\). The high covariance between \(Y_i(1)\) and \(Y_i(0)\) will cancel out the other two terms

• Experiments give us samples of \(Y_i(1)\) (from the treatment group) and \(Y_i(0)\) (from the control group)
• From these, we can estimate the marginal distributions of \(Y(1)\) and \(Y(0)\)
• We can estimate, for instance, \(E[Y(1)]\), \(E[Y(0)]\), \(Var(Y(1))\), and \(Var(Y(0))\)
• BUT: We never observe both \(Y_i(1)\) and \(Y_i(0)\) for the same individual \(i\)
• Therefore, we cannot directly estimate the joint distribution of potential outcomes
• So we cannot estimate \(Cov(Y_i(1), Y_i(0))\)
• Without the covariance, we cannot directly calculate \(Var(\tau)\)!

• Suppose \(N=6\) subjects. We observe outcomes:
  • Control Group (\(Y(0)\)): \({1, 2, 3}\) -> \(\hat{Var}(Y(0))=1\)
  • Treatment Group (\(Y(1)\)): \({4, 5, 6}\) -> \(\hat{Var}(Y(1))=1\)
• Estimated ATE = \(E[Y(1)] - E[Y(0)] = 5 - 2 = 3\)
• What is \(Var(\tau)\)? It depends on how potential outcomes are paired (which we don’t know):
• Scenario 1 (perfect positive correlation): Pairs are \({(1,4), (2,5), (3,6)}\)
  • Individual effects \(\tau_i\) are \({3, 3, 3}\)
  • \(Var(\tau) = 0\). Homogeneous effect
• Scenario 2 (perfect negative correlation): Pairs are \({(1,6), (2,5), (3,4)}\)
  • Individual effects \(\tau_i\) are \({5, 3, 1}\)
  • \(Var(\tau) = Var(\{5,3,1\}) = 4\). Heterogeneous effect
• Scenario 3 (Mixed): Pairs are \({(1,6), (2,4), (3,5)}\)
  • Individual effects \(\tau_i\) are \({5, 2, 2}\)
  • \(Var(\tau) = Var(\{5,2,2\}) = 3\). Heterogeneous effect
• Experimental data alone do not distinguish these scenarios!

• A simpler approach: Test the null hypothesis \(H_0: Var(\tau) = 0\) (homogeneous effects)
• If \(\tau_i = \tau\) (constant) for all \(i\), then \(Var(\tau)=0\)
• Also, if \(\tau\) is constant, \(Cov(Y(0), \tau) = Cov(Y(0), \text{constant}) = 0\)
• Recall: \(Var(Y(1)) = Var(Y(0)) + Var(\tau) + 2Cov(Y(0), \tau)\)
• Under \(H_0: Var(\tau)=0\), this simplifies to \(Var(Y(1)) = Var(Y(0))\)
• Test Idea: If we observe significantly different variances in the treatment and control groups, we can reject the null hypothesis of homogeneous effects
• Caution: Equality of variances does not strictly prove homogeneity, but large differences suggest heterogeneity

• Bounds on \(Var(\tau)\) are often very wide, making them uninformative
• Tests comparing \(Var(Y(1))\) and \(Var(Y(0))\) often lack statistical power, especially with smaller samples
• These methods don’t tell us why effects might vary or who is affected differently
• They serve as a preliminary step. Significant results encourage more structured investigation

➡️ Need approaches that examine variation across specific subgroups or conditions

• We can partition subjects into subgroups based on pre-treatment covariates (\(X\)) and estimate the ATE within each subgroup
• CATE: Conditional Average Treatment Effect
  • \(CATE(X=x) = E[Y_i(1) - Y_i(0) | X_i = x]\)
• Example: Estimate the ATE for men vs. women, old vs. young, high income vs. low income
• Interaction Effect: The difference between CATEs across subgroups
  • \(Interaction = CATE(X=x_1) - CATE(X=x_2)\)
• This approach uses observable characteristics to explore potential sources of heterogeneity
• Often guided by theory about why effects might differ

• Researchers explored if teacher incentive effects varied by school characteristics (Muralidharan & Sundararaman 2011)
• Covariate: Average parent literacy level (pre-treatment). Partitioned schools into below-median and above-median literacy
• Estimated CATEs:
  • Low Literacy Schools: CATE = 11.14 - 7.83 = 3.31
  • High Literacy Schools: CATE = 12.26 - 8.57 = 3.69
• Interaction Effect: 3.69 - 3.31 = 0.38
• Hypothesis Test: Is this difference statistically significant?
  • Using randomisation inference (or regression F-test), p = 0.88
  • No significant evidence that the treatment effect differs based on parental literacy levels in the school

• Regression provides a flexible way to estimate and test CATEs and interactions
• Let \(I_i\) be the treatment indicator (1=Treat, 0=Control)
• Let \(P_i\) be the covariate indicator (e.g., 1=High Literacy, 0=Low Literacy)
• Null Model (Homogeneous Effect): \(Y_i = \alpha + \beta I_i + \gamma P_i + u_i\)
  • Assumes a single ATE (\(\beta\))
• Alternative Model (Interaction): \(Y_i = \alpha + \beta I_i + \gamma P_i + \delta (I_i \times P_i) + u_i\)
  • CATE for Low Literacy (\(P_i=0\)): \(\beta\)
  • CATE for High Literacy (\(P_i=1\)): \(\beta + \delta\)
  • Interaction Effect: \(\delta\)
• Use an F-test to see if the model with the interaction term (\(\delta\)) fits significantly better than the null model (i.e., test \(H_0: \delta = 0\))
• In the teacher incentive example, the F-test yields \(p = 0.88\), matching the randomisation inference

• Datasets often contain many potential covariates (age, gender, income, education, location, etc.)
• If we test for interactions with many covariates, the chance of finding a significant interaction purely by chance increases
• Example: Test 20 uncorrelated covariates for interaction at \(\alpha=0.05\). Assume the true effect is homogeneous
  • \((1 - \alpha)^q\) gives the probability of no false positives across \(q\) tests, so \(1 - (1 - \alpha)^q\) is the probability of at least one false positive
  • Probability of finding at least one significant interaction: \(1 - (1 - 0.05)^{20} \approx 0.64\). (High risk of Type I error!)
• Solution 1: Bonferroni Correction:
  • If conducting \(q\) tests, adjust significance level to \(\alpha / q\)
  • E.g., for 20 tests at \(\alpha=0.05\), require \(p < 0.05 / 20 = 0.0025\)
  • Simple, but can be overly conservative (reduces power)
  • Other similar solutions exist (e.g., False Discovery Rate control, Holm-Bonferroni, etc.)
• Solution 2: Pre-specification:
  • Specify the few theoretically motivated interactions you will test before looking at the data (e.g., in a Pre-Analysis Plan)

• Covariates used for subgroup analysis are observed characteristics, not randomly assigned treatments
• Finding that CATEs differ between, say, high-education and low-education groups means the treatment effect correlates with education
• It does not necessarily mean that changing someone’s education level would change how they respond to the treatment
• Education might be a marker for other underlying factors (e.g., income, access to information, underlying health) that are the “true” drivers of the heterogeneity
• Subgroup analysis is essentially observational/descriptive regarding the source of heterogeneity
• It’s useful for prediction (who is likely to respond more?) and generating hypotheses, but not for establishing the causal impact of the covariate itself on the treatment effect

• Gerber, Green, Larimer (2008) sent mailers showing recipients’ own (and neighbours’) past voting records to encourage turnout
• Finding: Mailer effect (ATE) was larger among people who had voted in a previous election (2004) compared to those who hadn’t
• Superficial interpretation: Seeing your past voting record is more motivating than seeing your past non-voting record
• However, past voting behaviour (the covariate) is not random. People who voted in 2004 are different from those who didn’t in many ways (more politically engaged, different demographics, etc.)
• Follow-up Experiment (Gerber, Green, Larimer 2010): Randomly varied whether the mailer showed a record of voting or a record of abstention
• Result: Showing a record of abstention was significantly more motivating!
• The subgroup analysis conflated the type of person with the content of the message. Direct experimental manipulation was needed to isolate the causal effect of the message content

• To make causal claims about why treatment effects vary, we need to experimentally manipulate the hypothesised moderating factors
• This is done using factorial experimental designs
• An experimental design that includes two or more “factors” (independent variables), where subjects are randomly assigned to a combination of the levels of these factors
• Example: Factor A (Treatment vs. Control), Factor B (Context 1 vs. Context 2)
• Subjects randomly assigned to one of four groups: (Treat, Cxt 1), (Treat, Cxt 2), (Control, Cxt 1), (Control, Cxt 2)
• Allows us to estimate:
  • Main effect of Factor A
  • Main effect of Factor B
  • Interaction effect: Does the effect of Factor A depend on the level of Factor B?
• This provides stronger causal evidence about moderators than treatment-by-covariate analysis

• Gerber & Green (2000) Voter Mobilisation: Crossed multiple communication methods (Factor 1: Canvassing Y/N, Factor 2: Phone Call Y/N, Factor 3: Direct Mail Y/N) to see if effects were additive or interactive
• Olken (2007) Corruption Monitoring: Crossed top-down audits (Factor 1: Audit Y/N) with bottom-up community monitoring (Factor 2: Invitations to meetings Y/N) in Indonesian road projects. Tested if audits were more effective when community was involved
• Rosen (2010) Discrimination: Crossed putative email sender ethnicity (Factor 1: Hispanic/Non-Hispanic name) with email grammar quality (Factor 2: Good/Bad grammar) sent to state legislators. Tested if ethnic discrimination depended on perceived social class (signalled by grammar)

• Do U.S. state legislators respond differently to constituent emails based on perceived ethnicity and writing quality?

• 2x2 Factorial Design:

  • Factor 1: Sender Name (Colin Smith vs. José Ramirez)
  • Factor 2: Email Grammar (Good vs. Bad)

• Outcome: Percentage of emails receiving a reply

• Results (Subset, N=100 per cell):

• Interpretation:
  • With good grammar, Colin gets more replies (-15% effect for José)
  • With bad grammar, José gets slightly more replies (+5% effect)
  • The effect of ethnicity depends on grammar quality (Interaction)
  • Poor grammar hurts Colin much more (-23%) than José (-3%)

• Advantages:
  • Allows causal estimation of interaction effects
  • Efficient way to study multiple factors simultaneously
  • Can reveal complex relationships (e.g., conditions where a treatment is effective/ineffective)
• Caveats:
  • Require larger sample sizes to detect interactions (interactions are often smaller than main effects)
  • Can become complex logistically with many factors/levels
  • Non-compliance can be problematic: estimating effects for those receiving multiple treatments requires sufficient compliers in those specific cells, which can be rare

• Regression is a powerful tool for modelling both treatment-by-covariate and treatment-by-treatment interactions. And you already know how to use it!
• Systematic Heterogeneity: Variation in \(\tau\) that can be predicted by observed covariates or experimental factors (modelled by interaction terms)
• Idiosyncratic Heterogeneity: Residual variation in \(\tau\) not explained by the model (part of the error term \(u_i\))
• We use interaction terms to capture systematic heterogeneity

• Let \(J_i = 1\) if sender is José, 0 if Colin
• Let \(G_i = 1\) if grammar is bad, 0 if good
• Model: \(Y_i = \alpha + \beta J_i + \gamma G_i + \delta (J_i \times G_i) + u_i\)
• Interpreting Coefficients:
  • \(\alpha\): Mean outcome for baseline (Colin, Good Grammar) = 52%
  • \(\beta\): Effect of José vs. Colin, when grammar is good = 37% - 52% = -15%
  • \(\gamma\): Effect of Bad vs. Good grammar, when sender is Colin = 29% - 52% = -23%
  • \(\delta\): Interaction effect. How the effect of José changes when grammar goes from good to bad.
    • Effect of José (Bad Grammar) = (\(\alpha + \gamma\)) + (\(\beta + \delta\))
    • Effect of José (Good Grammar) = \(\alpha + \beta\)
    • Difference = \(\gamma + \delta\)
    • Alternatively: \(\delta\) = (Effect of Bad Grammar for José) - (Effect of Bad Grammar for Colin) = (-3%) - (-23%) = +20%
• Estimates for Rosen data: \(\hat{Y}_i = 0.52 - 0.15 J_i - 0.23 G_i + 0.20 (J_i \times G_i)\)
• Testing the Interaction: F-test for \(H_0: \delta = 0\). In Rosen’s data, p = 0.037. Significant interaction

• Manually specifying and testing interactions becomes infeasible with many covariates/factors
• Machine Learning Approaches: Algorithms designed to automatically search for interactions
  • E.g., Generalised Random Forests (Athey & Imbens)
  • Methodically partition the data based on covariates to find subgroups with different treatment effects
  • Often use techniques like cross-validation to avoid overfitting and false discoveries
• Can be a great exploratory tool, especially in high-dimensional settings
• Still requires careful interpretation and follow-up validation
• Beyond the scope of this lecture, but good to be aware of! 🤓