QTM 385 - Experimental Methods

Lecture 19 - Texts for Discussion

Danilo Freire
danilo.freire@emory.edu

Emory University

Ready for some applied causal inference? 🤓

Brief recap 📚

Last time we discussed

  • Why average treatment effects (ATEs) aren’t the whole story
  • Defining and understanding treatment effect variability
  • Fundamental challenge: We cannot directly observe individual treatment effects or their variance
  • Methods for exploring HTE:
    • Bounding the variance of treatment effects
    • Testing for the presence of heterogeneity
  • Structured approaches:
    • Treatment-by-Covariate Interactions (Subgroup Analysis / CATEs)
    • Treatment-by-Treatment Interactions (Factorial Designs)
  • Modelling HTE using regression
  • Pitfalls and best practices:
    • Multiple comparisons
    • Causal interpretation challenges

Today’s plan 📅

Networks, Spillovers, and Identification

  • We will examine two influential papers today:

  • Paper 1: Munshi (2003)

    • Topic: Identifying network effects in the labour market.
    • Context: Mexican migrants in the U.S.
    • Challenge: Endogeneity and selection bias.
    • Method: Instrumental Variables (IV) using rainfall.
  • Paper 2: Miguel & Kramer (2004)
    • Topic: Identifying treatment impacts with externalities.
    • Context: Deworming programme in Kenyan schools.
    • Challenge: Standard methods underestimate effects due to spillovers (SUTVA violation).
    • Method: Randomised phase-in design.
  • Our focus will be on clever identification strategies when standard RCT assumptions are violated or RCTs are not feasible.

But first, the experiment of the week 🔬

The Experiment of the Week

Blocking mobile internet on smartphones improves well-being

  • Castello et al (2025) conducted a one-month field experiment to evaluate how mobile internet access affects well-being.
  • The results were highly significant:
    • The RCT improved attention, subjective well-being, and mental health.
    • 91% of participants reported a positive experience (!)
  • Mediation analysis suggests that the effects are driven by how participants use their time.
  • People spend more time socialising, exercising, reading, and being outdoors.
  • https://doi.org/10.1093/pnasnexus/pgaf017

Part 1: Munshi (2003) - Networks in the Modern Economy 🌐

Introduction: The Power of Networks

  • Paper: Munshi, K. (2003). Networks in the Modern Economy: Mexican Migrants in the U. S. Labor Market. The Quarterly Journal of Economics, 118(2), 549-599.
  • Core Question: Do social networks improve labour market outcomes for Mexican migrants in the U.S.?
  • Motivation:
    • Non-market institutions (like networks) are crucial, especially where markets are imperfect (e.g., information gaps in labour markets).
    • Migrant communities often rely heavily on social ties for finding jobs, housing, etc. [Rees 1966, Montgomery 1991].
    • Particularly relevant for undocumented migrants who face barriers to formal channels.
  • Specific Focus: Does a larger network (from the same origin community) lead to better employment prospects and higher-paying jobs?

Theoretical Motivation

  • Why networks might matter:
    • Information Asymmetry: Employers don’t know worker quality; workers don’t know about all job openings. Networks provide referrals.
    • Screening/Referrals: Incumbent workers might refer others from their network, potentially signalling quality [Montgomery 1991].
    • Support: Networks can provide financial help, housing, advice, reducing job search costs.
  • Who contributes? Likely established migrants (longer duration, more connections, more employed).
  • Who benefits? Likely new arrivals or those who would otherwise struggle to find employment/good jobs.
  • Likely Outcomes: Networks could lead to higher employment rates and channel members into better (e.g., non-agricultural) jobs.

Data: The Mexican Migration Project (MMP)

  • Conducted jointly by researchers in Mexico and the US since 1982.
  • Surveys households in multiple Mexican communities with high migration rates.
  • Collects retrospective life histories: migration patterns, work history (location, job type, employment status) year-by-year.
  • An important strength is defining the social unit based on the origin community in Mexico – a well-established social unit (paisanaje, origin-community). This is arguably better than using arbitrary administrative boundaries.
  • Munshi uses data from 24 communities across 7 Mexican states, focusing on the 15 years prior to the survey year for each community.

Measuring the Network

  • Network Definition: For an individual \(i\) from community \(c\) in year \(t\), the network size (\(X_{t-1}\)) is measured as the proportion of sampled individuals from community \(c\) who were located in the U.S. in the previous year (\(t-1\)).
  • A lagged measure is used (\(t-1\) network size to predict \(t\) outcomes) to reduce immediate simultaneity concerns.
  • Network size changes over time within a community due to new migration and return migration. Munshi primarily uses this within-community time variation.
  • The analysis later refines this by distinguishing between new migrants (in US < 3 years) and established migrants (in US ≥ 3 years), based on theoretical predictions.

Identification Challenge 1: Selection Bias

  • Problem: Migrants are not a random sample of the origin community. Individuals who migrate might differ in unobserved ways (e.g., ability, motivation) that also affect their labour market success.
  • \(Y_{it} = \beta X_{t-1} + \alpha_i + \delta_t + \epsilon_{it}\)
    • \(Y_{it}\): Labour outcome (e.g., employment)
    • \(X_{t-1}\): Network size
    • \(\alpha_i\): Individual unobserved heterogeneity (ability)
    • \(\delta_t\): Year fixed effects (common shocks)
  • If higher ability (\(\alpha_i\)) individuals are more likely to migrate and get better jobs, simply regressing \(Y_{it}\) on \(X_{t-1}\) will yield biased estimates of \(\beta\). The network effect might be confounded with individual ability.

Solution 1: Individual Fixed Effects

  • Method: Utilise panel data (repeated observations for the same individual over time) and include individual fixed effects (\(\alpha_i\)).
  • Logic: Fixed effects control for any time-invariant unobserved characteristics of the individual (like innate ability, stable motivation).
  • Identification: The effect of the network (\(\beta\)) is now identified from changes in network size over time for the same individual, comparing their outcomes when their network was larger versus smaller.
  • Assumption: Unobserved individual ability (\(\alpha_i\)) is constant over the sample period (perhaps reasonable for low-skill jobs).
  • Limitation: Fixed effects do not solve simultaneity bias.

Identification Challenge 2: Endogeneity / Simultaneity

  • Problem: Network size (\(X_{t-1}\)) itself might be correlated with unobserved factors affecting current outcomes (\(Y_{it}\)).
  • For instance, positive economic shocks in the U.S. (\(\epsilon_{it}\)) could simultaneously:
    1. Improve employment outcomes (\(Y_{it}\)) directly.
    2. Attract more migrants, increasing network size (\(X_{t-1}\) or \(X_t\)).
    3. Reduce return migration among established migrants.
  • This simultaneity would lead OLS (even with fixed effects) to potentially overestimate the positive effect of networks.
  • Alternatively, good U.S. conditions might allow migrants to reach savings targets faster and return home, reducing network size, biasing estimates downwards.

Solution 2: Instrumental Variables (IV)

  • Goal: Find an instrument (\(Z\)) that satisfies two conditions:
    1. Relevance: Correlated with the endogenous variable (network size, \(X_{t-1}\)).
    2. Exclusion Restriction: Uncorrelated with the error term (\(\epsilon_{it}\)), meaning it only affects the outcome (\(Y_{it}\)) through its effect on the network size (\(X_{t-1}\)).
  • Munshi’s Instrument: Lagged rainfall in the migrant’s origin community in Mexico.
  • Rationale:
    • Agriculture (often rain-fed) is a major occupation in origin communities.
    • Low rainfall (drought) likely worsens agricultural outcomes in Mexico, increasing migration to the U.S. (push factor), thereby influencing network size in subsequent years.

Assessing the Rainfall Instrument

  • Checking Relevance: Is Mexican rainfall correlated with network size in the US?
    • This is plausible: Droughts push people to migrate.
    • Munshi provides empirical evidence in the first-stage regressions (Table V), showing low rainfall increases migration, particularly affecting the stock of established migrants later on.
  • Checking the Exclusion Restriction: Does Mexican rainfall affect U.S. employment only via network size?
    • This seems plausible: Origin communities are far from U.S. destinations. Mexican weather shocks should be uncorrelated with U.S. labour market shocks. (Munshi reports a correlation coefficient of 0.01 between origin rainfall and US destination rainfall).
    • Are there potential threats? Could low rainfall affect migrant characteristics (e.g., desperation, reservation wage) directly influencing US employment? Munshi argues the observed timing (long lags) makes this less likely to be the primary channel for the employment effect.

Results: Reduced Form (Rainfall & Employment)

  • Table IV directly regresses employment (in US or Mexico) on lagged rainfall, controlling for individual FE and year FE.
  • For US Employment (Column 1):
    • Recent-past rainfall (t to t-2) shows no significant effect.
    • Distant-past rainfall (t-3 to t-6) shows a negative and significant effect. Less rain 3-6 years ago correlates with higher current US employment.
  • This suggests a mechanism: Low rainfall long ago \(\rightarrow\) induced migration then \(\rightarrow\) larger pool of established migrants now \(\rightarrow\) better network \(\rightarrow\) higher current employment for all migrants.
  • For Mexico Employment (Column 2):
    • Recent rainfall is strongly positively correlated with origin employment (more rain = better farming jobs), supporting the push factor idea.

Results: First Stage (Rainfall & Network Size)

  • Table V, Columns 6 & 7 regress network size measures (proportion of new / established migrants) on rainfall lags.
  • The results show:
    • Recent-past rainfall (t to t-2 average) is negatively correlated with the proportion of new migrants (Column 6). Low recent rain \(\rightarrow\) more new migrants pushed out.
    • Distant-past rainfall (t-3 to t-6 average) is negatively correlated with the proportion of established migrants (Column 7). Low distant rain \(\rightarrow\) larger stock of established migrants now.
  • This empirically validates the instrument’s relevance and the mechanism suggested by the reduced form.

Results: Non-Parametric Evidence

  • Figure I provides a graphical look at the relationships.
  • It plots employment (US) and network size (established migrants) against distant-past rainfall, after netting out fixed effects.
  • The figure visually suggests:
    • As distant-past rainfall increases (wetter conditions in Mexico years ago), current US employment tends to decrease.
    • As distant-past rainfall increases, the current size of the established network tends to decrease.
  • This supports a positive link between the established network size and employment outcomes.

Results: OLS and IV Estimates (Employment)

  • Table VI compares OLS and IV estimates of network effects on US employment probability.
  • The OLS estimates (Column 1) indicate:
    • New migrants have a small, insignificant effect.
    • Established migrants have a positive, significant effect (coefficient 0.670).
  • The IV estimates (Column 2 - Using Rainfall Instruments) show:
    • New migrants remain insignificant, though the point estimate is larger.
    • The effect for established migrants becomes much larger and remains significant (coefficient 1.554).
  • Further robustness checks (columns 3-11) support these patterns. Effects seem larger for women/older men/less educated, and when restricting to fresh arrivals (col 4).

Munshi (2003): Contributions & Implications

  • This study finds that:
    • Social networks significantly improve labour market outcomes for Mexican migrants in the US, increasing both employment probability and access to better (non-agricultural) jobs.
    • Established migrants are the crucial part of the network driving these positive effects.
  • Methodologically, it demonstrates how to combine IV (using a plausible natural instrument - rainfall) with fixed effects to identify network effects while addressing selection and endogeneity concerns.
  • It underscores the vital role non-market institutions play in modern economies, particularly for vulnerable populations, and shows that network size and composition matter.

Points for Discussion (Munshi)

  1. How convincing is the rainfall instrument’s exclusion restriction? Could distant rainfall affect current US employment through channels other than the measured network size (e.g., affecting savings, remittances, characteristics of the migrant pool)?
  2. The IV estimates for established networks are much larger than OLS. Besides endogenous return migration and measurement error, what else might explain this?
  3. What are the limitations of measuring the network as the proportion of the sampled community in the US? How might alternative measures change the story?
  4. Would network effects likely differ for high-skill migrants? Why or why not?
  5. What policy insights regarding immigration or labour market integration emerge from these findings?

Issues with rainfall as an instrument?

  • Mellon (2021) evaluates whether rainfall is a valid instrument for social studies.
  • He uses Cinelli and Hazlett’s (2020) sensitivity analysis to assess the robustness of about 200 published studies using rainfall as an instrument.
  • Mellon is not very convinced that rainfall is a good instrument instrument 😂
  • What do you think?

Part 2: Miguel & Kramer (2004) - Worms 🐛

Introduction: Health, Education, and Externalities

  • Paper: Miguel, E., & Kremer, M. (2004). Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities. Econometrica, 72(1), 159-217.
  • Context: Intestinal worms (helminths) are extremely common in developing countries, especially among school-aged children. Associated with malnutrition, listlessness.
  • Core Question: What is the impact of a school-based mass deworming programme on children’s health and education (school participation, test scores)?
  • Key Problem: Previous studies often randomised treatment at the individual level. This likely underestimates the true benefits due to treatment externalities (spillovers).

The Problem: Treatment Externalities

  • Mechanism: Worm infections spread through contact with contaminated soil/water. Treating infected children reduces the overall pool of infection in the environment.
  • Externality: Treating child A can benefit untreated child B if they live/play/attend school nearby, because B is less likely to get infected or re-infected.
  • This creates issues for individual-level randomisation:
    • The comparison group (untreated individuals within a treated community/school) benefits from reduced transmission from the treated group.
    • The estimated treatment effect (Treated Outcome - Control Outcome) is biased downwards because the control outcome is better than it would be without the program.
    • This represents a violation of the Stable Unit Treatment Value Assumption (SUTVA).

Research Setting and Intervention

  • Location: Busia District, Western Kenya. Poor, densely settled farming region. High prevalence of intestinal worms (hookworm, roundworm, whipworm, schistosomiasis).
  • Programme: Primary School Deworming Project (PSDP) by NGO (ICS) and Ministry of Health.
  • Intervention: Provision of free deworming drugs (albendazole, praziquantel) to pupils in primary schools. Phased in over time.
  • Sample: 75 rural primary schools, ~30,000 pupils.
  • Schools were randomly assigned to 3 groups (25 schools each) after stratification.
  • Group 1: Received treatment in 1998 and 1999.
  • Group 2: Received treatment in 1999 (Control in 1998).
  • Group 3: Control in 1998 and 1999 (Received treatment later).
  • Advantages of this design:
    1. Allows estimation of treatment effects by comparing outcomes across groups over time (e.g., Group 1 vs Group 2&3 in 1998). This provides a school-level Intention-to-Treat (ITT).
    2. Crucially, random assignment at the school level generates variation in the local density of treatment, enabling estimation of cross-school externalities.
    3. Arguably more ethical than a permanent control group, as all schools eventually receive treatment.

Identification Strategy

  • Overall Effect (ITT at School Level): Compare mean outcomes in treatment schools vs. comparison schools in a given year, controlling for baseline differences. \(Y_{ijt} = \alpha + \beta_1 T_{1it} + \beta_2 T_{2it} + X_{ijt}\delta + u_i + \epsilon_{ijt}\) (Where T1, T2 indicate year 1 / year 2 treatment assignment)
  • Cross-School Externalities: Exploit the exogenous variation in treatment density induced by school-level randomisation. \(Y_{ijt} = \alpha + ... + \sum_d (\gamma_d \cdot N^T_{dit}) + \sum_d (\phi_d \cdot N_{dit}) + u_i + \epsilon_{ijt}\)
    • \(N^T_{dit}\): Number of treated pupils within distance d of school i.
    • \(N_{dit}\): Total number of pupils within distance d.
    • \(\gamma_d\) captures the externality effect from nearby treated pupils, after controlling for overall density \(\phi_d\).
  • Within-School Externalities: Harder to identify experimentally due to lack of within-school randomisation. The paper uses non-experimental comparisons (e.g., untreated pupils in T schools vs C schools - Table VI), acknowledging potential selection issues.

Baseline Characteristics

  • Table I compares schools/pupils across the 3 groups before the intervention (early 1998) to check the randomisation.
  • Groups should be similar on average.
  • The table shows groups are broadly similar on most characteristics (enrolment, facilities, distance, assets, malaria, local density).
  • However, Group 1 pupils appear slightly worse off on some measures (e.g., self-reported blood in stool, cleanliness, 1996 exam scores) despite randomisation. The authors control for these baseline covariates in regressions to improve precision and account for chance imbalances.

Treatment Compliance

  • Table III details the proportion of eligible pupils who actually received the drugs.
  • In 1998 (Group 1): Approx. 78% of eligible pupils (girls < 13, all boys) received treatment. Compliance was lower for older girls (19% received, despite protocol). The main reason for non-compliance was absence on treatment day.
  • In 1999 (Groups 1 & 2): Compliance was lower (around 55-59% overall for eligible), partly due to requiring signed parental consent in Year 2.
  • Comparison Schools: Very low rates (<5%) of pupils receiving deworming drugs independently.

Health Impacts (Infections)

  • Table V compares infection rates in Group 1 (treated ’98) vs Group 2 (control ’98) based on parasitological exams in early 1999.
  • Group 1 pupils had significantly lower rates of moderate-to-heavy infection (any worm) compared to Group 2 (27% vs 52%, a 25 percentage point difference). Significant reductions were seen for hookworm, roundworm, and schistosomiasis.
  • Table VII (Regression) confirms these health benefits and estimates externalities:
    • Within-School Externality (Col 2): Untreated pupils in Group 1 schools had ~12 pp lower infection rates than comparison pupils (\(\beta_1\) estimate).
    • Direct Effect + Within-School (Col 2): Treated pupils in Group 1 had ~26 pp lower infection rates (\(\beta_1 + b_2\) estimate).
    • Cross-School Externality (Col 1): Nearby treatment pupils had significant negative effects on infection rates (esp. within 3km). Each 1000 treated pupils within 3km reduced infection risk by ~26 pp (\(\gamma_{03}\) estimate).

Health Impacts

School Participation (Attendance)

  • Table IX (Regression) provides more detail:
    • Overall ITT (Col 1): Deworming increased participation by approx. 5.1 pp over the two years.
    • Cross-School Externality (Col 3): Positive externality found - each 1000 treated pupils within 3km increased participation in nearby schools by approx. 4.4 pp.
    • Within-School Externality (Col 5): Untreated pupils in treatment schools experienced ~5.6 pp higher participation (\(\beta_1\) estimate for Year 1).
  • Absenteeism was reduced by about 25%.

Test Scores

  • Table X examines the impact on academic test scores (English, Maths, Science-Ag) administered by the project, with scores normalised.
  • The main result is no significant difference in test scores between treatment and comparison schools (Column 2).
  • This result holds when restricting the sample to pupils present at baseline (Column 3).
  • Why no test score effect?
    • Attendance gains (~7.5 pp) might be insufficient to significantly boost scores.
    • Deworming might not improve learning efficiency per day in school (especially as severe anaemia was rare).
    • Potential classroom congestion effects from increased attendance?
    • Test scores might be noisy measures of learning.

Cost-Effectiveness

  • Deworming drugs are extremely inexpensive (< $0.50 per child per year via PCD estimates).
  • Health Perspective: Cost per DALY averted is estimated around $5 (including externalities), making it highly cost-effective by WHO standards, especially driven by schistosomiasis impact. Geohelminth treatment alone appears less cost-effective based only on health metrics.
  • Education Perspective: Cost per additional year of school participation gained is approx. $3.50 (including externalities). This is substantially cheaper than alternative participation-boosting interventions evaluated in the area (e.g., free uniforms ~$99).
  • Human Capital View: Rough calculations suggest future wage benefits (>$30) greatly exceed program costs ($0.49), implying a potentially high rate of return on investment.

The “Worm Wars” Controversy

  • This paper became highly influential, often cited by international organisations promoting mass deworming policies.
  • However, its findings, particularly regarding school participation, sparked considerable debate and scrutiny.
  • Re-analyses & Critiques:
  • Defenses & Counter-Arguments:
  • Subsequent Evidence:
    • Ozier (2018, JHR) reported long-term benefits (earnings, cognition) for younger siblings of treated children and those exposed to externalities in early life, suggesting impacts might manifest later or differently.
  • This debate highlights the importance of replication, data transparency, sensitivity analysis, and cautious interpretation of field experiment results, especially complex ones with potential spillover effects.

Miguel & Kramer (2004): Contributions & Implications

  • This study demonstrated that:
    • School-based deworming significantly reduced worm infections and increased school participation (~7.5 pp), though had no discernible effect on test scores.
    • There were large positive externalities (spillovers) affecting the health and participation of untreated children, both within the same school and in nearby schools. These externalities constitute a major portion of the total benefits.
    • Deworming is a highly cost-effective intervention, particularly for boosting school participation.
  • Methodologically, it powerfully illustrates the importance of accounting for externalities in program evaluation. It shows how school-level (or cluster) randomisation can facilitate the identification of overall effects and the estimation of spillovers.
  • A key policy insight is that ignoring externalities likely leads to under-investment in public health programmes like deworming. Full subsidies (or even payments, considering the large externalities) might be optimal.

Points for Discussion (Miguel & Kramer)

  1. What mechanisms could explain why deworming improved school participation but not test scores in this context?
  2. Explain how the randomised phase-in design allows the estimation of cross-school externalities. What are the key assumptions underlying the externality estimates derived from local treatment density?
  3. The study relies significantly on NGO-collected data. What advantages and disadvantages might this have compared to using official administrative records?
  4. In light of the “Worm Wars,” how should policymakers weigh the evidence on deworming? What further research could clarify the remaining uncertainties?
  5. How likely are these findings to generalise to regions with different epidemiological profiles (worm types, intensity) or different educational systems and costs?

Conclusion

Reflections

  • Munshi (2003) Takeaways:
    • Networks demonstrably influence labour outcomes.
    • Combining FE and IV offers a strategy to identify causal network effects amidst selection and endogeneity.
    • Discovering plausible natural experiment variation (like rainfall) can be powerful for IV.
  • Miguel & Kramer (2004) Takeaways:
    • Externalities are often significant and substantial.
    • Overlooking them results in biased impact estimates and potentially flawed policy decisions.
    • Cluster randomisation / phase-in designs are valuable tools for identifying overall effects and quantifying spillovers.
  • Overarching Points:
    • The standard RCT framework often requires adaptation for real-world phenomena like networks and spillovers.
    • A deep understanding of the context and potential sources of bias is essential for crafting credible identification strategies.
    • Innovative research designs can yield robust causal insights even when facing observational challenges (IV) or complex interventions (externalities).
    • The process of scientific inquiry involves scrutiny, replication, and transparency, as exemplified by the “Worm Wars.”

Thanks very much! 😊

See you next time! 👋