QTM 385 - Experimental Methods

Lecture 06 - Texts for Discussion and More

Danilo Freire
danilo.freire@emory.edu

Department of Data and Decision Sciences
Emory University

Hi, there! 👋
Tudo bem? 😄🇧🇷

Brief recap 📚

Last time, we…

  • Covered adding covariates with lm_lin(), centring, and sub-group analysis with interaction terms using lm_robust.
  • Compared the Neyman-Rubin model (ATE) with the Ronald Fisher model (sharp null hypothesis)
  • Talked about Type I and Type II errors
  • Discussed sampling variability and how it affects the interpretation of experimental estimates
  • Introduced randomisation inference with the ri2 package
  • Demonstrated RI with two examples, including one about female council heads and the other about runoff elections
  • Real experiment: Analysed a paper using linear regression models as well as randomisation inference

Today, we will…

Group Work 🤝

What I have in mind…

  • I’d like you to start working on your pre-analysis plan as soon as possible 🤓
  • I’ll give you some time to discuss your ideas and start writing your plan
  • My plan is that you should slowly think about your final project, one step at a time
  • What do you think about having two weeks to write the following:
    • Submit at most 2 paragraphs summarising an experiment that you want to develop in this course. At minimum, your summary should include a research question, why the question is important, and a rough sketch of how you plan to answer the question
  • In three weeks:
    • Write a title and abstract for a paper you imagine writing based on your proposed experiment. Assume that your findings align with your theoretical predictions. Remember to establish why the findings matter for your intended audience
  • In four weeks:
    • Outline your pre-analysis plan. Your outline should include sections on the research question, the experimental design, the data you will collect, and the analysis you will conduct

What I have in mind…

  • In five weeks:
    • Use Quarto and DeclareDesign to write your report and simulate your experiment
  • In six weeks:
    • Revised outline, now including a new section titled “Potential Threats.” In this section, diagnose threats and briefly describe potential countermeasures. This new section should discuss false positives, statistical power, demand effects, noncompliance, spillover, and attrition
  • In seven weeks:
    • Revised outline, now included a new section on “Heterogeneous Treatment Effects.” In this section, discuss how you might explore heterogeneity in treatment effects. This section should include a discussion of how you might use covariates to explore heterogeneity
  • In eight weeks:
    • I will give you feedback on your outline and provide you with the simulated data for your experiment
    • You will have two weeks to write your final report
  • Last week of class:
    • You will present your findings to the class

R packages 📦

fabricatr: Simulating data for experiments

fabricatr: Simulating data for experiments

library(fabricatr)

voters <- fabricate(
  N = 1000,
  group_id = rep(1:10, 100),
  ideology = draw_normal_icc(mean = 0, N = N, clusters = group_id, ICC = 0.7),
  ideological_label = draw_ordered(
    x = ideology, break_labels = c("Very Conservative", "Conservative", "Liberal", "Very Liberal")
  ),
  Q1_immigration = draw_likert(x = ideology, min = -5, max = 5, bins = 7),
  Q2_defence = draw_likert(x = ideology + 0.5, min = -5, max = 5, bins = 7),
  treatment = draw_binary(0.5, N = N),
  proposition_vote = draw_binary(latent = ideology + 1.2 * treatment, link = "probit")
)

head(voters)
    ID group_id   ideology ideological_label Q1_immigration Q2_defence
1 0001        1 -3.1214379 Very Conservative              2          2
2 0002        2  0.5875727           Liberal              4          5
3 0003        3  1.2026365      Very Liberal              5          5
4 0004        4 -1.0366495 Very Conservative              3          4
5 0005        5  0.2787654           Liberal              4          5
6 0006        6 -0.1366222      Conservative              4          4
  treatment proposition_vote
1         1                0
2         1                1
3         0                1
4         0                0
5         0                0
6         0                1

randomizr: Randomisation for experiments

  • randomizr is another package that I use a lot
  • It is the easiest way to randomise your treatment assignment
  • It also has all the types of randomisation we use in experiments
    • Simple randomisation with simple_ra()
    • Complete randomisation with complete_ra()
    • Block randomisation with block_ra()
    • Cluster randomisation with cluster_ra()
    • Block and cluster randomisation with block_and_cluster_ra()
  • The source code is available at https://github.com/DeclareDesign/randomizr/

randomizr: Randomisation for experiments

library(randomizr)

# Simple randomisation
set.seed(42)
N <- 100
Z <- simple_ra(N = N, prob = 0.5)
table(Z)
Z
 0  1 
45 55 
# Complete randomisation 
Z <- complete_ra(N = N, m = 50)
table(Z)
Z
 0  1 
50 50 
# Block randomisation 
set.seed(42)
N <- 100
participants <- data.frame(
  id = 1:N,
  gender = sample(c("Female", "Male"), N, replace = TRUE)
)

participants$assignment <- block_ra(
  blocks = participants$gender
)
table(participants$assignment, participants$gender)
   
    Female Male
  0     22   28
  1     22   28
# Cluster randomisation
set.seed(42) 
students <- data.frame(
  student_id = 1:2000,
  school_id = rep(1:10, each = 200) 
  # Each school (10) is a cluster of 200 students
)

# `m = 5` to assign 5 clusters to the treatment condition
students$assignment <- cluster_ra(
  clusters = students$school_id, m = 5
)
table(students$assignment)

   0    1 
1000 1000

estimatr: Estimating treatment effects

  • We have already seen estimatr in action
  • The package is particularly useful to estimate linear models with robust standard errors
    • Economists use OLS for everything 😅
  • But you can also use it to estimate instrumental variables models, difference-in-differences, and more
  • estimatr integrates well with the tidyverse, so you can use it with dplyr, ggplot2, and other packages
  • Source code: https://github.com/DeclareDesign/estimatr

estimatr: Estimating treatment effects

library(estimatr)
fit <- lm_robust(Fertility ~ Agriculture + Catholic, data = swiss)
tidy(fit)
         term   estimate  std.error statistic      p.value    conf.low
1 (Intercept) 59.8639237 5.47384281 10.936361 3.934173e-14 48.83211840
2 Agriculture  0.1095281 0.10305115  1.062852 2.936478e-01 -0.09815783
3    Catholic  0.1149621 0.03854836  2.982283 4.651169e-03  0.03727301
   conf.high df   outcome
1 70.8957290 44 Fertility
2  0.3172141 44 Fertility
3  0.1926512 44 Fertility

estimatr: ggplot2 integration

library(ggplot2)
ggplot(swiss, aes(x = Agriculture, y = Fertility)) +
  geom_point() +
  geom_smooth(method = "lm_robust") +
  theme_bw()

library(dplyr)
fit %>% 
  tidy %>% 
  filter(term != "(Intercept)") %>%
  ggplot(aes(y = term, x = estimate)) + 
  geom_vline(xintercept = 0, linetype = 2) + 
  geom_point() + 
  geom_errorbarh(aes(xmin = conf.low, xmax = conf.high, height = 0.1)) + 
  theme_bw()

ri2: Randomisation inference

library(ri2)
N <- 100
declaration <- declare_ra(N = N, m = 50)

Z <- conduct_ra(declaration)
X <- rnorm(N)
Y <- .9 * X + .2 * Z + rnorm(N)
dat <- data.frame(Y, X, Z)

ri_out <-
  conduct_ri(
    formula = Y ~ Z,
    declaration = declaration,
    assignment = "Z",
    sharp_hypothesis = 0,
    data = dat
  )
summary(ri_out)
plot(ri_out)
  term   estimate two_tailed_p_value
1    Z -0.2361121              0.396

Questions? 😉

Papers for Discussion 📚

Kalla, J. & D. Broockman. 2015

Campaign Contributions Facilitate Access to Congressional Officials: A Randomized Field Experiment

Article link and replication data

Overview

  • Overview
    • Do campaign donations secure preferential treatment from policymakers?
    • Challenge: It is hard to isolate the effect of donations on policymakers’ behaviour
    • First randomised field experiment on campaign contributions and access
    • Do donations facilitate access to influential policymakers?
  • The Experiment
    • A political organisation tried to schedule meetings between congressional offices and their members (who were donors)
    • The organisation randomly revealed to offices if attendees were donors
    • Key finding: When informed attendees were donors, policymakers were available 3-4 times more often
    • Underscores concerns about campaign finance deregulation
  • Hypotheses
    • H1: Senior policy makers will make themselves available more often when they know that prospective attendees are political donors
    • H2: That knowing about donations matters more when scheduling meetings with higher level officials
  • Why Should We Care?
    • Political Inequality: Campaign donations may amplify the voices of the wealthy
    • Policy Decisions: Understanding who has access informs how policies are made
    • Democracy: This research is crucial for a more equitable political system
  • Policy Relevance
    • Campaign finance reform: Findings can inform debates about the necessity of these reforms
    • Regulation: This study helps to clarify the relationship between campaign donations and access to power
    • Supreme Court decisions: Results bear on recent deregulation (at the time)

Intervention

  • Intervention
    • A grassroots political organisation attempts to schedule meetings with congressional offices
    • Whether congressional offices are informed that attendees are political donors
    • Control Condition: Offices are informed that the meeting attendees are “local constituents”
  • Outcome Measurement
    • Primary Outcome: The seniority/level of the official who agreed to attend the meeting
    • Rank Order: The access level was ranked from 1 (most desirable: member of Congress) to 6 (least desirable: no meeting)
    • Data Collection: The level of staff who agreed to the meeting was recorded
    • Meeting: Attendees confirmed that the promised staffer attended the meeting
  • Random Assignment
    • Blocking: The researchers blocked congressional offices into triplets based on factors that could be associated with legislative access (prior voting record, cosponsorship of the bill, years of service, ideology, local population)
    • Within the blocks, offices were randomly assigned to the ‘donor revealed’ condition
    • 191 congressional offices were contacted, 96 in the treatment group and 95 in the control group

Treatment

Follow-up

Results

Results (cont.)

Discussion

  • Implications
    • The study provides evidence that campaign donations can facilitate access to policymakers
    • The findings underscore concerns about the influence of money in politics
    • The results suggest that campaign finance reform may be necessary to ensure a more equitable political system
  • Limitations
    • Mechanisms: The exact reasons why legislators reacted to donations needs further study
    • Causal Chain: The link between access and influence requires further investigation
    • Generalisability: More studies with different actors, organisations and contexts are required

Bertrand and Mullainathan (2004) 🇺🇸

Bertrand, M., & Mullainathan, S. (2004)

Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination

Article link and replication data

Overview

  • Overview
    • Research Question: Is there racial bias in callback rates for job interviews?
    • Challenge: Difficult to isolate racial discrimination from other factors
    • Method: A field experiment using fictitious resumes
    • Manipulation: Randomly assigned names to signal race (White vs. African-American)
  • The Experiment
    • Fictitious Resumes: Sent to real job ads in Boston and Chicago
    • Name Manipulation: Resumes randomly assigned White-sounding or African-American-sounding names
    • Callback Rate: Measured the rate at which resumes received interview requests
    • Resume Quality: Varied to test if bias differs by qualification
  • Hypotheses
    • H1: White names will receive a significantly higher callback rate
    • H2: The racial gap in callback rates will persist even for higher quality resumes
  • Why Should We Care?
    • Racial Inequality: Persistent racial disparities in the labour market
    • Discrimination: Understanding the mechanisms of racial discrimination
    • Policy Implications: Informative for policies aimed at promoting equal opportunity
  • Policy Relevance
    • Affirmative Action: Assessing the effectiveness of current policies
    • Anti-discrimination Law: Evidence for the continued relevance of anti-discrimination efforts
    • Social Justice: Highlighting ongoing challenges to racial equality

Intervention

  • Experimental Design
    • Field Experiment: Real-world setting of job applications
    • Correspondence Testing: Methodology of sending matched resumes with manipulated variables
    • Random Assignment: Names randomly assigned to resumes
  • Intervention: Name Manipulation
    • White Names: Emily Walsh, Greg Baker (distinctively White-sounding)
    • African-American Names: Lakisha Washington, Jamal Jones (distinctively African-American-sounding)
    • Control: Implicit control through random assignment – each resume acts as its own control when names are varied
  • Outcome Measurement
    • Primary Outcome: Callback for an interview (binary - yes/no)
    • Measurement: Tracked responses to resumes (phone or email)
    • Robustness: Consistent measurement across conditions
  • Creating Resumes
    • Resume Templates: Based on real resumes from job search websites (Boston & Chicago)
    • Quality Variation: Created “high-quality” and “low-quality” resumes with subtle differences in experience and skills
    • Matching: Resumes tailored to fit specific job ad requirements
    • Large Scale: Over 5000 resumes sent in response to over 1300 job ads
  • Sample and Data Collection
    • Job Ads: Newspaper ads in Boston Globe and Chicago Tribune (Sales, Admin, Clerical, Customer Service).
    • Cities: Boston and Chicago (conducted in 2001-2002)
    • Data: Callback rates for each resume type, analysed by name, resume quality, job/employer characteristics

Results

Results

Results

Discussion

  • Key Findings
    • Significant Racial Gap: White names receive 50% more callbacks
    • Resume Quality Matters Less for African-Americans: Improved credentials do not close the racial gap
    • Uniformity of Gap: Bias is consistent across occupations, industries, and employer types
  • Implications
    • Persistent Discrimination: Racial bias remains a significant factor in the labour market
    • Limited Returns to Human Capital: Efforts to improve skills may not equally benefit all racial groups in terms of job access
    • Challenges for Policy: Highlights the complexity of addressing racial inequality, requiring more than just skills-based interventions
  • Limitations
    • Callback as Outcome: Measures access to interview, not job offers or wages
    • Name as Proxy for Race: Confounding factors (social class inferences from names) are possible, although tested against
    • Generalisability: Focus on specific job categories and cities (Boston & Chicago)

Chattopadhyay and Duflo (2004) 🇮🇳

Chattopadhyay, R., & Duflo, E. (2004)

Women as Policy Makers: Evidence from a Randomized Policy Experiment in India

Article link

Overview

  • Overview
    • How does women’s leadership affect policy decisions?
    • Challenge: It is hard to isolate the effect of representation on policy choices
    • India: A unique experiment with reserved seats for women in local government
    • A quasi-experimental study to investigate this effect
  • The Experiment
    • Randomised: One-third of Village Council head positions were randomly reserved for women
    • Dataset collected in 265 Village Councils in West Bengal and Rajasthan
    • Comparison: type of public goods in reserved and unreserved Village Councils
    • Key Finding: Reservation affects the types of public goods provided, with leaders investing more in infrastructure relevant to their own gender
  • Hypotheses
    • H1: Leaders will invest more in infrastructure that is directly relevant to the needs of their own genders
    • H2: The effect of reservation is caused by gender, and not other characteristics of women leaders
  • Why Should We Care?
    • Underrepresentation: Women are underrepresented in political positions worldwide
    • Policy Impact: Understanding if women make different policy choices
    • Political Equality: Is vital for representative democracy
  • Policy Relevance
    • Affirmative Action: Findings inform debates about the necessity of affirmative action in politics
    • Decentralisation: Results shed light on the consequences of decentralising power in India
    • Institutions: Helps understand how the design of political institutions can affect outcomes

Intervention

  • Context
    • India: A constitutional amendment required the reservation of one-third of all positions of village chief (Pradhan) to women
    • Implementation: This was implemented in two states: West Bengal and Rajasthan
    • GPs: Local Village Councils, which have influence on local public goods
  • Intervention
    • Randomisation: GPs were randomly selected to be reserved for women
    • Treatment: Whether the village council was randomly reserved for a woman leader
      • Control Condition: The council is open to all candidates
  • Outcome Measurement
    • Primary Outcome: The types of public goods provided
    • Focus: Investments in specific goods such as drinking water, roads, and other local infrastructure
    • Secondary Outcome: requests and complaints made by different genders
  • The Process
    • Allocation of GPs for reservation: This was done based on specific pre-existing lists, and the selection was effectively random
    • Implementation: A specific set of rules ensures the random selection
    • Data collection: Surveys of the investment made in the villages, and interviews with the village leaders
    • Surveys collected in the summer of 2000 (West Bengal and Rajasthan) and at the end of 2002 (Rajasthan)

Balance test

Results

Results (cont.)

Results (cont.)

Discussion

  • Implications
    • Policy choices are affected by the gender of the decision maker
    • Women leaders invest more in public goods aligned with women’s priorities, and vice versa
    • This effect is causal, and not driven by characteristics of women leaders, but their gender per se
  • Theoretical Contribution
    • Evidence of the effect of identity on the policymaking process
    • Contradicts Downsian models of policy preferences (which assume that politicians are purely self-interested)
    • Shows the importance of the design of political institutions in ensuring the representation of different social groups
  • Limitations
    • The study focuses on local governance in India, and may not generalise to other contexts
    • The model does not consider dynamic effects of the policy
    • Mechanisms: the exact mechanisms of the decision-making process are still not fully clear

Questions? 😉

And that’s it for today! 🎉

All the very best!
See you soon! 🚀