Source: Knittel et al (2024)

• Now we have the final list of students enrolled in the course!
  
• So we can start thinking about your project 🤓
• It will be a group project, with 3-4 students per group
• You will have to submit a pre-analysis plan, I will create a dataset for you to work on, and you will have to write a report
• and I have a few questions for you (don’t worry, they’re easy! 😂)

• We can add covariates to experimental models to increase precision
• The same way we do in any regression:
  • \(Y_i = \alpha + \beta \text{T}_i + \gamma \text{X}_i + \epsilon_i\)
• Freedman (2008) demonstrated that pre-treatment covariate adjustment may bias estimates of average treatment effects, mainly in small samples
• Lin (2013) proposed that if covariate correlations differ between treatment and control groups, centering and interacting covariates with the treatment variable will balance the groups
• This adjusts for covariates separately in each group, which is equivalent to including treatment-by-covariate interactions
  • \(Y_i = \alpha + \beta \text{T}_i + \gamma \text{W}_i + \delta \text{T}_i \times \text{W}_i + \epsilon_i\)
  • Where \(W_i = \text{X}_i - \bar{\text{X}}\) and \(\bar{\text{X}}\) is the mean of \(\text{X}_i\)
  • For ease of interpretation, we can centre the covariates to have a mean of zero
• Anyway, worry not! 😅 Our friends at DeclareDesign have done all the work for us and created a function called lm_lin() that does everything automatically!

• A hypothesis test chooses whether or not to reject the null hypothesis based on the data we observe.
• Rejection based on a test statistic, \(T_n = T(Y_1, ..., Y_n)\)
  • Will help us adjudicate between the null and the alternative
  • Typically: larger values of \(T_n → null\) less plausible
• The rejection region, \(R\), contains the values of \(T_n\) for which we reject the null.
  • These are the areas that indicate that there is evidence against the null.
• Two-sided alternative (our focus):
  • \(H_0 : \mu_y - \mu_x = 0\) and \(H_a : \mu_y - \mu_x \neq 0\)
  • Implies that \(T_n >> 0\) or \(T_n << 0\) will be evidence against the null
  • Rejection regions: \(|T_n| > c\) for some value \(c\)
• How to determine these regions?

• The rejection region is determined by the critical values, \(c\), which are defined by the significance level, \(\alpha\)

• There is an R package for that! 😊
• The ri2 package provides tools for randomisation inference, and it was made by the same people who created the estimatr package
• You can find more information about the package here
• Just install it with install.packages("ri2") and load it with library(ri2)
• The package supports all the randomisation procedures that can be described by the randomizr package
• Let’s see how it works!
• Another package that I found useful (and broadly similar to ri2) is the ritest package, available on GitHub: https://github.com/grantmcdermott/ritest

• Gerber and Green (2012) describe a hypothetical experiment in which 2 of 7 villages are assigned a female council head and the outcome is the share of the local budget allocated to water sanitation
• Their table 2.2 describes one way the experiment could have come out

library(ri2)
library(estimatr)

# Create the data
table_2_2 <- data.frame(T = c(1, 0, 0, 0, 0, 0, 1),
                        Y = c(15, 15, 20, 20, 10, 15, 30))

summary(lm_robust(Y ~ T, data = table_2_2))

Call:
lm_robust(formula = Y ~ T, data = table_2_2)

Standard error type:  HC2 

Coefficients:
            Estimate Std. Error t value Pr(>|t|) CI Lower CI Upper DF
(Intercept)     16.0      1.871  8.5524  0.00036    11.19    20.81  5
T                6.5      7.730  0.8409  0.43876   -13.37    26.37  5

Multiple R-squared:  0.2485 ,   Adjusted R-squared:  0.09824 
F-statistic: 0.7071 on 1 and 5 DF,  p-value: 0.4388

• Alternative explanations if results are consistent with hypotheses:

  • Alternative theories: Increased community engagement and public awareness, that generates other forms of engagement and pressuring. Increased media coverage of school construction could coincide with the programme

• Hypothesis for alternative outcome: With an increase in media attention regarding public works, we would expect an overall improvement in public services, not a change that is specific to the treatment group.

• Alternative explanations if results are inconsistent with hypotheses:

  • Alternative theories: Difficulties in differentiating political corruption from budget issues; the lack of political pressure; the lack of trust in representatives
  • Hypothesis for alternative outcome: Citizens may not trust their representatives and think the problems are related to austerity instead of corruption

• Population of interest: Brazilian municipalities & citizens; school construction projects.
• Where and when: Brazil; Intervention 1: Aug 2017-July 2018; Intervention 2: Aug 2018-July 2019.
• Context match?: Matches, but economic crisis & electoral cycle are specific conditions.
• Sample size:
  • Intervention 1: 344 control, 2642 treatment municipalities.
  • Intervention 2: 659 control, 3717 treatment schools.
• Sample selection:
  • Intervention 1: Random assignment of municipalities.
  • Intervention 2: Random assignment of schools, stratified by state, construction status, spending.
• Consent
  • Obtained?: No. App use was voluntary, data focused on public works.
  • Vulnerable population?: No coercion risk; participation was voluntary.
• Ethics:
  • Power sufficient?: Yes. Sample size provides power to detect plausible effects.
  • Size necessary?: Yes. Sample was determined by power analysis, not unnecessarily large.
  • Risk of targeting?: No. Results inform general policy, not targeted at individuals.

• Randomisation strategy:
  • Intervention 1: Cluster randomisation at the municipal level.
  • Intervention 2: Blocked randomisation at the school level.
• Blocks:
  • Intervention 1: No blocks.
  • Intervention 2: Blocks created based on the strata (Brazilian states, construction status, and municipal spending median).
• Clusters:
  • Intervention 1: Clusters are municipalities. 344 control clusters and 2642 treatment clusters.
  • Intervention 2: Not clustered.
  • Randomising at the individual level was not feasible.

• Estimator:
  • Linear Regression Model with a treatment indicator and control variables.
• Standard Errors:
  • Cluster-robust standard errors at the municipality level for intervention 1.
  • Cluster-robust standard errors at the school level for intervention 2.
• Test:
  • Randomisation Inference tests are employed in addition to standard t-tests for p-values
• Missing Data:
  • Missing data is not a significant concern.
• Effect size:
  • Expected effect size is based on prior studies and theoretical expectations.
  • Minimum effect sizes are not determined a priori.
  • Similar studies had mixed effect sizes.
• Power:
  • Sample sizes were chosen to detect effect sizes that are likely to have practical implications on policy.

• Randomisation:
  • Randomisation was conducted on a computer.
• Implementation:
  • Transparência Brasil implemented the intervention by creating the app and by making it available in a subset of municipalities/schools.
  • No direct dangers to the research team or enumerators, as all data was collected through APIs and public databases.
  • Implementation was tracked via app downloads and user sessions.
• Compliance:
  • Compliance measured by analysing whether the app was available for use in a given location.
  • All compliance data came from the app store and from usage logs.
• Data management:
  • Data was stored securely on cloud servers.
  • Data was anonymised by using city/school identification and not user’s information.
  • Publicly available data was collected directly via APIs or downloaded directly from the ministry’s site.