• Imagine you are trying to improve school grades
• You think that awarding a prize to students will help motivate them
• You randomly assign students to treatment and control groups
• You find a positive effect of the program on test scores
• But you also find that depending on who gets the treatment, the effect varies

• You find that the results vary depending on who wins the prize
• Let’s calculate the ATEs in this case?

• If we simply calculate the ATE using the last column as \(Y_{i}(0)\) and the column corresponding to \(i\)’s award as \(Y_{i}(1)\), we have: \(\frac{((10 - 7) + (5 - 5) + (9 - 9))}{3} = 1\)
• However… if we randomise the treatment, we have these results:
• If Alice wins the prize, the ATE is \(10 - \frac{5+9}{2} = 3\)
• If Bob wins the prize, the ATE is \(5 - \frac{5+5}{2} = 0\)
• If Charlie wins the prize, the ATE is \(9 - \frac{7+5}{2} = 3\)
• The average of these three is \(2\), which is different from the first calculation

• Interference can be a nuisance, but also an opportunity
• So it is a good idea to model it explicitly in our analysis
• In this case, we wouldn’t need to assume “no spillovers”, but rather “not modelled spillovers”
• Good design can help us leverage spillovers to our advantage
• There are several methods to do this:
  • Clustered randomisation (this one you already know!)
  • Multi-level designs (e.g., schools within districts)
  • Within-subject designs (e.g., repeated measures)
  • Waitlist designs (e.g., staggered rollouts)
• Let’s see them in more detail 😎

• Multi-level designs are nested experiments
• This means that we have multiple levels of randomisation
• For example, in a school-based intervention:
  • Schools are randomised to treatment or control
  • Within schools, students are also randomised
• Political interventions can also be nested:
  • We randomise the treatment at the district level
  • Then we randomise the treatment again at the voter level
• But we need to expand our potential outcomes notation to account for this…

• Potential outcomes depend on both own and housemate’s treatment status
• Revised notation system accounts for dual treatment influences (\(Y_{ab}\) where):
  • a = housemate’s treatment status (0=control, 1=treated)
  • b = own treatment status (0=control, 1=treated)

• Key assumption: Strict household containment
  • Outcomes only affected by within-household interventions
  • No interference from external treatment assignments

• From the four potential outcomes, key estimands emerge:
  • \(Y_{01} - Y_{00}\): Direct effect of treatment when housemate is untreated
  • \(Y_{11} - Y_{10}\): Treatment effect when housemate receives mail
  • \(Y_{10} - Y_{00}\): Spillover effect on untreated with treated housemate
  • \(Y_{11} - Y_{01}\): Spillover interaction among treated pairs
• Critical considerations:
  • Effects depend on housemate’s treatment status
  • Estimands capture combined influences of:
    • Direct intervention impacts
    • Communication spillovers
    • Shared resource effects
  • Does not isolate specific mechanisms behind spillovers

• So far, we have weakened the SUTVA assumption by considering interference in a small setting (households)
• Sometimes, spillovers are not confined to specific units, but spread across space
• This is particularly common in urban settings:
  • Crime in adjacent neighbourhoods
  • Pollution in nearby areas
  • Economic development in neighbouring regions
  • Health outcomes in surrounding communities
• While it seems tempting to just include a measure of proximity in our models, it is not that simple…
  • Not easy to be very precise about spillover effects
  • Not valid when spillovers are not spatially confined (e.g., pollution)
  • Physical proximity does not always make sense (e.g., social networks, phones)

• Imagine you want to improve healthcare in five villages
• Assume also that the proper distance between villages is known, so this is not a spatial problem
• Subjects reside in villages A, B, C, D and F, and E is unoccupied
• Only one village will receive the treatment, which is a new healthcare facility
• There are three types of potential outcomes:
  • \(Y_{01}\): Healthcare level of village X if it receives the treatment
  • \(Y_{10}\): Healthcare level of village X if the adjacent village receives the treatment
  • \(Y_{00}\): Healthcare level of village X if village X or its neighbours do not receive the treatment

• Notice that some villages can never have some outcomes
• Village E is unoccupied, so it is not included in the analysis
• NA indicates that Village F can never be adjacent to a treated village
• In this case, location F can never manifest a \(Y_{10}\) outcome
• As this potential outcome can never be observed, we exclude F from the definition of the average treatment effect \(E[Y_{10} - Y_{00}]\)

• Second, the probability of assignment to each treatment condition varies from one observation to the next
• For instance, village A has a 0.20 probability of being exposed to spillovers from an adjacent treated location, whereas the village at location B has a 0.40 probability
• As we have seen previously, we should weight observations by inverse probability of assigned experimental condition, excluding subjects with zero probability (e.g., F)

• The key assumption in within-subject designs is that the intervention is the only thing that changes
• Therefore, we need to make two additional assumptions:
  • No-anticipation: Subjects do not know when the intervention will occur, so they cannot change their behaviour in anticipation
  • For example, if we are studying the effect of a new policy on crime rates, we need to make sure that criminals do not know when the policy will be implemented
  • No persistence: Potential outcomes in one period are unaffected by treatments administered in prior periods
  • This is a bit tricky, as we are assuming that the effect of the treatment disappears after a certain period
  • Experimenters often include “washout periods” between experimental sessions so as to allow the previous period’s effects to dissipate

• Clifford et al (2021) evaluate how different “pre-post” designs can improve experimental estimates, mainly by reducing standard errors
• Regular experiments (“post only”) have relatively low precision because the outcome is measured just once
• This is particularly problematic when the outcome is noisy or if the experiment involves multiple treatment arms, moderators, or small treatment effects
• Pre-post designs measure the outcome prior to the experimental manipulation at point \(t_0\) and after the manipulation at point \(t_1\) (as we’ve just seen)
• The pre-post design can also be a between-subjects design because some respondents are never exposed to the treatment; respondents difference scores are compared between groups

• The final type of design we will discuss today is the waitlist design, also known as the stepped-wedge design
• Their scientific value comes from their ability to track treatment effects among several subjects as they play out over time
• Waitlists play a diplomatic role because they overcome the problem of withholding treatment from a control group (remember our ethics discussion?)
• In this design, every subject is treated eventually; random assignment determines when they receive treatment
• It is a hybrid between a within-subject and a between-subject design, and it is widely used in public health interventions (e.g., vaccine rollouts)

Source: Hemming et al (2014)

• The immediate treatment effect is \(Y_{01} - Y_{00}\), that is, the effect of being treated in the current period but not in the preceding period
• We just need to take the numbers from the tables and apply inverse probability weighting again

\[ \begin{aligned} \widehat{E}[Y_{01} - Y_{00}] &= \frac{\frac{7 + 10}{0.25} + \frac{2 + 1}{0.25} + \frac{7 + 10}{0.25}}{\frac{2}{0.25} + \frac{2}{0.25} + \frac{2}{0.25}} \\ &- \frac{\frac{7 + 1 + 4 + 4 + 3 + 2 + 3}{0.75} + \frac{5 + 3 + 3 + 3}{0.50} + \frac{3 + 4}{0.25}}{\frac{6}{0.75} + \frac{4}{0.50} + \frac{2}{0.25}} = 2.72. \end{aligned} \]

• Finally, we will estimate the cumulative effect of TV advertising, which is \(Y_{11} - Y_{00}\), that is, the effect of being treated in both the preceding and current periods
• We do the same thing again, but now we consider the \(Y_{11}\) potential outcomes
• However, we must restrict our attention to the second and third weeks, because this type of treatment cannot occur in the first week

\[ \begin{aligned} \widehat{E}[Y_{11} - Y_{00}] &= \frac{\frac{9 + 8}{0.25} + \frac{4 + 10 + 10 + 3}{0.50}}{\frac{2}{0.25} + \frac{4}{0.50}} \\ &- \frac{\frac{5 + 3 + 3 + 3}{0.50} + \frac{3 + 4}{0.25}}{\frac{4}{0.50} + \frac{2}{0.25}} = 4.13. \end{aligned} \]

• Spillovers are a common problem in social science research
• They can be positive or negative, and they can be modelled explicitly in our analysis
• There are several methods to deal with spillovers, such as multi-level designs, within-subject designs, repeated-measures experiments, and waitlist designs
• Each design has its advantages and disadvantages, and the choice of design should be based on the research question and the context
• You can use DeclareDesign to simulate spillover designs and pretest posttest designs
• The statistical analysis of waitlist designs are a little tricky, but you can use the swCRTdesign package in R to help you with that