• Two-sided non-compliance occurs when both treatment and control groups experience imperfect compliance
• Four compliance types:
  • Compliers: Follow treatment assignment
  • Never-takers: Never receive treatment
  • Always-takers: Always seek treatment
  • Defiers: Do opposite of assignment (rare)
• Monotonicity assumption required (no defiers) for identification
• IV methods estimate CACE: \(\tau_{LATE} = \frac{ITT_Y}{ITT_D}\)

• Analysis methods:
  • Inverse probability weighting (IPW)
  • Extreme bounds analysis (EBA)
  • Lee bounds (trimming the bounds)
• IPW estimates ATE when missingness is related to observed variables
• EBA is a robustness check that evaluates the sensitivity of the ATE to different assumptions about the missing data
• Lee bounds are a way to trim the bounds to make them more informative

• RAND Health Insurance Experiment (HIE) was a large-scale study in the 1970s that randomly assigned families to different insurance plans
• The goal was to assess the impact of insurance on health outcomes
• The experiment covered 5%, 50%, 75%, or 100% of each family’s health costs, with the remaining covered by the family
• to insure families in the first three experimental groups against catastrophic financial loss, costs over $1,000 were covered by the experimental insurance plan
• The study found that insurance coverage did not significantly affect health outcomes
• The conclusion was that public health insurance was not cost-effective, thus the government did not need to expand it
• Overall, the study cost about $550 million in today’s dollars 😮
• However, the study had high attrition rates, which may have biased the results!

Column (1) shows the average for the group assigned catastrophic coverage. Columns (2)–(5) compare averages in the deductible, cost-sharing, free care, and any insurance groups with the average in column (1).

Source: Angrist and Piscke (2014, 19)

• You’re all familiar with the potential outcomes framework by now 🤓
• Remember our variables from last class?
  • \(Y_i(0)\) and \(Y_i(1)\): Potential outcomes for individual \(i\)
  • \(z_i\): Treatment assignment
  • \(d_i\): Actual treatment received by individual \(i\)
  • \(Y_i\): Observed outcome for individual \(i\)
• Here, let’s first assume that there is full compliance with treatment assignment, that is, \(d_i = z_i\)
• So far, so good? 😊
• Now, let’s introduce attrition to the mix and use a similar notation

• Bias arises if the characteristics of those who stay in the study \((R_i(z) = 1)\) are systematically different from those who drop out (\(R_i(z) = 0)\), and if these differences are related to the potential outcomes \(Y_i(z)\)
• In other words, if attrition is non-random, we have a problem!
• We are essentially missing a piece of the puzzle (the dropouts), and that missing piece can distort our view of the true treatment effect
• Sadly, if there’s non-random attrition, the difference-in-means estimator not recover the ATE for the entire subject pool, and it will not recover the ATE for any meaningful subgroup either!
• Why?
  • Attrition might be non-random within each of these baseline subgroups
  • For example, maybe younger people in the treatment group drop out more than older people, but the opposite can be true in the control group!

• We can compensate for missing data by weighting the observed data
• The idea is to upweight the observations that are similar to the missing ones
• Let me give you a simple example:
  • Suppose we have 40 people in our experiment
  • 30 are men and 10 are women
  • The overall average would be \(\frac{30}{40} \times\) men’s average + \(\frac{10}{40} \times\) women’s average
• Now assume that 15 out of the 30 men drop out
• “Controlling for gender” (i.e., weighting):
  • The remaining 15 men produce an unbiased estimate of the average amongst 30 men
  • So we can just count them twice!
  • This is the essence of inverse probability weighting (IPW)

• IPW estimates ATE when \(MIPO|X\) holds
• To estimate ATE, we need \(E[Y_i(1)]\) and \(E[Y_i(0)]\) (as you know!)
• When \(MIPO|X\) holds, \(E[Y_i(1)]\) is weighted average:

\[E[Y_i(1)] = \frac{1}{N} \sum_{i=1}^{N} \frac{Y_i(1) r_i(1)}{\pi_i(z = 1, x)},\]

• \(\pi_i(z = 1, x)\): share of non-missing subjects among treated with covariate profile \(x\)
• Missing outcomes have no effect on sum
• Reported outcomes weighted by \(1/\pi_i(z = 1, x)\)
• Weights replace missing values with copies of non-missing values
• Weighting scheme called inverse probability weighting because observations weighted by inverse probability of being observed

• Treatment: Educational programme at a community centre
• \(X_i\): Covariate indicating proximity to the center
  • \(X_i = 1\): Lives near the community center
  • \(X_i = 0\): Lives far away
• Follow-up evaluation at the community center
• Attrition is related to \(X_i\): People living far are less likely to show up
• Potential outcomes differ by \(X_i\): People living far away have higher potential outcomes
• Let’s see how it can recover ATE when \(MIPO|X\) holds!

• ATE with complete data: \(E[Y_i(1)] - E[Y_i(0)] = 3.5\)
• Relevant proportions of non-missingness:
  • \(\pi_i(z = 1, x = 1) = 1\)
  • \(\pi_i(z = 1, x = 0) = 1\)
  • \(\pi_i(z = 0, x = 1) = 1\)
  • \(\pi_i(z = 0, x = 0) = 0.5\)
• Applying the weighted average formula:

\[E[Y_i(1)] = (\frac{1}{8}) (\frac{4}{1} + \frac{7}{1} + \frac{4}{1} + \frac{7}{1} + \frac{14}{0.5} + \frac{18}{0.5}) = 10.75,\]

\[E[Y_i(0)] = (\frac{1}{8}) (\frac{3}{1} + \frac{4}{1} + \frac{3}{1} + \frac{4}{1} + \frac{10}{0.5} + \frac{12}{0.5}) = 7.25,\]

\[E[Y_i(1)] - E[Y_i(0)] = 3.5.\]

• IPW recovers the true ATE! 🎉

• Incorrect \(MIPO|X\) assumption leads to misleading estimates
• IPW gives largest weights to high-attrition subgroups
• Biased subgroup estimate can disproportionately influence overall ATE
• IPW average may be more biased than unweighted data
• Re-weighting increases sampling variability
  • Extra weight on subsamples with many missing observations
• \(MIPO|X\) discussion: hypothetical scenarios with known potential outcomes
• In practice, researchers must make assumptions about attrition
• Evaluating \(MIPO|X\): detective work and speculation 🕵️‍♂️

• Now, let’s analyse the worst case scenario: when attrition is so severe that it invalidates the \(MIPO|X\) assumption
• In this case, there is no clear way to recover the ATE
• But we can do something about it!
• Extreme bounds analysis (EBA - Manski bounds) is a robustness check that evaluates the sensitivity of the ATE to different assumptions
• The idea is to bound the ATE by the most extreme assumptions about the missing data
  • That is, we assume the best and worst-case scenarios for the missing data
• Sensitivity methods are a growing area of research in causal inference, and EBA is one of the most popular methods

• Imagine the missing outcomes from those who dropped out

• We don’t know what those outcomes are, but we can consider two extreme scenarios:

• Best-case scenario: Assume the dropouts in the treatment group would have had the best possible outcomes, and dropouts in the control group would have had the worst possible outcomes (relative to the observed data)

• Worst-case scenario: Assume the dropouts in the treatment group would have had the worst possible outcomes, and dropouts in the control group would have had the best possible outcomes

• By calculating the ATE under these extreme assumptions, we get a range (upper bound and lower bound) that (hopefully 😅) contains the true ATE

• To find the upper bound on the treatment effect estimate, substitute 10 for the missing values in the treatment group and 0 for the missing value in the control group

• Upper bound: \(\frac{(7 + 10 + 10 + 10)}{4} - \frac{(0 + 7 + 5 + 6)}{4} = \frac{37}{4} - \frac{18}{4} = 4.75\)

• To find the lower bound on the treatment effect estimate, substitute 0 for the missing values in the treatment group and 10 for the missing value in the control group

• Lower bound: \(\frac{(7 + 10 + 0 + 0)}{4} - \frac{(10 + 7 + 5 + 6)}{4} = \frac{17}{4} - \frac{28}{4} = -2.75\)

• So, the ATE is between -2.75 and 4.75… and indeed it is (2)! 🎉

• But as you can see, the bounds are pretty wide…

• Lee (2009) suggested that we can trim the bounds to make them more informative (and narrower)
• The key assumption of Lee is monotonicty, similar to what we discussed in the context of non-compliance
• Lee proposed that we have four types of respondents, defined with respect to specific post-treatment outcomes (some observed, others not):
  • Never-reporters: Those who would never report their outcomes (\(R(1) = 0\) and \(R(0) = 0\))
  • Always-reporters: Those who would always report their outcomes (\(R(1) = 1\) and \(R(0) = 1\))
  • If-treated reporters: Those who would report their outcomes if treated (\(R(1) = 1\) and \(R(0) = 0\))
  • If-untreated reporters: Those who would report their outcomes if in the control group (\(R(1) = 0\) and \(R(0) = 1\))
• The assumption is that there are no if-untreated reporters, similar to the “no defiers” assumption in non-compliance

• Under monotonicity, we can bound the average treatment effect for Always-Reporters
• For them, the ATE is defined as:

\[E[Y_i(1) | R_i(1) = 1, R_i(0) = 1] - E[Y_i(0) | R_i(1) = 1, R_i(0) = 1]\]

• That is, we observe \(Y(0)\) only for the Always-Reporters, as If-Treated-Reporters and Never-Reporters produce only missing values, and under monotonicity we assume there are no If-Untreated-Reporters
• It sounds a bit confusing, because it is! 😅
• The difficult part is to estimate the ATE for Always-Reporters, as we observe \(Y(1)\) for both Always-Reporters and If-Treated-Reporters
• We can estimate the share of Always-Reporters by assuming that the share of Always-Reporters in the control group is the same as in the treatment group
• The difference is the share of If-Treated-Reporters in the treatment group, we can use the lower and upper bound of the distribution of outcomes to estimate the ATE for Always-Reporters

• We can bound the effect for always-responders by taking the means of trimmed versions of the treated outcome distribution
• A lower bound comes from trimming off the share of compliers from the top of the distribution, and an upper bound comes from trimming this share from the bottom
• There are several R packages that calculate Lee bounds
  • Have a look at https://github.com/lbassan/attritevis
• Although this is a good solution to attrition, it has received some criticism too
• For instance, McKenzie (2024) argues that it’s not clear how to use Lee bounds with binary variables, that incorporating covariates isn’t straightforward (paper proposing a solution here), and that bounds are not easy to use with clustered data

• The attritevis package has several useful functions to visualise the attrition process, calculate the bounds, and estimate the ATE
• It has very few assumptions:
  • Data must be ordered by survey questions, i.e. if respondents answered Q1 before Q2, the variable Q1 must appear before Q2 (i.e. in an earlier column) in the dataframe
  • Attrition is defined as completely leaving the survey, not just skipping a question.
  • For balance tests, treatment and control conditions must be defined
• The package is available on GitHub: https://github.com/lbassan/attritevis

• To install and load the package, run the following code:

# devtools::install_github("lbassan/attritevis", dependencies = TRUE)
library(attritevis)
library(ggrepel)

test_data <- read.csv("./test_data.csv")
names(test_data)

 [1] "X"              "consent"        "age"            "sex"           
 [5] "education"      "state"          "income"         "part_id"       
 [9] "race"           "religion"       "attrition_1"    "attrition_2"   
[13] "cards_a"        "pa"             "pb_1"           "pb_2"          
[17] "pb_3"           "pc"             "cards_b"        "p2a"           
[21] "p2b_1"          "p2b_2"          "p2b_3"          "p2c"           
[25] "treat1"         "Happy_1_1"      "Happy_1_2"      "Happy_1_3"     
[29] "cards1"         "X1a"            "X1b_1"          "X1b_2"         
[33] "X1b_3"          "X1c"            "treat2"         "Happy_2_1"     
[37] "Happy_2_2"      "Happy_2_3"      "cards2"         "X2a"           
[41] "X2b_1"          "X2b_2"          "X2b_3"          "X2c"           
[45] "treat3"         "Happy_3_1"      "Happy_3_2"      "Happy_3_3"     
[49] "cards3"         "X3a"            "X3b_1"          "X3b_2"         
[53] "X3b_3"          "post1"          "post2_7"        "post3"         
[57] "post4"          "post5"          "post6"          "post7"         
[61] "post8"          "post9"          "post10"         "post11_1"      
[65] "post11_8"       "post13_1"       "post14_1"       "post15_1"      
[69] "post16_1"       "post17"         "ideology"       "trump_approval"
[73] "pres_approval" 

• The attrition() function creates a frame that indicates, per question:
  • attrited – how many respondents attrited (left the survey) at each question
  • proportion – number of attrited respondents / number of respondents who entered survey
    
  • prop_q – number of attrited respondents / number of respondents entering into the question
  • questions – question names
  • responded – how many respondents responded in each question
    
  • prop_r – number of respondents who responded / number of respondents who entered survey

attrition_data <- attritevis::attrition(test_data)
head(attrition_data, 20)

   attrited prop_q proportion   questions responded prop_r
1         0   0.00       0.00           X       624   1.00
2         0   0.00       0.00     consent       624   1.00
3         3   0.00       0.00         age       621   1.00
4         0   0.00       0.00         sex       618   0.99
5         0   0.00       0.00   education       621   1.00
6         0   0.00       0.00       state       621   1.00
7         0   0.00       0.00      income       621   1.00
8         0   0.00       0.00     part_id       621   1.00
9         0   0.00       0.00        race       621   1.00
10        0   0.00       0.00    religion       621   1.00
11        1   0.00       0.00 attrition_1       620   0.99
12        6   0.01       0.01 attrition_2       614   0.98
13       37   0.06       0.06     cards_a       577   0.92
14        0   0.00       0.00          pa       553   0.89
15        0   0.00       0.00        pb_1       536   0.86
16        0   0.00       0.00        pb_2       536   0.86
17        0   0.00       0.00        pb_3       536   0.86
18        0   0.00       0.00          pc       534   0.86
19        0   0.00       0.00     cards_b       534   0.86
20        0   0.00       0.00         p2a       522   0.84

sum(attrition_data$attrited) #How many respondents attrited overall?

[1] 129

sum(attrition_data$attrited)/nrow(test_data) #What proportion of the overall sample is this?

[1] 0.2067308

• The plot_attrition() function shows the attrition process for the post-treatment questions

plot_attrition(test_data)  

• We can specify y="resonded" to account for response, rather than attrition

plot_attrition(test_data, y="responded")  

• With treatment_q, we can specify the treatment question, and with outcome_q, we can specify the outcome questions

attritevis::plot_attrition(test_data,
              y = "responded",
              outcome_q = c("cards1", "cards2",  "cards3"),
              treatment_q = "treat1",
              mycolors = c(treatment = "#000066",
                           control = "#CC0033"))

• The balance_cov() function calculates the balance of covariates between the treatment and control groups

attritevis::balance_cov(data = test_data, 
        treatment = "treat1", 
        question = "age")

    Welch Two Sample t-test

data:  treat_data$question1 and control_data$question1
t = -0.32688, df = 568.57, p-value = 0.7439
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -2.002600  1.431137
sample estimates:
mean of x mean of y 
 37.42361  37.70934 

• We can also use the function balance_cov() when the covariate (question) is a factor, but we must specify which factor we are interested in

attritevis::balance_cov(data = test_data, 
        treatment = "treat1", 
        question = "sex",
        factor = TRUE,
        factor_name = "female")

    2-sample test for equality of proportions with continuity correction

data:  x out of n
X-squared = 1.1305, df = 1, p-value = 0.2877
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.12931498  0.03623038
sample estimates:
   prop 1    prop 2 
0.3576389 0.4041812 

• Next, we can check whether our treatment is correlated with attrition at any moment in the survey with the balance_attrite() function

attritevis::balance_attrite(data = test_data, 
        treatment = "treat1", 
        question = "Happy_3_1")

Call:
glm(formula = question1 ~ treatment1, family = binomial(link = "logit"), 
    data = data2)

Coefficients:
                    Estimate Std. Error z value Pr(>|z|)    
(Intercept)          -1.7441     0.1653 -10.552   <2e-16 ***
treatment1treatment  -0.1704     0.2415  -0.706     0.48    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 464.49  on 576  degrees of freedom
Residual deviance: 463.99  on 575  degrees of freedom
  (47 observations deleted due to missingness)
AIC: 467.99

Number of Fisher Scoring iterations: 4

• Finally, we can estimate both Manski bounds and Lee bounds with the bounds() function
• We have to install the attrition package to estimate the bounds

# devtools::install_github("acoppock/attrition")
attritevis::bounds(data = test_data, 
        treatment = "treat1", 
        DV = "cards1")

    ci_lower     ci_upper      low_est      upp_est      low_var      upp_var 
-0.003727682  0.144394733  0.070333526  0.070333526  0.001427859  0.001427859 

attritevis::bounds(data = test_data, 
        treatment = "treat1", 
        DV = "cards1", 
        type = "Lee")

    upper_bound     lower_bound       Out0_mono      Out1L_mono      Out1U_mono 
     0.07968127      0.06517928      0.29482072      0.36000000      0.37450199 
control_group_N   treat_group_N               Q              f1              f0 
   251.00000000    254.00000000      0.01523036      0.11805556      0.13148789 
         pi_r_1          pi_r_0 
     0.88194444      0.86851211 

• Attrition is a common issue in experiments, and it can bias estimates (as everything else we’ve seen so far 😅)
• We can handle attrition by assuming that it is related to observed variables
• In this case, we can use inverse probability weighting (IPW) to estimate the ATE
• We can also use extreme bounds analysis (EBA) to evaluate the sensitivity of the ATE to different assumptions about the missing data
• Lee bounds are a way to trim the bounds to make them more informative
• The attritevis package in R has several functions to visualise the attrition process, calculate the bounds, and estimate the ATE
• Consider using these functions in your next experiment! 🤓