• Potential outcomes framework help understand treatment effects
• Fundamental problem of causal inference: we only see one potential outcome
• Treatment effect is the difference between potential outcomes (\(\tau_{i} = Y_{1,i} - Y_{0,i}\))
• Average causal effect is the goal, but simple means comparison is usually biased
• Selection bias: treated/untreated groups may differ even before treatment
• Bias does not vanish in large samples
• Mathematical expectation \(E[Y]\) is the population average
• Law of large numbers: sample average converges to population average
• Conditional expectation \(E[Y_i|X_i]\) is average outcome given \(X_i\)

• SUTVA (Stable Unit Treatment Value Assumption)
• Violations of SUTVA: spillovers, contamination
• External validity: generalising results to other settings
• Heterogeneous treatment effects: effects vary across units
• Compliance problem: people may not follow random assignment
• Intent-to-Treat (ITT) analyses address compliance

• A little more about potential outcomes
• Measuring variability
• Different types of selection bias, e.g.:
  • Inappropriate controls
  • Loss to follow-up
  • Attrition bias
  • Survivorship bias
  • Post-treatment bias
• How to use R and DeclareDesign to estimate regression models
• But first…

• Selection bias occurs when the groups are systematically different before treatment
• Using an example from Angrist and Pischke (2021), let’s say we have a student called Khuzdar from Kazakhstan, who is considering studying in the US and is worried about the cold weather
• Should he get health insurance? 🤔
• Let’s imagine that, without insurance, Khuzdar has a potential outcome of \(Y_{0,i} = 3\) and, with insurance, \(Y_{1,i} = 4\). So the treatment effect is \(\tau_i = 1\), that is, he gains 1 “health point” by getting insurance

• Now, let’s imagine that Khuzdar has a Chilean colleague called Maria Moreno, who is also considering studying in the US
• But since she comes from chilly Santiago, she is not worried about the cold weather
• So, without insurance, Maria has a potential outcome of \(Y_{0,i} = 5\) and, with insurance, \(Y_{1,i} = 5\). So the treatment effect is \(\tau_i = 0\), that is, she gains no “health points” by getting insurance

• We use balance tests or (randomisation checks) to check if the groups are similar before treatment
• The idea is to compare the means of the covariates for the treated and untreated groups
• If the means are similar, it provides evidence that nothing systematic is driving the treatment effect
• We are never 100% sure, but we trust the random assignment
• Small differences in means are acceptable, as long as they are not systematic
• Why? Because some variation can happen only by chance
• 100% personal opinion: I think they aren’t always useful… (can you guess why? 🤔)
• Mutz et al (2018) make a good case for that

• Causal diagrams are a great way to visualise the relationships between variables
  • They are also called directed acyclic graphs (DAGs)
  • They are “directed” because they show the direction of the relationships, and “acyclic” because they do not have loops (one cause cannot cause itself)
• They have been introduced by Judea Pearl and are widely used in causal inference (quick guide here)
• They help us understand the associations between variables and identify potential sources of bias
• And they are super easy to draw! 🤓
• For instance, let’s illustrate a classic example of selection bias with a causal diagram (borrowed from Hernán et al., 2019)

• A common problem in experiments is individuals dropping out before the study ends
• This is an issue because the treatment effect may be different for those who leave
• Example: A study on antiretroviral therapy for HIV-infected patients. Let’s define the variables:
  • E (Exposure): The antiretroviral therapy
  • D (Disease): The outcome, developing AIDS
  • L (Measured): Presence of other HIV symptoms (e.g., fever)
  • U (Unmeasured): True level of immunosuppression
  • C (Censoring): The patient drops out of the study (\(C=1\))
• Patients with greater immunosuppression (U) are more likely to have symptoms (L) and are also more likely to drop out (C)

• Understanding the Bias:
  • In the diagram, both L (symptoms) and U (immunosuppression) cause patients to drop out (C). A variable that has two arrows pointing into it is called a collider
  • When we analyse only the people who complete the study, we are “conditioning on the collider”
  • This action distorts the estimated relationship between the therapy (E) and the outcome (D), leading to selection bias
• How can we correct for this?
  • One way is to use stratified analysis

• Stratification is quite simple:
  • 1. Divide: Assuming L is measured, we split participants into groups (strata) based on their symptoms (e.g., mild, moderate, severe)
  • 2. Estimate: We calculate the treatment effect separately within each stratum
  • 3. Average: We then average the stratum-specific estimates to get the overall treatment effect
• By doing this, we control for the effect of L
• Important caveat: This method only corrects for bias from measured variables like L. It cannot fix bias from unmeasured variables like U

• During World War II, the US military wanted to improve the armour on its aircraft
• They analysed the bullet holes on returning aircraft and decided to add more armour to the areas with the most bullet holes
• Considering the picture you see here, where would you add the armour?
• More here
• Another example: Imagine you are studying mutual fund performance, and you look at funds that are still in operation. You see that they beat the S&P 500 by 2% per year
• What’s the problem here?

Avoidable post-treatment bias

• Causal effect of party ID on the vote
  • Do control for race
  • Do not control for voting intentions five minutes before voting
• Causal effect of race on salary in a firm
  • Do control for qualifications
  • Don’t control for position in the firm
• Causal effect of medicine on health
  • Do control for health prior to the treatment decision
  • Do not control for side effects or other medicines taken later

Unavoidable post-treatment bias

• Causal effect of democratisation on civil war; do we control for GDP?
  • Yes, since GDP → democratisation we must control to avoid omitted variable bias
  • No, since democratisation → GDP, we would have post-treatment bias
• Causal effect of legislative effectiveness on state failure; do we control for trade openness?
  • Yes, since trade openness → legislative effectiveness, we must control to avoid omitted variable bias
  • No, since legislative effectiveness → trade openness, we would have post-treatment bias

• Apart from averages, we are also interested in variability: how spread out is our data?
• The foundational measure is variance, the average squared deviation from the mean
• The sample variance of \(Y_i\) in a sample of size \(n\) is defined as:
  • \(S(Y_i)^2 = \frac{1}{n-1} \sum_{i=1}^{n} (Y_i - \bar{Y})^2\)
• Since variance is in squared units, we use its square root, the standard deviation (SD)
• The SD is a descriptive measure: it summarises the variability within your dataset

• Example: Suppose you measure the IQ of 50 students
  • You find the average IQ is \(\bar{IQ} = 100\)
  • You calculate the \(SD = 15\)
• What does this mean?
  • It describes the spread of IQs in your sample
  • “A typical student in your sample has an IQ that is about 15 points away from the sample average of 100”
  • It answers the question: “How much do the individual data points differ from each other?”

• The standard error (SE) is different: it is the standard deviation of a statistic (like the sample mean)
• It measures how precise your sample mean is as an estimate of the true population mean
• The standard error of the sample mean is:
  • \(SE(\bar{Y}) = \frac{\sigma(Y_i)}{\sqrt{n}}\)
• Since we rarely know the population SD (\(\sigma\)), we use the sample SD (\(S\)) instead:
  • \(SE(\bar{Y}) = \frac{S(Y_i)}{\sqrt{n}}\)
• The SE is about precision and is used for inference (e.g., confidence intervals), while the SD is about the variability of the data

• Example: For the same 50 students with \(\bar{IQ} = 100\) and \(SD = 15\)
  • The Standard Error (SE) is:
  • \(SE = \frac{15}{\sqrt{50}} \approx 2.12\)
• What does this mean?
  • It describes the precision of your sample average
  • “If you repeated the study with another 50 students, you would expect the new sample mean to be roughly 2.12 points away from the true population mean”
  • It answers the question: “How much would my sample average likely change if I took a new sample?”

The concept: Precision of the treatment effect

• In an experiment, we don’t just care about one mean; we care about the difference in means (\(\bar{Y}_{1} - \bar{Y}_{0}\))
• Just as each sample mean has its own precision (its SE), the difference between them also has a measure of precision
• The standard error of the difference tells us how much we’d expect our estimated treatment effect to vary if we repeated the experiment
• Analogy: Imagine trying to measure the height difference between two platforms, but both are a bit wobbly. The uncertainty in your final measurement comes from the wobbliness of both platforms
• The SE of the difference combines the uncertainty from both the treatment and control group means

The formula & its use

• The uncertainty (variance) of a difference between two independent groups is the sum of their individual variances:
  • \(Var(\bar{Y}_{1} - \bar{Y}_{0}) = Var(\bar{Y}_{1}) + Var(\bar{Y}_{0})\)
• This leads to the formula for the standard error of the difference, which combines the variability (\(S^2\)) and sample size (\(n\)) from both groups:
  • \(SE(\bar{Y}_{1} - \bar{Y}_{0})=\sqrt{\frac{S(Y_{1})^2}{n_{1}}+\frac{S(Y_{0})^2}{n_{0}}}\)
• Why it’s crucial: This value is the foundation for hypothesis testing! It’s the denominator in a t-test:
  • \(t = \frac{\text{Difference in means}}{\text{SE of the difference}}\), or \(t = \frac{\bar{Y}_{1} - \bar{Y}_{0}}{SE(\bar{Y}_{1} - \bar{Y}_{0})}\)
• It also determines the width of the confidence interval around our treatment effect
• While your software computes this for you 🤖, knowing what’s “under the hood” helps you interpret your results! 😉

• Imagine that we want to estimate the effect of a job training programme on the number of interviews that unemployed individuals get, similar to the example we discussed earlier
• The treatment variable is treat, which takes the value 1 if the individual received the training and 0 otherwise
• We will assign 1000 individuals to the treatment and control groups, with an equal probability of being assigned to each group (simple random assignment)
• The outcome variable is interviews, which is the number of interviews the individual got after the training
• The treatment effect is 5, that is, individuals who received the training got 5 more interviews than those who did not

• Now that we have the data, we can estimate the treatment effect using the lm_robust() function from the estimatr package
• The function is similar to the lm() function, but it uses robust standard errors, which are more appropriate for experiments
• Why? Because robust standard errors correct for heteroskedasticity and clustering
• This is a free lunch: if your data is not heteroskedastic or clustered, the robust standard errors will be the same as the regular standard errors
• The function takes a formula as an argument, which is the outcome variable regressed on the treatment variable

• The output of the summary() function gives us the coefficient estimate of the treatment variable
• The intercept is 10.008, which is the average number of interviews for the control group
• The coefficient of the treatment variable is 4.958, which is the average treatment effect
• The standard error of the treatment effect is 0.129, which is the standard error of the difference in means
• Our results are statistically significant at the 5% level, as the \(p\)-value is less than 0.05

model <- lm_robust(interviews ~ treat, data = data)
summary(model)

Call:
lm_robust(formula = interviews ~ treat, data = data)

Standard error type:  HC2 

Coefficients:
            Estimate Std. Error t value   Pr(>|t|) CI Lower CI Upper  DF
(Intercept)   10.008    0.08986  111.37  0.000e+00    9.832   10.184 998
treat          4.958    0.12922   38.37 1.353e-198    4.704    5.211 998

Multiple R-squared:  0.5961 ,   Adjusted R-squared:  0.5957 
F-statistic:  1472 on 1 and 998 DF,  p-value: < 2.2e-16

• The R^2 and its adjusted version are measures of the goodness of fit of the model
  • They are the proportion of the variance in the outcome variable explained by the model
• The F statistic is a test of the overall significance of the model
  • The null hypothesis is that all coefficients are zero
  • But as we see, the treatment variable is significant

• We can also include covariates in the model to increase precision
• In contrast with the treatment, the covariates have no causal interpretation, that is, they are not affected by the treatment
• Even with randomisation, there can be small imbalances in covariates between treatment and control groups, especially in small samples
• Including these covariates in the regression helps to adjust for any residual imbalances that may exist due to random chance, making the analysis more accurate
• Including covariates can also help bolster the credibility of the results by demonstrating that the treatment effect persists even after controlling for relevant variables
• But remember to include only pre-treatment covariates in the model, as post-treatment covariates can introduce bias

• Let’s simulate a new dataset with two pre-treatment covariates: age and education

set.seed(385)

data2 <- fabricate(
  N = 1000,
  treat = complete_ra(N, m = 500),
  age = round(rnorm(N, mean = 30, sd = 5)),
  education = round(rnorm(N, mean = 12, sd = 2)),
  interviews = round(rnorm(N, mean = 10, sd = 2) + 5 * treat)
)

head(data2)

    ID treat age education interviews
1 0001     0  32        11          9
2 0002     1  30        13         16
3 0003     1  31        12         16
4 0004     1  33        13         15
5 0005     0  27        15          7
6 0006     0  33        12          9

• The lm_lin() function is similar to the lm_robust() function, but it includes the covariates argument
• We pass a simple R formula to it
• As you can see, the results are pretty similar to the previous model
• The treatment effect is slightly higher, and the standard error is slightly lower
• The _c part of the variable names indicates that the variables are centred, as recommended by Lin (2013)

• We can also estimate the treatment effect for sub-groups of the population
• This is useful when we suspect that the treatment effect may vary across known dimensions
• For instance, we can estimate the treatment effect for people with high and low levels of education
• We can do this by including an interaction term between the treatment variable and the covariate of interest
• We will discuss this in further detail in the next lectures, but here is how to do it

• The output of the summary() function gives us the coefficient estimate of the treatment variable for the high education group
  • It is in the last line, treat:high_edu
• Interpretation: The treatment effect for the high-education subgroup (high_edu = 1) is 0.097 larger than for the low-education subgroup
• Total treatment effect for high_edu = 1: 5.101 + 0.097 = 5.198
• Significance: The interaction is not significant (CI: -0.395 to 0.590), meaning there’s no evidence the treatment effect differs between education subgroups
• This is expected, as we simulated the data to have the same treatment effect for all individuals (5)
• But it is good to know how to do it, right? 😉

• We discussed the potential outcomes framework in further detail, focusing on issues of selection bias
• We talked about different types of selection bias, such as inappropriate controls, loss to follow-up, attrition bias, survivorship bias, and counfounders
• We also discussed how to measure variability using the standard deviation and standard error
• Then, we talked about regression analysis of experiments, including how to simulate data, randomise treatment assignment, estimate the treatment effect, and include pre-treatment covariates in the model
• We also introduced the fabricatr, estimatr, and randomizr packages, which are part of the DeclareDesign suite
• Finally, we discussed sub-group analysis and how to estimate the treatment effect for different sub-groups of the population
• I hope you enjoyed the lecture! 🤓
• Next time, we will discuss more about hypothesis testing and confidence intervals, so stay tuned! 😉