Schedule

Last time

Research basics, our class, and R

Today

Admin: Canvas exists. Updated class website.
Material: The Rubin causal model (not mine), Chapter 2 MHE.

Schedule

Last time

Research basics, our class, and R

Today

Admin: Canvas exists. Updated class website.
Material: The Rubin causal model (not mine), Chapter 2 MHE.
Assignment1 Install R and RStudio on your computer.

Schedule

Last time

Research basics, our class, and R

Today

Admin: Canvas exists. Updated class website.
Material: The Rubin causal model (not mine), Chapter 2 MHE.
Assignment1 Install R and RStudio on your computer.
Assignment2 Take 15 minutes to quietly think about your interests.

Schedule

Last time

Research basics, our class, and R

Today

Admin: Canvas exists. Updated class website.
Material: The Rubin causal model (not mine), Chapter 2 MHE.
Assignment1 Install R and RStudio on your computer.
Assignment2 Take 15 minutes to quietly think about your interests.
Assignment3 First step of project proposal due April 15th.

Schedule

Last time

Research basics, our class, and R

Today

Admin: Canvas exists. Updated class website.
Material: The Rubin causal model (not mine), Chapter 2 MHE.
Assignment1 Install R and RStudio on your computer.
Assignment2 Take 15 minutes to quietly think about your interests.
Assignment3 First step of project proposal due April 15th.

Future

Lab: Matrix work, regression, functions, simulation
Long run: Deepen understandings/intuitions for causality and inference.

Review

Research fundamentals

Angrist and Pischke provide four fundamental questions for research:

1. What is the causal relationship of interest?

2. How would an ideal experiment capture this causal effect of interest?

3. What is your identification strategy?

4. What is your mode of inference?

Review

Research fundamentals

Angrist and Pischke provide four fundamental questions for research:

1. What is the causal relationship of interest?

2. How would an ideal experiment capture this causal effect of interest?

3. What is your identification strategy?

4. What is your mode of inference?

Seemingly straightforward questions can be fundamentally unanswerable.

Review

General research recommendations

More unsolicited advice:

• Be curious.

• Ask questions.

• Attend seminars.

• Meet faculty (UO + visitors).

• Focus on learning—especially intuition.^†

• Be kind and constructive.

The experimental ideal

What's so great about experiments?

Science widely regards experiments as the gold standard for research.

But why? The costs can be substantial.

Costs

• slow and expensive
• heavily regulated by review boards
• can abstract away from the actual question/setting

Benefits

So the benefits need to be pretty large, right?

The experimental ideal

Example: Hospitals and health

Imagine we want to know the causal effect of hospitals on health.

The experimental ideal

Example: Hospitals and health

Imagine we want to know the causal effect of hospitals on health.

Research question

Within the population of poor, elderly individuals, does visiting the emergency room for primary care improve health?

The experimental ideal

Example: Hospitals and health

Imagine we want to know the causal effect of hospitals on health.

Research question

Within the population of poor, elderly individuals, does visiting the emergency room for primary care improve health?

Empirical exercise

1. Collect data on health status and hospital visits.
2. Summarize health status by hospital-visit group.

The experimental ideal

Example: Hospitals and health

Our empirical exercise from the 2005 National Health Inteview Survey:

The experimental ideal

Example: Hospitals and health

Our empirical exercise from the 2005 National Health Inteview Survey:

We get a $t$ statistic of 58.9 when testing a difference in groups' means (0.72).

The experimental ideal

Example: Hospitals and health

Our empirical exercise from the 2005 National Health Inteview Survey:

We get a $t$ statistic of 58.9 when testing a difference in groups' means (0.72).

Conclusion? Hospitals make folks worse. Hospitals make sick people sicker.

The experimental ideal

Example: Hospitals and health

Our empirical exercise from the 2005 National Health Inteview Survey:

We get a $t$ statistic of 58.9 when testing a difference in groups' means (0.72).

Conclusion? Hospitals make folks worse. Hospitals make sick people sicker.

Alternative conclusion: Perhaps we're making a mistake in our analysis...

The experimental ideal

Example: Hospitals and health

Our empirical exercise from the 2005 National Health Inteview Survey:

We get a $t$ statistic of 58.9 when testing a difference in groups' means (0.72).

Conclusion? Hospitals make folks worse. Hospitals make sick people sicker.

Alternative conclusion: Perhaps we're making a mistake in our analysis...
maybe sick people go to hospitals?

The experimental ideal

Potential outcomes framework

Let's develop a framework to better discuss the problem here.

The experimental ideal

Potential outcomes framework

Let's develop a framework to better discuss the problem here.

• Binary treatment variable (e.g., hospitalized): $\text{D}_i = {0,1}$
• Outcome for individual $i$ (e.g., health): $\text{Y}_i$

This framework has a few names...

• Neyman potential outcomes framework
• Rubin causal model
• Neyman-Rubin "potential outcome"|"causal" "framework"|"model"

The experimental ideal

Potential outcomes framework

Research question: Does $\text{D}_i$ affect $\text{Y}_i$?

The experimental ideal

Potential outcomes framework

Research question: Does $\text{D}_i$ affect $\text{Y}_i$?

For each individual $i$, there are two potential outcomes (w/ binary $\text{D}_i$)

The experimental ideal

Potential outcomes framework

Research question: Does $\text{D}_i$ affect $\text{Y}_i$?

For each individual $i$, there are two potential outcomes (w/ binary $\text{D}_i$)

1. $\color{#e64173}{\text{Y}_{1i}}$ if $\color{#e64173}{\text{D}_i = 1}$
   $\color{#e64173}{i}$'s health outcome if she went to the hospital

The experimental ideal

Potential outcomes framework

Research question: Does $\text{D}_i$ affect $\text{Y}_i$?

For each individual $i$, there are two potential outcomes (w/ binary $\text{D}_i$)

1. $\color{#e64173}{\text{Y}_{1i}}$ if $\color{#e64173}{\text{D}_i = 1}$
   $\color{#e64173}{i}$'s health outcome if she went to the hospital

2. $\color{#6A5ACD}{\text{Y}_{0i}}$ if $\color{#6A5ACD}{\text{D}_i = 0}$
   $\color{#6A5ACD}{i}$'s health outcome if she did not go to the hospital

The experimental ideal

Potential outcomes framework

Research question: Does $\text{D}_i$ affect $\text{Y}_i$?

For each individual $i$, there are two potential outcomes (w/ binary $\text{D}_i$)

1. $\color{#e64173}{\text{Y}_{1i}}$ if $\color{#e64173}{\text{D}_i = 1}$
   $\color{#e64173}{i}$'s health outcome if she went to the hospital

2. $\color{#6A5ACD}{\text{Y}_{0i}}$ if $\color{#6A5ACD}{\text{D}_i = 0}$
   $\color{#6A5ACD}{i}$'s health outcome if she did not go to the hospital

The difference between these two outcomes gives us the causal effect of hospital treatment, i.e.,

$$\begin{align} \color{#FFA500}{\tau_i} = \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \end{align}$$

The experimental ideal

#problems

This simple equation $$\begin{align} \color{#FFA500}{\tau_i} = \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \end{align}$$ leads us to the fundamental problem of causal inference.

The experimental ideal

#problems

This simple equation $$\begin{align} \color{#FFA500}{\tau_i} = \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \end{align}$$ leads us to the fundamental problem of causal inference.

We can never simultaneously observe $\color{#e64173}{\text{Y}_{1i}}$ and $\color{#6A5ACD}{\text{Y}_{0i}}$.

The experimental ideal

#problems

This simple equation $$\begin{align} \color{#FFA500}{\tau_i} = \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \end{align}$$ leads us to the fundamental problem of causal inference.

We can never simultaneously observe $\color{#e64173}{\text{Y}_{1i}}$ and $\color{#6A5ACD}{\text{Y}_{0i}}$.

Most of applied econometrics focuses on addressing this simple problem.

The experimental ideal

#problems

This simple equation $$\begin{align} \color{#FFA500}{\tau_i} = \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \end{align}$$ leads us to the fundamental problem of causal inference.

We can never simultaneously observe $\color{#e64173}{\text{Y}_{1i}}$ and $\color{#6A5ACD}{\text{Y}_{0i}}$.

Most of applied econometrics focuses on addressing this simple problem.

Accordingly, our methods try to address the related question

For each $\color{#e64173}{\text{Y}_{1i}}$, what is a (reasonably) good counterfactual?

The experimental ideal

Solutions?

Problem We cannot directly calculate $\color{#FFA500}{\tau_i} = \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}}$.

The experimental ideal

Solutions?

Problem We cannot directly calculate $\color{#FFA500}{\tau_i} = \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}}$.

Proposed solution
Compare outcomes for people who visited the hospital $\left( \color{#e64173}{\text{Y}_{1i}\mid \color{#e64173}{\text{D}_{i}=1}} \right)$
to outcomes for people who did not visit the hospital $\left( \color{#6A5ACD}{\text{Y}_{0j}\mid \color{#6A5ACD}{\text{D}_{j}=0}} \right)$.

The experimental ideal

Solutions?

Problem We cannot directly calculate $\color{#FFA500}{\tau_i} = \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}}$.

Proposed solution
Compare outcomes for people who visited the hospital $\left( \color{#e64173}{\text{Y}_{1i}\mid \color{#e64173}{\text{D}_{i}=1}} \right)$
to outcomes for people who did not visit the hospital $\left( \color{#6A5ACD}{\text{Y}_{0j}\mid \color{#6A5ACD}{\text{D}_{j}=0}} \right)$.

$$\begin{align} \mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right] \end{align}$$ which gives us the observed difference in health outcomes.

The experimental ideal

Solutions?

Problem We cannot directly calculate $\color{#FFA500}{\tau_i} = \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}}$.

Proposed solution
Compare outcomes for people who visited the hospital $\left( \color{#e64173}{\text{Y}_{1i}\mid \color{#e64173}{\text{D}_{i}=1}} \right)$
to outcomes for people who did not visit the hospital $\left( \color{#6A5ACD}{\text{Y}_{0j}\mid \color{#6A5ACD}{\text{D}_{j}=0}} \right)$.

$$\begin{align} \mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right] \end{align}$$ which gives us the observed difference in health outcomes.

Q This comparison will return an answer, but is it the answer we want?

The experimental ideal

Selection

Q What does $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$ actually tell us?

The experimental ideal

Selection

Q What does $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$ actually tell us?

A First notice that we can write $i$'s outcome $\text{Y}_{i}$ as $$\begin{align} \text{Y}_{i} &= \color{#6A5ACD}{\text{Y}_{0i}} + \text{D}_{i} \underbrace{\left( \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \right)}_\color{#FFA500}{\tau_i} \end{align}$$

The experimental ideal

Selection

Q What does $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$ actually tell us?

A First notice that we can write $i$'s outcome $\text{Y}_{i}$ as $$\begin{align} \text{Y}_{i} &= \color{#6A5ACD}{\text{Y}_{0i}} + \text{D}_{i} \underbrace{\left( \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \right)}_\color{#FFA500}{\tau_i} \end{align}$$

Now write out our expectation, apply this definition, do creative math.

$\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

The experimental ideal

Selection

Q What does $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$ actually tell us?

A First notice that we can write $i$'s outcome $\text{Y}_{i}$ as $$\begin{align} \text{Y}_{i} &= \color{#6A5ACD}{\text{Y}_{0i}} + \text{D}_{i} \underbrace{\left( \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \right)}_\color{#FFA500}{\tau_i} \end{align}$$

Now write out our expectation, apply this definition, do creative math.

$\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

The experimental ideal

Selection

Q What does $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$ actually tell us?

A First notice that we can write $i$'s outcome $\text{Y}_{i}$ as $$\begin{align} \text{Y}_{i} &= \color{#6A5ACD}{\text{Y}_{0i}} + \text{D}_{i} \underbrace{\left( \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \right)}_\color{#FFA500}{\tau_i} \end{align}$$

Now write out our expectation, apply this definition, do creative math.

$\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] + \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

The experimental ideal

Selection

Q What does $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$ actually tell us?

A First notice that we can write $i$'s outcome $\text{Y}_{i}$ as $$\begin{align} \text{Y}_{i} &= \color{#6A5ACD}{\text{Y}_{0i}} + \text{D}_{i} \underbrace{\left( \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \right)}_\color{#FFA500}{\tau_i} \end{align}$$

Now write out our expectation, apply this definition, do creative math.

$\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \underbrace{\mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]}_\text{Average treatment effect on the treated 😀} + \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

The experimental ideal

Selection

Q What does $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$ actually tell us?

A First notice that we can write $i$'s outcome $\text{Y}_{i}$ as $$\begin{align} \text{Y}_{i} &= \color{#6A5ACD}{\text{Y}_{0i}} + \text{D}_{i} \underbrace{\left( \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \right)}_\color{#FFA500}{\tau_i} \end{align}$$

Now write out our expectation, apply this definition, do creative math.

$\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \underbrace{\mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]}_\text{Average treatment effect on the treated 😀} + \underbrace{\mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]}_\text{Selection bias 😞}$

The experimental ideal

Selection

The first term is good variation—essentially the answer that we want.
  $\mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$

The experimental ideal

Selection

The first term is good variation—essentially the answer that we want.
  $\mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$
   $=\mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \mid \color{#e64173}{\text{D}_{i}=1} \right]$

The experimental ideal

Selection

The first term is good variation—essentially the answer that we want.
  $\mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$
   $=\mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \mid \color{#e64173}{\text{D}_{i}=1} \right]$
   $=\mathop{E}\left[ \color{#FFA500}{\tau_i} \mid \color{#e64173}{\text{D}_{i}=1} \right]$

The experimental ideal

Selection

The first term is good variation—essentially the answer that we want.
  $\mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$
   $=\mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \mid \color{#e64173}{\text{D}_{i}=1} \right]$
   $=\mathop{E}\left[ \color{#FFA500}{\tau_i} \mid \color{#e64173}{\text{D}_{i}=1} \right]$
The average causal effect of hospitalization for hospitalized individuals.

The experimental ideal

Selection

The first term is good variation—essentially the answer that we want.
  $\mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$
   $=\mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \mid \color{#e64173}{\text{D}_{i}=1} \right]$
   $=\mathop{E}\left[ \color{#FFA500}{\tau_i} \mid \color{#e64173}{\text{D}_{i}=1} \right]$
The average causal effect of hospitalization for hospitalized individuals.

The second term is bad variation—preventing us from knowing the answer.
  $\mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

The experimental ideal

Selection

The first term is good variation—essentially the answer that we want.
  $\mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$
   $=\mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \mid \color{#e64173}{\text{D}_{i}=1} \right]$
   $=\mathop{E}\left[ \color{#FFA500}{\tau_i} \mid \color{#e64173}{\text{D}_{i}=1} \right]$
The average causal effect of hospitalization for hospitalized individuals.

The second term is bad variation—preventing us from knowing the answer.
  $\mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
The difference in the average untreated outcome between the treatment and control groups.

The experimental ideal

Selection

The first term is good variation—essentially the answer that we want.
  $\mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$
   $=\mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \mid \color{#e64173}{\text{D}_{i}=1} \right]$
   $=\mathop{E}\left[ \color{#FFA500}{\tau_i} \mid \color{#e64173}{\text{D}_{i}=1} \right]$
The average causal effect of hospitalization for hospitalized individuals.

The second term is bad variation—preventing us from knowing the answer.
  $\mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
The difference in the average untreated outcome between the treatment and control groups.

Selection bias The extent to which the "control group" provides a bad counterfactual for the treated individuals.

The experimental ideal

Selection

Angrist and Pischke (MHE, p. 15),

The goal of most empirical economic research is to overcome selection bias, and therefore to say something about the causal effect of a variable like $\text{D}_{i}$.

The experimental ideal

Selection

Angrist and Pischke (MHE, p. 15),

The goal of most empirical economic research is to overcome selection bias, and therefore to say something about the causal effect of a variable like $\text{D}_{i}$.

Q So how do experiments—the gold standard of empirical economic (and scientific) research—accomplish this goal and overcome selection bias?

The experimental ideal

Back to experiments

Q How do experiments overcome selection bias?

The experimental ideal

Back to experiments

Q How do experiments overcome selection bias?
A Experiments break the link between potential outcomes and treatment.

In other words: Randomly assigning $\text{D}_{i}$ makes $\text{D}_{i}$ independent of which outcome we observe (meaning $\color{#e64173}{\text{Y}_{1i}}$ or $\color{#6A5ACD}{\text{Y}_{0i}}$).

The experimental ideal

Back to experiments

Q How do experiments overcome selection bias?
A Experiments break the link between potential outcomes and treatment.

In other words: Randomly assigning $\text{D}_{i}$ makes $\text{D}_{i}$ independent of which outcome we observe (meaning $\color{#e64173}{\text{Y}_{1i}}$ or $\color{#6A5ACD}{\text{Y}_{0i}}$).

Difference in means with random assignment of $\text{D}_{i}$
$\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

The experimental ideal

Back to experiments

Q How do experiments overcome selection bias?
A Experiments break the link between potential outcomes and treatment.

In other words: Randomly assigning $\text{D}_{i}$ makes $\text{D}_{i}$ independent of which outcome we observe (meaning $\color{#e64173}{\text{Y}_{1i}}$ or $\color{#6A5ACD}{\text{Y}_{0i}}$).

Difference in means with random assignment of $\text{D}_{i}$
$\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

The experimental ideal

Back to experiments

Q How do experiments overcome selection bias?
A Experiments break the link between potential outcomes and treatment.

In other words: Randomly assigning $\text{D}_{i}$ makes $\text{D}_{i}$ independent of which outcome we observe (meaning $\color{#e64173}{\text{Y}_{1i}}$ or $\color{#6A5ACD}{\text{Y}_{0i}}$).

Difference in means with random assignment of $\text{D}_{i}$
$\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$

The experimental ideal

Back to experiments

Q How do experiments overcome selection bias?
A Experiments break the link between potential outcomes and treatment.

In other words: Randomly assigning $\text{D}_{i}$ makes $\text{D}_{i}$ independent of which outcome we observe (meaning $\color{#e64173}{\text{Y}_{1i}}$ or $\color{#6A5ACD}{\text{Y}_{0i}}$).

Difference in means with random assignment of $\text{D}_{i}$
$\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$  from random assignment of ${\text{D}_{i}}$

The experimental ideal

Back to experiments

Q How do experiments overcome selection bias?
A Experiments break the link between potential outcomes and treatment.

In other words: Randomly assigning $\text{D}_{i}$ makes $\text{D}_{i}$ independent of which outcome we observe (meaning $\color{#e64173}{\text{Y}_{1i}}$ or $\color{#6A5ACD}{\text{Y}_{0i}}$).

Difference in means with random assignment of $\text{D}_{i}$
$\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$  from random assignment of ${\text{D}_{i}}$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$

The experimental ideal

Back to experiments

Q How do experiments overcome selection bias?
A Experiments break the link between potential outcomes and treatment.

In other words: Randomly assigning $\text{D}_{i}$ makes $\text{D}_{i}$ independent of which outcome we observe (meaning $\color{#e64173}{\text{Y}_{1i}}$ or $\color{#6A5ACD}{\text{Y}_{0i}}$).

Difference in means with random assignment of $\text{D}_{i}$
$\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$  from random assignment of ${\text{D}_{i}}$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$
  $= \mathop{E}\left[ \color{#FFA500}{\tau_i}\mid \color{#e64173}{\text{D}_{i}=1} \right]$

The experimental ideal

Back to experiments

Q How do experiments overcome selection bias?
A Experiments break the link between potential outcomes and treatment.

In other words: Randomly assigning $\text{D}_{i}$ makes $\text{D}_{i}$ independent of which outcome we observe (meaning $\color{#e64173}{\text{Y}_{1i}}$ or $\color{#6A5ACD}{\text{Y}_{0i}}$).

Difference in means with random assignment of $\text{D}_{i}$
$\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$  from random assignment of ${\text{D}_{i}}$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$
  $= \mathop{E}\left[ \color{#FFA500}{\tau_i}\mid \color{#e64173}{\text{D}_{i}=1} \right]$
  $= \mathop{E}\left[ \color{#FFA500}{\tau_i} \right]$

The experimental ideal

Back to experiments

Q How do experiments overcome selection bias?
A Experiments break the link between potential outcomes and treatment.

In other words: Randomly assigning $\text{D}_{i}$ makes $\text{D}_{i}$ independent of which outcome we observe (meaning $\color{#e64173}{\text{Y}_{1i}}$ or $\color{#6A5ACD}{\text{Y}_{0i}}$).

Difference in means with random assignment of $\text{D}_{i}$
$\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}}\mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$  from random assignment of ${\text{D}_{i}}$
  $= \mathop{E}\left[ \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#e64173}{\text{D}_{i}=1} \right]$
  $= \mathop{E}\left[ \color{#FFA500}{\tau_i}\mid \color{#e64173}{\text{D}_{i}=1} \right]$
  $= \mathop{E}\left[ \color{#FFA500}{\tau_i} \right]$       Random assignment of $\text{D}_{i}$ breaks selection bias.

The experimental ideal

Randomly assigned treatment

The key to avoiding selection bias: random assignment of treatment

The experimental ideal

Randomly assigned treatment

The key to avoiding selection bias: random assignment of treatment
(or as-good-as random assignment, e.g., natural experiments).

The experimental ideal

Randomly assigned treatment

The key to avoiding selection bias: random assignment of treatment
(or as-good-as random assignment, e.g., natural experiments).

Random assignment of treatment gives us $$\begin{align} \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right] = \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}} \mid \color{#e64173}{\text{D}_{i}=1} \right] \end{align}$$ meaning the control group's mean now provides a good counterfactual for the treatment group's mean.

The experimental ideal

Randomly assigned treatment

The key to avoiding selection bias: random assignment of treatment
(or as-good-as random assignment, e.g., natural experiments).

Random assignment of treatment gives us $$\begin{align} \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}}\mid \color{#6A5ACD}{\text{D}_{i}=0} \right] = \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}} \mid \color{#e64173}{\text{D}_{i}=1} \right] \end{align}$$ meaning the control group's mean now provides a good counterfactual for the treatment group's mean.

In other words, there is no selection bias, i.e.,

The experimental ideal

Randomly assigned treatment

Additional benefit of randomization:

The average treatment effect is now representative of the population average, rather than the treatment-group average.

The experimental ideal

Randomly assigned treatment

Additional benefit of randomization:

The average treatment effect is now representative of the population average, rather than the treatment-group average.

$$\begin{align} \mathop{E}\left[ \color{#FFA500}{\tau_i} \mid \color{#e64173}{\text{D}_{i}=1} \right] = \mathop{E}\left[ \color{#FFA500}{\tau_i} \mid \color{#6A5ACD}{\text{D}_{i}=0} \right] = \mathop{E}\left[ \color{#FFA500}{\tau_i} \right] \end{align}$$

The experimental ideal

Example: Training programs

Governments subsidize training programs to assist disadvantaged workers.

The experimental ideal

Example: Training programs

Governments subsidize training programs to assist disadvantaged workers.

Q Do these programs have the desired effects (i.e., increase wages)?

The experimental ideal

Example: Training programs

Governments subsidize training programs to assist disadvantaged workers.

Q Do these programs have the desired effects (i.e., increase wages)?

A Observational studies—comparing wage data from participants and non-participants—often find that people who complete these programs actually make lower wages.

The experimental ideal

Example: Training programs

Governments subsidize training programs to assist disadvantaged workers.

Q Do these programs have the desired effects (i.e., increase wages)?

A Observational studies—comparing wage data from participants and non-participants—often find that people who complete these programs actually make lower wages.

Challenges Participants self select. + Programs target lower-wage workers.

The experimental ideal

Example: Training programs

How do we formalize these concerns in our framework?

The experimental ideal

Example: Training programs

How do we formalize these concerns in our framework?

Observational program evaluations
$\quad\mathop{E}\left[ \text{Wage}_{i}\mid \color{#e64173}{\text{Program}_{i}=1} \right] - \mathop{E}\left[ \text{Wage}_i \mid \color{#6A5ACD}{\text{Program}_{i}=0} \right]=$ $$\begin{align} &\underbrace{\mathop{E}\left[ \color{#e64173}{\text{Wage}_{1i}}\mid \color{#e64173}{\text{Program}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Wage}_{0i}}\mid \color{#e64173}{\text{Program}_{i}=1} \right]}_{\text{Average causal effect of training program on wages for participants, }\mathit{i.e.} \text{, } \overline{\tau}_{1}} \, + \\[0.5em] &\underbrace{\mathop{E}\left[ \color{#6A5ACD}{\text{Wage}_{0i}}\mid \color{#e64173}{\text{Program}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Wage}_{0i}}\mid \color{#6A5ACD}{\text{Program}_{i}=0} \right]}_\text{Selection bias} \end{align}$$

The experimental ideal

Example: Training programs

How do we formalize these concerns in our framework?

Observational program evaluations
$\quad\mathop{E}\left[ \text{Wage}_{i}\mid \color{#e64173}{\text{Program}_{i}=1} \right] - \mathop{E}\left[ \text{Wage}_i \mid \color{#6A5ACD}{\text{Program}_{i}=0} \right]=$ $$\begin{align} &\underbrace{\mathop{E}\left[ \color{#e64173}{\text{Wage}_{1i}}\mid \color{#e64173}{\text{Program}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Wage}_{0i}}\mid \color{#e64173}{\text{Program}_{i}=1} \right]}_{\text{Average causal effect of training program on wages for participants, }\mathit{i.e.} \text{, } \overline{\tau}_{1}} \, + \\[0.5em] &\underbrace{\mathop{E}\left[ \color{#6A5ACD}{\text{Wage}_{0i}}\mid \color{#e64173}{\text{Program}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Wage}_{0i}}\mid \color{#6A5ACD}{\text{Program}_{i}=0} \right]}_\text{Selection bias} \end{align}$$

If the program attracts/selects individuals who, on average, have lower wages without the program (sort of the point of the program), then we have negative selection bias.

The experimental ideal

Example: Training programs

$\quad\mathop{E}\left[ \text{Wage}_{i}\mid \color{#e64173}{\text{Program}_{i}=1} \right] - \mathop{E}\left[ \text{Wage}_i \mid \color{#6A5ACD}{\text{Program}_{i}=0} \right] =$ $$\begin{align} &\mathop{E}\left[ \color{#e64173}{\text{Wage}_{1i}}\mid \color{#e64173}{\text{Program}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Wage}_{0i}}\mid \color{#e64173}{\text{Program}_{i}=1} \right] \, + \\[0.5em] &\mathop{E}\left[ \color{#6A5ACD}{\text{Wage}_{0i}}\mid \color{#e64173}{\text{Program}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Wage}_{0i}}\mid \color{#6A5ACD}{\text{Program}_{i}=0} \right] \\[0.5em] \end{align}$$

So even if the program, on average, has an positive wage effect (in the participant group), i.e., $\overline{\tau}_1>0$, we will detect a lower effect due to the negative selection bias.

The experimental ideal

Example: Training programs

$\quad\mathop{E}\left[ \text{Wage}_{i}\mid \color{#e64173}{\text{Program}_{i}=1} \right] - \mathop{E}\left[ \text{Wage}_i \mid \color{#6A5ACD}{\text{Program}_{i}=0} \right] =$ $$\begin{align} &\mathop{E}\left[ \color{#e64173}{\text{Wage}_{1i}}\mid \color{#e64173}{\text{Program}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Wage}_{0i}}\mid \color{#e64173}{\text{Program}_{i}=1} \right] \, + \\[0.5em] &\mathop{E}\left[ \color{#6A5ACD}{\text{Wage}_{0i}}\mid \color{#e64173}{\text{Program}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Wage}_{0i}}\mid \color{#6A5ACD}{\text{Program}_{i}=0} \right] \\[0.5em] \end{align}$$

So even if the program, on average, has an positive wage effect (in the participant group), i.e., $\overline{\tau}_1>0$, we will detect a lower effect due to the negative selection bias.

If the bias is sufficiently large (relative to the treatment effect), our estimate will even get the sign of the effect wrong.

The experimental ideal

Example: Training programs

$\quad\mathop{E}\left[ \text{Wage}_{i}\mid \color{#e64173}{\text{Program}_{i}=1} \right] - \mathop{E}\left[ \text{Wage}_i \mid \color{#6A5ACD}{\text{Program}_{i}=0} \right] =$ $$\begin{align} &\mathop{E}\left[ \color{#e64173}{\text{Wage}_{1i}}\mid \color{#e64173}{\text{Program}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Wage}_{0i}}\mid \color{#e64173}{\text{Program}_{i}=1} \right] \, + \\[0.5em] &\mathop{E}\left[ \color{#6A5ACD}{\text{Wage}_{0i}}\mid \color{#e64173}{\text{Program}_{i}=1} \right] - \mathop{E}\left[ \color{#6A5ACD}{\text{Wage}_{0i}}\mid \color{#6A5ACD}{\text{Program}_{i}=0} \right] \\[0.5em] \end{align}$$

So even if the program, on average, has an positive wage effect (in the participant group), i.e., $\overline{\tau}_1>0$, we will detect a lower effect due to the negative selection bias.

If the bias is sufficiently large (relative to the treatment effect), our estimate will even get the sign of the effect wrong.

Related While observational studies typically found negative program effects, several experiments found positive program effects.

The experimental ideal

Example: The STAR experiment

The Tennessee STAR experiment is a famous/popular example of an experiment that allows us to answer an important social/policy question.

Research question Do classroom resources affect student performance?

The experimental ideal

Example: The STAR experiment

The Tennessee STAR experiment is a famous/popular example of an experiment that allows us to answer an important social/policy question.

Research question Do classroom resources affect student performance?

• Statewide(-ish) in Tennessee for the 1985–1986 kindergarten cohort
• Ran for 4 years with ∼11,600 children. Cost ∼$12 million.

The experimental ideal

Example: The STAR experiment

The Tennessee STAR experiment is a famous/popular example of an experiment that allows us to answer an important social/policy question.

Research question Do classroom resources affect student performance?

• Statewide(-ish) in Tennessee for the 1985–1986 kindergarten cohort
• Ran for 4 years with ∼11,600 children. Cost ∼$12 million.

Treatments

1. Small classes (13–17 students)
2. Regular classes (22–35 students) plus part-time teacher's aide
3. Regular classes (22–35 students) plus full-time teacher's aide

The experimental ideal

Example: The STAR experiment

First question Did the randomization balance participants' characteristics across the treatment groups?

The experimental ideal

Example: The STAR experiment

First question Did the randomization balance participants' characteristics across the treatment groups?

Ideally, we would have pre-experiment data on outcome variable.

Unfortunately, we only have a few demographic attributes.

The experimental ideal

The STAR experiment

The previous table estimated/compared the treatment effects using simple differences in means.

We can make the same comparisons using regressions.

Specifically, we regress our outcome (test percentile) on dummy variables (binary indicator variables) for each treatment group.

The experimental ideal

Example of our three treatment dummies.

$$\begin{matrix} \color{#c2bebe}{i} & \color{#c2bebe}{y_i} & \color{#c2bebe}{\text{Trt}_{1i}} & \color{#c2bebe}{\text{Trt}_{2i}} & \color{#c2bebe}{\text{Trt}_{3i}} \\ \color{#c2bebe}{1 } & y_1 & 1 & 0 & 0\\ \color{#c2bebe}{2 } & y_2 & 1 & 0 & 0\\ \color{#c2bebe}{\vdots } & \vdots & \vdots & \vdots & \vdots\\ \color{#c2bebe}{\ell } & y_\ell & 1 & 0 & 0\\ \color{#c2bebe}{\ell + 1} & y_{\ell-1} & 0 & 1 & 0\\ \color{#c2bebe}{\vdots } & \vdots & \vdots & \vdots & \vdots\\ \color{#c2bebe}{p } & y_p & 0 & 1 & 0\\ \color{#c2bebe}{p+1 } & y_{p+1} & 0 & 0 & 1\\ \color{#c2bebe}{\vdots } & \vdots & \vdots & \vdots & \vdots\\ \color{#c2bebe}{N } & y_N & 0 & 0 & 1 \end{matrix}$$

The experimental ideal

Regression analysis

Assume for the moment that the treatment effect is constant^†, i.e.,

†You'll often hear econometricians say "homogeneous" (vs. "hetergeneous").

$$\begin{align} \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} = \color{#FFA500}{\rho} \quad \forall i \end{align}$$

The experimental ideal

Regression analysis

Assume for the moment that the treatment effect is constant^†, i.e.,

†You'll often hear econometricians say "homogeneous" (vs. "hetergeneous").

$$\begin{align} \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} = \color{#FFA500}{\rho} \quad \forall i \end{align}$$ then we can rewrite $$\begin{align} \text{Y}_{i} = \color{#6A5ACD}{\text{Y}_{0i}} + \text{D}_{i}\left( \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \right) \end{align}$$

The experimental ideal

Regression analysis

Assume for the moment that the treatment effect is constant^†, i.e.,

†You'll often hear econometricians say "homogeneous" (vs. "hetergeneous").

$$\begin{align} \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} = \color{#FFA500}{\rho} \quad \forall i \end{align}$$ then we can rewrite $$\begin{align} \text{Y}_{i} = \color{#6A5ACD}{\text{Y}_{0i}} + \text{D}_{i}\left( \color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}} \right) \end{align}$$ as $$\begin{align} \text{Y}_{i} = \underbrace{\alpha}_{=\mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}} \right]} + \text{D}_{i} \underbrace{\color{#FFA500}{\rho}}_{\color{#e64173}{\text{Y}_{1i}} - \color{#6A5ACD}{\text{Y}_{0i}}} + \underbrace{\eta_i}_{\color{#6A5ACD}{\text{Y}_{0i}} - \mathop{E}\left[ \color{#6A5ACD}{\text{Y}_{0i}} \right]} \end{align}$$

The experimental ideal

Regression analysis

$$\begin{align} \text{Y}_{i} = \alpha + \text{D}_{i} \color{#FFA500}{\rho} + \eta_i \end{align}$$

Now write out the conditional expectation of $\text{Y}_{i}$ for both levels of $\text{D}_{i}$

The experimental ideal

Regression analysis

$$\begin{align} \text{Y}_{i} = \alpha + \text{D}_{i} \color{#FFA500}{\rho} + \eta_i \end{align}$$

Now write out the conditional expectation of $\text{Y}_{i}$ for both levels of $\text{D}_{i}$

  $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] =$

The experimental ideal

Regression analysis

$$\begin{align} \text{Y}_{i} = \alpha + \text{D}_{i} \color{#FFA500}{\rho} + \eta_i \end{align}$$

Now write out the conditional expectation of $\text{Y}_{i}$ for both levels of $\text{D}_{i}$

  $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] =$ $\mathop{E}\left[ \alpha + \color{#FFA500}{\rho} + \eta_i \mid \color{#e64173}{\text{D}_{i}=1} \right]$

The experimental ideal

Regression analysis

$$\begin{align} \text{Y}_{i} = \alpha + \text{D}_{i} \color{#FFA500}{\rho} + \eta_i \end{align}$$

Now write out the conditional expectation of $\text{Y}_{i}$ for both levels of $\text{D}_{i}$

  $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] =$ $\mathop{E}\left[ \alpha + \color{#FFA500}{\rho} + \eta_i \mid \color{#e64173}{\text{D}_{i}=1} \right]$ $=\alpha + \color{#FFA500}{\rho} + \mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right]$

The experimental ideal

Regression analysis

$$\begin{align} \text{Y}_{i} = \alpha + \text{D}_{i} \color{#FFA500}{\rho} + \eta_i \end{align}$$

Now write out the conditional expectation of $\text{Y}_{i}$ for both levels of $\text{D}_{i}$

  $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] =$ $\mathop{E}\left[ \alpha + \color{#FFA500}{\rho} + \eta_i \mid \color{#e64173}{\text{D}_{i}=1} \right]$ $=\alpha + \color{#FFA500}{\rho} + \mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right]$

  $\mathop{E}\left[ \text{Y}_{i} \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

The experimental ideal

Regression analysis

$$\begin{align} \text{Y}_{i} = \alpha + \text{D}_{i} \color{#FFA500}{\rho} + \eta_i \end{align}$$

Now write out the conditional expectation of $\text{Y}_{i}$ for both levels of $\text{D}_{i}$

  $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] =$ $\mathop{E}\left[ \alpha + \color{#FFA500}{\rho} + \eta_i \mid \color{#e64173}{\text{D}_{i}=1} \right]$ $=\alpha + \color{#FFA500}{\rho} + \mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right]$

  $\mathop{E}\left[ \text{Y}_{i} \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$ $= \mathop{E}\left[ \alpha + \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

The experimental ideal

Regression analysis

$$\begin{align} \text{Y}_{i} = \alpha + \text{D}_{i} \color{#FFA500}{\rho} + \eta_i \end{align}$$

Now write out the conditional expectation of $\text{Y}_{i}$ for both levels of $\text{D}_{i}$

  $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] =$ $\mathop{E}\left[ \alpha + \color{#FFA500}{\rho} + \eta_i \mid \color{#e64173}{\text{D}_{i}=1} \right]$ $=\alpha + \color{#FFA500}{\rho} + \mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right]$

  $\mathop{E}\left[ \text{Y}_{i} \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$ $= \mathop{E}\left[ \alpha + \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$ $= \alpha + \mathop{E}\left[ \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

The experimental ideal

Regression analysis

$$\begin{align} \text{Y}_{i} = \alpha + \text{D}_{i} \color{#FFA500}{\rho} + \eta_i \end{align}$$

Now write out the conditional expectation of $\text{Y}_{i}$ for both levels of $\text{D}_{i}$

  $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] =$ $\mathop{E}\left[ \alpha + \color{#FFA500}{\rho} + \eta_i \mid \color{#e64173}{\text{D}_{i}=1} \right]$ $=\alpha + \color{#FFA500}{\rho} + \mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right]$

  $\mathop{E}\left[ \text{Y}_{i} \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$ $= \mathop{E}\left[ \alpha + \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$ $= \alpha + \mathop{E}\left[ \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

Take the difference...

  $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i} \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

The experimental ideal

Regression analysis

$$\begin{align} \text{Y}_{i} = \alpha + \text{D}_{i} \color{#FFA500}{\rho} + \eta_i \end{align}$$

Now write out the conditional expectation of $\text{Y}_{i}$ for both levels of $\text{D}_{i}$

  $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] =$ $\mathop{E}\left[ \alpha + \color{#FFA500}{\rho} + \eta_i \mid \color{#e64173}{\text{D}_{i}=1} \right]$ $=\alpha + \color{#FFA500}{\rho} + \mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right]$

  $\mathop{E}\left[ \text{Y}_{i} \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$ $= \mathop{E}\left[ \alpha + \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$ $= \alpha + \mathop{E}\left[ \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

Take the difference...

  $\mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i} \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$
   $= \color{#FFA500}{\rho} + \underbrace{\mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]}_\text{Selection bias}$

The experimental ideal

Regression analysis

$$\begin{align} \mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i} \mid \color{#6A5ACD}{\text{D}_{i}=0} \right] = \color{#FFA500}{\rho} + \mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right] \end{align}$$

Again, our estimate of the treatment effect $\left( \color{#FFA500}{\rho} \right)$ is only going to be as good as our ability to shut down the selection bias.

Selection bias in regression model: $\mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

Selection bias here should remind you a lot of

The experimental ideal

Regression analysis

$$\begin{align} \mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i} \mid \color{#6A5ACD}{\text{D}_{i}=0} \right] = \color{#FFA500}{\rho} + \mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right] \end{align}$$

Again, our estimate of the treatment effect $\left( \color{#FFA500}{\rho} \right)$ is only going to be as good as our ability to shut down the selection bias.

Selection bias in regression model: $\mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

Selection bias here should remind you a lot of omitted-variable bias.

There is something in our disturbance $\eta_i$ that is affecting $\text{Y}_{i}$ and is also correlated with $\text{D}_{i}$.

The experimental ideal

Regression analysis

$$\begin{align} \mathop{E}\left[ \text{Y}_{i} \mid \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \text{Y}_{i} \mid \color{#6A5ACD}{\text{D}_{i}=0} \right] = \color{#FFA500}{\rho} + \mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right] \end{align}$$

Again, our estimate of the treatment effect $\left( \color{#FFA500}{\rho} \right)$ is only going to be as good as our ability to shut down the selection bias.

Selection bias in regression model: $\mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

Selection bias here should remind you a lot of omitted-variable bias.

There is something in our disturbance $\eta_i$ that is affecting $\text{Y}_{i}$ and is also correlated with $\text{D}_{i}$.

In other metrics-y words: Our treatment $\text{D}_{i}$ is endogenous.

The experimental ideal

Solutions and covariates

Selection bias in regression model: $\mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

The experimental ideal

Solutions and covariates

Selection bias in regression model: $\mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

As before, if we randomly assign $\text{D}_{i}$, then selection bias disappears.

The experimental ideal

Solutions and covariates

Selection bias in regression model: $\mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

As before, if we randomly assign $\text{D}_{i}$, then selection bias disappears.

Another potential route to identification is to condition on covariates in the hopes that they "take care of" the relationship between $\text{D}_{i}$ and whatever is in our disturbance $\eta_i$.

The experimental ideal

Solutions and covariates

Selection bias in regression model: $\mathop{E}\left[ \eta_i | \color{#e64173}{\text{D}_{i}=1} \right] - \mathop{E}\left[ \eta_i \mid \color{#6A5ACD}{\text{D}_{i}=0} \right]$

As before, if we randomly assign $\text{D}_{i}$, then selection bias disappears.

Another potential route to identification is to condition on covariates in the hopes that they "take care of" the relationship between $\text{D}_{i}$ and whatever is in our disturbance $\eta_i$.

Without very clear reasons explaining how you know you've controlled for the "bad variation", clean and convincing identification on this path is going to be challenging.

The experimental ideal

Covariates

That said, covariates can help with two things:

1. Even experiments may need conditioning/controls: The STAR experiment was random within school—not across schools.

2. Covariates can soak up unexplained variation—increasing precision.

The experimental ideal

Covariates

That said, covariates can help with two things:

1. Even experiments may need conditioning/controls: The STAR experiment was random within school—not across schools.

2. Covariates can soak up unexplained variation—increasing precision.

Now that we've seen regression can analyze experiments, let's estimate the STAR example...

Table of contents