• Mediation analysis is a statistical technique to understand the mechanism through which an independent variable affects a dependent variable
• It identifies intermediate variables (mediators) that lie on the causal pathway between the intervention and the outcome
• Divides total effect: direct (intervention → outcome) and indirect (via mediator)
• Mediation analysis is based on the counterfactual framework and requires strong assumptions to identify causal effects
• An important set of assumptions is sequential ignorability: The mediator is not affected by the treatment after controlling for confounders
  • This is a strong assumption that is often violated in practice
  • When this assumption is violated, the direct and indirect effects are not identified

• Mediation analysis can be extended to include multiple mediators, either acting in parallel (independent pathways) or in sequence (a chain of mediators)
  • Even stronger assumptions!
• Longitudinal RCTs allow for the examination of time-varying mediators and outcomes
• Even in well-conducted RCTs, mediation analysis is still observational with respect to the mediator
• Sensitivity analysis can be used to assess the robustness of the results to violations of the assumptions

Question 02

• Estimate \(E[Y_{01} - Y_{00}]\) for the random assignment that places the treatment at location A
• Village A is treated. So we have this (from the table):
  • \(Y_{01}^{(A)} = 0\) and \(Pr(treat) = 0.2\)
• Treated part of the equation: \[ \frac{0 / 0.2}{1 / 0.2} = \frac{0}{5} = 0 \]
• Villages C, D, and F are untreated and not next to A (\(Y_{00}\). So:
  • \(Y_{00}^{(C)} = 0, \quad Pr(control) = 0.4\)
  • \(Y_{00}^{(D)} = 6, \quad Pr(control) = 0.6\)
  • \(Y_{00}^{(F)} = 6, \quad Pr(control) = 0.8\) \[ \frac{0/0.4 + 6/0.6 + 6/0.8}{1/0.4 + 1/0.6 + 1/0.8} = \frac{0 + 10 + 7.5}{2.5 + 1.667 + 1.25} = \frac{17.5}{5.417} \approx 3.23 \]
• And the result is:

\[ \hat{E}[Y_{01} - Y_{00}] = 0 - 3.23 = -3.23 \]

Question 03

• Estimate \(E[Y_{10} - Y_{00}]\) for the random assignment that places the treatment at location A, restricting the sample to the set of villages that have a non-zero probability of expressing both of these potential outcomes.

• Village B is the only with spillovers (\(Y_{10}\))

  • \(Y_{10}^{(B)} = 2\), \(Pr(spillover) = 0.4\)
  • \(\frac{2/0.4}{1/0.4} = \frac{5}{2.5} = 2.0\)

• Villages C and D are neither treated nor next to A (\(Y_{00}\))

  • \(Y_{00}^{(C)} = 0\), \(Pr(control) = 0.4\)
    
  • \(Y_{00}^{(D)} = 6\), \(Pr(control) = 0.6\)

• Thus, for the control part of the equation: \[ \frac{0/0.4 + 6/0.6}{1/0.4 + 1/0.6} = \frac{0 + 10}{2.5 + 1.667} = \frac{10}{4.167} \approx 2.4 \]

• Final result:

\[ \hat{E}[Y_{10} - Y_{00}] = 2.0 - 2.4 = -0.40 \]

• Discuss survey experiments as a methodological tool for understanding attitudes, preferences, and behaviours
• Examine why they’ve become popular across social sciences and marketing research
• Analyse their versatile applications from policy research to behavioural economics
• Highlight the crucial balance between experimental simplicity and rigorous design requirements

Think about whether the population of Madrid is more or less than 500,000 people.

What do you think the population of Madrid is?

• Answer here: https://tinyurl.com/qtm385-survey

Think about whether the population of Madrid is more or less than 20,000,000 people.

What do you think the population of Madrid is?

• Answer here: https://tinyurl.com/qtm385-survey

• What did you guess? (See Responses)
• What’s going on here?
  • A survey experiment!
    
  • Demonstrates “anchoring” heuristic
• Experiments are easy to analyse, but only if designed and implemented well!

• Do you think the US should do more than it is now doing to help England and France?

• Yes: 13%

• No

• Do you think the US should do more than it is now doing to help England and France in their fight against Hitler?

• Yes: 22%

• No

• A survey experiment is a research design that combines the random assignment of treatments with survey data
• It is particularly used to understand attitudes, preferences, and behaviours
• Survey experiments are popular because they are easy to implement and analyse, and they can be used to test causal relationships in a controlled environment
• They can be implemented in many ways:
  • Face-to-face
  • Telephone
  • Online
  • Mail, etc

• What stimulus each group receives depends on hypotheses being tested
• Three ways hypotheses lead to stimuli:
  • Presence/absence: Does the stimulus have an effect?
  • Levels/dose: Does the amount of stimulus matter?
  • Qualitative variations: Does the type of stimulus matter?
• Survey experiments can be between-subjects or within-subjects
• They can also be used to measure sensitive topics, which is something we will discuss in the future

Presence/absence

• Theory: Negative campaigning reduces support for the party described negatively
• Hypothesis: Exposure to a negative advertisement criticising a party reduces support for that party
• Manipulation:
  • Control group receives no advertisement
  • Treatment group watches a video containing a negative ad describing a party
• Outcome: Support for the party described negatively

Levels/dose

• Theory: Negative campaigning reduces support for the party described negatively
• Hypothesis: Exposure to higher levels of negative advertising criticising a party reduces support for that party.
• Manipulation:
  • Control group receives no advertisement
  • Treatment group 1 watches a video containing 1 negative advert describing a party
  • Treatment group 2 watches a video containing 2 negative adverts describing a party
  • etc

• What are some criteria for a good survey experiment?

• Outcomes affected are consistent with theory
• Before the study, using pilot testing (or pre-testing)
• During the study, using manipulation checks
• During the study, using placebos
• During the study, using non-equivalent outcomes

• Outcomes affected:
  • True, but doesn’t tell us anything if we hypothesise null effects
• Pilot testing:
  • Goal: establish construct validity of manipulation
  • Assess whether a set of possible manipulations affect a measure of the independent variable
• Example:
  • Goal: Manipulate the “strength” of an argument
  • Write several arguments
  • Ask pilot test respondents to report how strong each one was

• Manipulation checks are items added post-treatment, post-outcome that assess whether the independent variable was affected by treatment
• We typically talk about manipulations as directly setting the value of \(X\), but in practice we are typically manipulating something that we think strongly modifies \(X\)
• Example: information manipulations aim to modify knowledge or beliefs, but are necessarily imperfect at doing so

• Example:
  • Treatment 1: Supply Information
  • Manipulation check 1: measure beliefs
  • Treatment 2: Prime a set of considerations
  • Outcome: Measure opinion
  • Manipulation check 2: measure dimension salience
• Manipulation checks should be innocuous: not affect the treatment or outcome
• Measure both what you wanted to manipulate and what you didn’t want to manipulate

Placebos

• Include an experimental condition that does not manipulate the variable of interest (but might affect the outcome)
• Example:
  • Study whether risk-related arguments about climate change increase support for a climate change policy
  • Placebo condition: control article with risk-related arguments about non-environmental issue (eg., terrorism)

Non-equivalent outcomes

• Measures an outcome that should not be affected by independent variable
• Example:
  • Assess effect of some treatment on attitudes toward group A
  • Focal outcome: attitudes toward group A
  • Non-equivalent outcome: attitudes toward group B

• How close do you feel to your ethnic or racial group?
• Some people have said that taxes need to be raised to take care of pressing national needs. How willing would you be to have your taxes raised to improve educational opportunities for minorities?

• How close do you feel to other Americans?
• Some people have said that taxes need to be raised to take care of pressing national needs. How willing would you be to have your taxes raised to improve educational opportunities for minorities?

Knowledge and political interest

• Do you happen to remember anything special that your US Representative has done for your district or for the people in your district while he has been in Congress?

• Is there any legislative bill that has come up in the House of Representatives, on which you remember how your congressman has voted in the last couple of years?

Instructions

In the next part of this study, you will be asked 14 questions about politics, public policy, and economics. Many people don’t know the answers to these questions, but it is helpful for us if you answer, even if you’re not sure what the correct answer is. We encourage you to take a guess on every question. At the end of this study, you will see a summary of how many questions you answered correctly.

• Vignettes are short descriptions of hypothetical scenarios
• Vignettes are probably the most common survey experimental paradigm, after question wording designs
• Take many forms and increasingly encompass non-textual stimuli
• Basically limited to web-based mode

• Gilens (1996)

• Now think about a (black/white) woman in her early thirties. She is a high school (graduate/drop out) with a ten-year-old child, and she has been on welfare for the past year.
  

• How likely is it that she will have more children in order to get a bigger welfare check? (1 = Very likely, . . . , 7 = Not at all likely)

• How likely do you think it is that she will really try hard to find a job in the next year? (1 = Very likely, . . . , 7 = Not at all likely)

• Images can also work well
• Subliminal primes possible, depending on software
• Lots of recent examples of facial manipulation
• Example:
  • Iyengar, S. et al (2008)

Audio and video

• Problematic for same reasons as long texts
• Best practices:
  • Keep it short
  • Have the video play automatically
  • Disallow survey progression
  • Control and validate
• Examples:
  • Television Advertisements
  • News Programs

Tasks

• Task designs ask respondents to perform a task
• Often developed for laboratory settings
• Most common example: writing something
• Can be problematic:
  • Time-intensive
  • Invites drop-off
  • Compliance problems