• Sensitive topics are difficult to study!
• Social desirability bias is a common problem in survey research
• List experiments measure prevalence indirectly through item counts
• Randomised response techniques use probability models for anonymity
• Endorsement experiments assess support without direct attribution
• Conjoint analysis measures preferences through trade-off scenarios
• Maintain plausible deniability for respondents
• Software to estimate these models: http://sensitivequestions.org/

• As you can expect, the authors use a list experiment in this article
• Their sample comes from 12 major universities competing in the Division I NCAA Athletics
• They accessed the athletic websites of all 12 Big Ten schools and obtained the full rosters for all sports at every school
• The authors located 6,375 names on rosters, and after discarding emails that bounced back, ended up with 4,093 e-mail addresses
• They sent the survey to all of these athletes, and received 1,303 responses (~31% response rate, not bad!)
• Questions:
  • Is it okay to collect emails from the internet and send unsolicited surveys?
  • What are the pros and cons of this approach?
  • How would you analyse the data? ATE or ITT?
  • The authors also argue that “the experimental nature of our key measurement approach means that obtaining a perfectly representative sample is of much less importance than is the random assignment of our experimental treatment between groups” (p. 374). Do you agree with this statement?

• The authors first measure the prevalence of drug and alcohol use in the College Senior Survey 2009
  • The CSS is a survey of college seniors conducted by the Higher Education Research Institute at UCLA
  • Sample includes 24,457 individuals from 111 colleges and universities
• Student-athletes are substantially less likely—in self-reports—to drink beer or use drugs than their non-athlete peers (p. 374)
• “Nearly 75 percent of the general student population say they “frequently” or “occasionally” drink beer, whereas only 46 percent of our student-athlete sample say the same (\(z = 20.55; p <0.01\))” (p. 374)
• Does that surprise you?

Below is a list of activities that you may have engaged in over your time in college. We are interested in HOW MANY of these activities you have engaged – NOT which ones. Thus, please simply choose the number at the end of the list.

• Sustained an injury during a practice or game that prevented you from playing at least one other game

• Joined a social club whose majority of members did/does not include varsity athletes

• Skipped a class because you felt so tired from a practice or a game

• Was unable to take a class that you hoped to take because of your practice or game schedule

• Knowingly took a drug banned by the NCAA that may improve your athletic performance

• Note that the authors asked about knowingly taking a drug banned by the NCAA
• The results were stark:
  • Control group = mean response of 3.31 (SD = 0.78; n = 553)
  • Treatment group = mean response of 3.68 (SD = 0.98; n = 510).
  • The difference between the two groups is statistically significant (\(t_{1061}\) = 6.90; p < 0.01)
  • The authors conclude that the list experiment “suggests that 37 percent of respondents have knowingly taken banned drugs” (p. 377)
• Direct question = 4.9 percent 😂
• What do you think about this result?

Below is a list of activities that you may have engaged in over your time in college. We are interested in HOW MANY of these activities you have engaged – NOT which ones. Thus, please simply choose the number at the end of the list.

• Your choice of which University to attend was determined largely by the sports opportunities (e.g., it weighed in at least in 50% of your decision)
• Stayed up past 1 AM, on average, during the season of your sport
• Plan to continue playing your sport after college, although not necessarily on the professional level
• Play other sports during the school year at least once a month
• In the typical week during the past academic year, consumed more than five alcoholic drinks

• Again, large statistical difference:
  • Control group = mean response of 2.76 (SD = 0.85; n = 544)
  • Treatment group = mean response of 3.22 (SD = 1.10; n = 556).
  • 46 percent consumed more than five drinks a week
• Less than 3 percent of athletes reported that they consumed more than five drinks a week when asked directly

• This paper is both a substantive and methodological contribution
• Their main methodological contribution is to compare (and eventually combine) list and endorsement experiments
• The setting is pretty challenging, too: Afghanistan
• Motivation: regime change, “hearts and minds” campaign, and the Taliban
• Outcome: support for the Taliban and foreign (NATO) forces (ISAF)
• Here, for the sake of time, we will focus on their experimental design, but I encourage you to read the paper! 🤓
• Note: all models were estimated with the list and endorse R packages

• Respondents also had the option to say “Refuse to answer” or “Don’t know”
• Refusal rates were very low (about 6%) when compared to direct questions

Source: Blair et al. (2014)

Source: Blair et al. (2014)

• These are very interesting results!
• Taliban victimisation leads to a modest increase in support for ISAF
• ISAF victimisation is associated with a consistently negative effect across all models
• List experiment model estimates a modest positive effect, suggesting that Taliban post-harm efforts may actually be associated with an increase in ISAF support; the endorsement experiment model suggests the opposite!
• Interestingly, the authors don’t know exactly why this happens 😂
• So maybe we need a “cleaner” design to understand this better?

Source: Blair et al. (2014)

• The first question was pre-tested in a pilot study and was just included to see if the respondent was being honest
• People mainly gave honest answers to the first question (about 90% of the time)
• The rest was maybe confused by the procedure, which is indeed one of the disadvantages of this method

Source: Rosenfeld et al. (2016)

Source: Rosenfeld et al. (2016)

Source: Rosenfeld et al. (2016)

• Control group: In Brazil, some people believe that lynching may be justified under certain conditions. To what degree do you agree or disagree that lynching can be justified? Please use the slider below to indicate your preference. For disagreement, use 0–49; for agreement, use 51–100. Please use 50 if you neither agree nor disagree.

• Treatment 1 (Legal punishment for lynching perpetrators): In Brazil, some people believe that lynching may be justified under certain conditions. However, the Brazilian constitution and penal code strictly forbid lynching and those involved can be accused of torture or murder. To what degree do you agree or disagree that lynching can be justified? Please use the slider below to indicate your preference. For disagreement, use 0 to 49; for agreement, use 51 to 100. Please use 50 if you neither agree nor disagree.

• Treatment 2 (Human rights): […] However, the Brazilian constitution states that all individuals have the right of not being tortured, including criminals.

• Treatment 3 (Vendettas): […] However, lynchings can trigger a new cycle of violence as the family or friends of the victim may retaliate against the community.

• I hope you liked the papers! 😊
• Lots of things to think about:
  • How to leverage list experiments to measure hidden behaviours
  • Combining different methods to assess support for foreign forces
  • Comparing results from survey experiments with a clear benchmark
  • How to use conjoint experiments in a real-world setting
• Be creative! 🤓