QTM 385 - Experimental Methods
Lecture 17 - Texts for Discussion about Interference and Spillovers
Danilo Freire
Emory University
Brief recap 📚
- Spillovers are a common problem in social science research
- They can be positive or negative, and they can be modelled explicitly in our analysis
- There are several methods to deal with spillovers, such as multi-level designs, within-subject designs, repeated-measures experiments, and waitlist designs
- Each design has its advantages and disadvantages, and the choice of design should be based on the research question and the context
- You can use DeclareDesign to simulate spillover designs and pretest posttest designs
- The statistical analysis of waitlist designs are a little tricky, but you can use the swCRTdesign package in R to help you with that
Interference and spillovers
- Three readings for today:
- Centola (2010): “The Spread of Behavior in an Online Social Network Experiment”
- Paluk et al (2016): “Changing climates of conflict: A social network experiment in 56 schools”
- Gerber and Green (2000): “The Effects of Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment”
Centola (2010)
The Spread of Behavior in an Online Social Network Experiment
- How do social networks affect the spread of behaviour?
- Do tight-knit or loose-knit networks facilitate the spread of behaviour?
- Clustered networks can spread behaviour more quickly than random networks because they have more connections and reinforce each other
- In contrast, weak ties can spread behaviour more widely than strong ties because they connect different clusters
- “Strength of weak ties” (Granovetter, 1973): paradoxical idea that weak ties can be more valuable than strong ties in spreading information
- Which one do you think is more important?
- If you want to spread rumours?
- If you want to spread a new product?
Research design
- Can you easily manipulate the network structure?
- Not always, but you can use random assignment to create different network structures
- The author created an online health community with 1528 participants recruited from internet forums
- All participants were anonymous and were partnered with “health buddies” to share information about health behaviours
- Participants made decisions about whether or not to adopt a health behaviour based on the adoption patterns of their health buddies
- The health behavior used for this study was the decision to register for an Internet-based health forum
- What do you think of this design so far?
Research design
- The author assigned participants to one of two network structures:
- Clustered network with highly connected participants
- Random network with weak ties between participants
- Treatment was assigned before the experiment started and participants were not aware of their treatment assignment
- The author claim the design is better than observational studies because it eliminates confounding variables such as
- Personality traits
- Geographic location
- Advertising and pricing
- Changes in the networks over time
- Can you do the same with massive data from social media?
Research design
- Diffusion was initiated by selecting a random “seed node”, which sent signals to its network neighbours encouraging them to adopt a health-related behaviour
- Every time a participant adopted the behaviour (i.e., registered for the health forum), messages were sent to their health buddies inviting them to adopt
- If a participant had multiple health buddies who adopted the behaviour, then they would receive multiple signals, one from each neighbour
- The more neighbors who adopted, the more reinforcing signals a participant received
- The author measured the fraction of participants who adopted the behaviour
Results
![]()
- Cumulative distribution of forum visits over time. The lower, the higher the likelihood of checking the forum
Discussion
- The author concludes that clustered networks are more effective than random networks in spreading behaviour
- But there are some limitations to this study, too
- Can you think of any?
- Can you think of policy implications?
Paluk et al (2016)
Changing climates of conflict: A social network experiment in 56 schools
- How can we change community-wide patterns of behaviour? (again!)
- Focus on adolescent school conflict - verbal/physical aggression, rumour-mongering, social exclusion
- Existing interventions target individual psychology (persuasion), mass education, or institutional regulation
- This study explores a social influence strategy:
- Seed a social network with individuals demonstrating new behaviours
- Rely on social influence to spread behaviour through network structures
- Key question: Can a small group of influential people shift a community’s behavioural climate?
Research design
- Multilevel experiment across 56 US middle schools (24,191 students)
- Randomised at school level (treatment vs. control)
- Randomised at student level within treatment schools (seed students)
- Comprehensive measurement of school social networks before intervention
- “Who do you choose to spend time with?” - measures attention in network
- Identified “social referents” - top 10% most connected students in each school
- Intervention: Anti-conflict campaign driven by seed students
- Encouraged public stance against conflict
- Seed students (20-32 per school in treatment schools) were selected to participate
- Intervention activities: creating slogans, posters, wristbands
Intervention details
- Grassroots campaign approach
- Seed students identified school-specific conflict behaviours
- Developed slogans and posters
- Distributed wristbands as rewards for positive behaviour
- No top-down, adult-defined problems: focused on student perspectives
- Standardised procedures with trained facilitators using scripts and activity guides
- High attendance at meetings (average >55% of invited students)
- Made meetings engaging: snacks, hands-on activities, technology
Measures
- Outcomes measured subjectively and administratively
- Subjective: student-reported norms about conflict (pre/post surveys)
- Descriptive norms: how many students participate in conflict?
- Prescriptive norms: how many students disapprove of conflict?
- Administrative: school disciplinary records of peer conflict events (across the year)
- Focus on disciplinary events as a key behavioural outcome
- Arguably less susceptible to reporting bias than self-reported conflict
Results
- Significant reduction in disciplinary reports in treatment schools ~30% reduction overall
- Effect stronger when seed groups contained more “social referent” students
- Increased talking about conflict reduction and wearing wristbands in treatment schools
- No average differences in social norms between treatment and control schools initially
- Social referent seeds were more influential than other seed students
- 20% social referent seeds -> 60% reduction in disciplinary events
- Students exposed to social referent seeds more likely to:
- Report friends discussing conflict reduction
- Perceive stronger anti-conflict norms
- Wear wristbands
- No peer-to-peer influence effect on discipline directly (climate-level effect stronger)
Interpretation
- Intervention shifted school climate - widespread behavioural pattern
- Social referents key to norm and behaviour change
- Outsized influence due to attention from peers
- Not just structural position, but also traits and experiences
- Norms findings nuanced:
- No overall shift in norms between schools
- But within treatment schools, norms did shift, especially via social referents
- Possible “signal” effect of intervention - increased attention to conflict, leading to revised norm evaluations
Implications
- Peer influence interventions can be effective in reducing school conflict
- Importance of social referents in changing behaviour
- Target these individuals in interventions for greater impact
- Highlights the usefulness of student-led initiatives
- Grassroots approach can be more effective than top-down
- Methodological contribution:
- Demonstrates rigorous social network experiment in real-world setting
- Multilevel randomisation for causal inference in networks
Limitations
- What are some potential limitations of this study?
- Generalisability to other contexts? (different types of conflict, schools, cultures)
- Long-term effects?
- Mechanism of change fully understood?
- Reliance on disciplinary records - are these fully objective?
- Ethical considerations of network interventions? (manipulation of social structures?)
- What about the students who were not selected as “seeds” but were still part of the network? Were they affected negatively?
Gerber and Green (2000)
The effects of canvassing, telephone calls, and direct mail on voter turnout: A field experiment
- How effective are different get-out-the-vote (GOTV) tactics?
- Focus on three common methods:
- Personal Canvassing (face-to-face)
- Direct Mail (mailings)
- Telephone Calls (phone banking)
- Context: Declining voter turnout in the US
- Hypothesis: Decline linked to shift from personal to impersonal mobilisation
- Key Question: Is face-to-face canvassing more effective than impersonal methods in boosting voter turnout?
Research design
- Large-scale field experiment in New Haven, Connecticut (1998 election)
- ~30,000 registered voters
- Random assignment at household level to different GOTV treatments & control
- 2x2x4 Factorial Design:
- Personal Canvassing
- Telephone Calls
- Direct Mail (None, One, Two, or Three mailings)
- Non-partisan messages used for all treatments
- Civic duty, close election, neighbourhood solidarity appeals
- Outcome: Validated voter turnout from official records
GOTV treatment details
- Personal Canvassing:
- Paid canvassers (grad students) in pairs, weekends before election
- Targeted specific households, not entire streets
- Varied message appeals (civic duty, close election, neighbourhood solidarity)
- Contact rate: ~28%
- Direct Mail:
- Postcards designed by political consulting firm
- Varied number of mailings (1, 2, or 3) and message appeals
- Telephone Calls:
- Out-of-state telemarketing firm, calls in days before election
- Scripted calls, civic duty and close election appeals only
- Contact rate: ~32% (reached household)
Main effects
- Personal Canvassing Highly Effective:
- +8.7 percentage points increase in turnout (2SLS estimate)
- ~9.8 points in multivariate model
- Most effective GOTV method tested
- Direct Mail Shows Small Effect:
- +0.6 points per mailing (multivariate model)
- Small but statistically significant
- Cumulative effect of 3 mailings ~ 2.5 points (probit)
- Telephone Calls Ineffective:
- No significant effect, slight negative effect in some models
- Despite professional firm and mirrored messaging
Message and cost-effectiveness
- Message Type Mattered Little for Personal Canvassing:
- Civic duty, close election, neighbourhood solidarity appeals similarly effective (though close election slightly higher)
- No significant interaction effects between treatments
- Cost-Effectiveness Analysis:
- Personal canvassing: ~$8 per additional vote
- Direct mail (3 mailings): ~$40 per additional vote
- Face-to-face mobilisation significantly more cost-effective
Interpretation
- Supports Hypothesis: Personal canvassing > impersonal methods for turnout
- Decline in face-to-face mobilisation may contribute to turnout decline
- Unexpected Finding: Ineffectiveness of telephone calls
- Despite professional execution & mirroring canvassing messages
- Possible reasons: Routinized/scripted nature, out-of-state callers
- Mechanism of Personal Canvassing Effect?
- Speculation: Salience, memorability, connection, urgency… (future research needed)
- Limitations: Non-partisan messages, specific election context, short-term effects
Implications
- Personal canvassing is a valuable GOTV tool
- More effective & cost-effective than mail/phone
- Challenges conventional wisdom? - Heavy reliance on mail/phone in modern campaigns
- Highlights importance of personal connection in political mobilisation
- Methodological Contribution:
- Rigorous field experiment in political science
- Addresses non-compliance with IV approach
- Future Research: Explore mechanisms, partisan vs. non-partisan, long-term effects, interactions of methods
Limitations
- What are potential limitations of this text?
- Generalisability to other elections/contexts?
- Focus on turnout only - what about vote choice/persuasion?
- Short-term effects only - sustained impact?
- Specific message content - would different messages work better for mail/phone?
- Ethical considerations of GOTV experiments? (Manipulation of voters?)
- Hawthorne effect of canvassing? (People vote more because of the attention, not just the message)
Conclusion
- Centola (2010)
- Showed how network structure (clustered vs. random) impacts spillovers
- Paluck et al. (2021)
- Addressed interference by focusing on network positions (social referents) as key spillover agents
- Multilevel design (school & student) managed school-level vs. peer-to-peer influence spillovers
- Gerber & Green (2000)
- Demonstrated differential spillovers: Face-to-face (strong), Mail (weak), Phone (none)
- IV approach key to address interference from self-selection into contact, isolating causal effect
- Common thread: addressing interference
- All three papers grapple with inherent dependencies in social settings (networks, communities, voter outreach)
- Highlight the need for designs and analysis that account for, or exploit, these interdependencies
- Beyond “simple” experiments
- Social science experiments increasingly move beyond simple treatment/control to address complex social interactions & spillovers
- These examples show innovative designs for studying influence in interconnected systems
And that’s all for today! 🎉
