Week 7

Computational Sociology

Christopher Barrie

Introduction

Housekeeping
Bots and Experiments

Introduction: Experiments

Experiments versus surveys:

In experiments – randomization into treatment not into the sample

Introduction: Experiments

So how representative does it have to be…?

--> Depends on the question and how unrepresentative

--> Depends on whether we think likely to be treatment effect heterogeneity

Introduction: Experiments

Treatment could be:

Exposure to some information
Encouragement to stop using social media
Playing a game with an individual who votes for opposing political party

…and countless more

Experiments and the Internet

Online crowd work using online workers
Online experiments with humans
Online treatments using bots
Machine-only experiments without any humans…

What am I talking about?

Online crowd work using online workers

Crowd workers

MTurk and Prolific (and others) provide:

Constantly live pool of online workers for:
1. For labelling tasks
2. Survey tasks
3. Experiments

Crowd workers

But are these representative…?

Posch et al. (2018)

Crowd workers

But do they need to be representative?

Remember: experimental research requires randomization into treatment not sample
- But how unrepresentative can it be?

Thomas and Clifford (2017)

Crowd workers

So how have these been used…?

Crowd workers

So how have these been used…?

For data labeling

Rodriguez and Spirling (2022)

Crowd workers

So how have these been used…?

For data labeling
As experimental samples for high-impact research

Brady et al. (2021)

Mize and Manago (2018)

Bots as treatment

Can code substitute for the confederate?

Munger (2016)

Bots as treatment

Haroon et al. (2022)

Experimenting with machines alone…

A note on computational thinking

This week:

Extending social theory to online world through use of platform affordances
Auditing algorithmic influence by experimenting with algorithmic output itself–i.e., doing social science experiments without involving any human beings…

References

Brady, William J., Killian McLoughlin, Tuan N. Doan, and Molly J. Crockett. 2021. “How Social Learning Amplifies Moral Outrage Expression in Online Social Networks.” Science Advances 7 (33). https://doi.org/10.1126/sciadv.abe5641.

Haroon, Muhammad, Anshuman Chhabra, Xin Liu, Prasant Mohapatra, Zubair Shafiq, and Magdalena Wojcieszak. 2022. “YouTube, the Great Radicalizer? Auditing and Mitigating Ideological Biases in YouTube Recommendations.” https://doi.org/10.48550/ARXIV.2203.10666.

Mize, Trenton D., and Bianca Manago. 2018. “Precarious Sexuality: How Men and Women Are Differentially Categorized for Similar Sexual Behavior.” American Sociological Review 83 (2): 305–30. https://doi.org/10.1177/0003122418759544.

Munger, Kevin. 2016. “Tweetment Effects on the Tweeted: Experimentally Reducing Racist Harassment.” Political Behavior 39 (3): 629–49. https://doi.org/10.1007/s11109-016-9373-5.

Posch, Lisa, Arnim Bleier, Fabian Flöck, and Markus Strohmaier. 2018. “Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographics.” https://doi.org/10.48550/ARXIV.1812.05948.

Rodriguez, Pedro L., and Arthur Spirling. 2022. “Word Embeddings: What Works, What Doesn’t, and How to Tell the Difference for Applied Research.” The Journal of Politics 84 (1): 101–15. https://doi.org/10.1086/715162.

Thomas, Kyle A., and Scott Clifford. 2017. “Validity and Mechanical Turk: An Assessment of Exclusion Methods and Interactive Experiments.” Computers in Human Behavior 77 (December): 184–97. https://doi.org/10.1016/j.chb.2017.08.038.