Welcome to QTM 385 - Experimental Methods
Lecture 01: Introduction
Danilo Freire 
Department of Quantitative Theory and Methods
Emory University
Instructor
A bit about me
Visiting Assistant Professor in the QTM
MA from the Graduate Institute Geneva, PhD from King’s College London, Postdoc at Brown University, Senior Lecturer at the University of Lincoln, UK
Research interests: computational social science, experimental methods, policy evaluation, political violence, organised crime
Class etiquette
- Learning a new topic can be tough and push you out of your comfort zone. If the course pace is too fast, let us know. I expect your commitment, but I do not want anyone to fail
- You are all keen on experimental methods, but your backgrounds vary. That is great! Some sessions might be more engaging than others. If you are bored, help others or explore new data science areas
- Always be respectful to each other
- Ask questions whenever you need to!
Randomised controlled trials (RCTs): diagram
Source: Liakos et al (2024)
What are randomised controlled trials (RCTs)?
- Randomised controlled trials (RCTs) are a type of scientific experiment that aim to reduce bias when testing a new treatment or intervention
- In its simplest form, a RCT consists of a control group and a treatment group, with the treatment group randomly receiving the intervention and the control group not receiving it
- They are often used in medicine, psychology, and social sciences to evaluate the effectiveness of new treatments or interventions
- It is easy to take them for granted, but these trials have transformed our understanding of cause and effect
- They are the best tool we have to illuminate what is unknown or uncertain, and to discern whether something works and how well it works
From xkcd
History of experiments
Francis Pretrarch
- Petrarch (1394): “I solemnly affirm and believe, if a hundred or a thousand of men of the same age, same temperament and habits, together with the same surroundings, were attacked at the same time by the same disease, that if the one half followed the prescriptions of doctors of the variety of those practicing at the present day, and that the other half took no medicine but relied on Nature’s instincts, I have no doubt as to which half would escape”
History of experiments
James Lind
- In 1747, Scottish naval surgeon James Lind sought to cure scurvy during a sea voyage
- Lind implemented the first (non-randomised) experiment comparing two treatments across six pairs of sailors
- One effective treatment involved administering two oranges and one lemon daily to sailors
- To ensure consistency, Lind aimed for similarity among participants: “The cases were as similar as I could have them. They all in general had putrid gums, the spots and lassitude, with weakness of their knees. They lay together in one place, being a proper apartment of the sick in the fore-hold; and had one diet common to all.”
History of experiments
- The first random assignment of treatment and control happened in Nuremberg, 1835, to assess homeopathic medicine (yes, it’s been ongoing for a while!)
- In 1884, Peirce and Jastrow conducted the first psychological experiment using randomisation to study sensory perception
- This approach later became standard practice in experimental psychology and social sciences
- In 1947, the Medical Research Council carried out the first RCT to evaluate streptomycin for tuberculosis
- Results showed streptomycin was effective, and the RCT became the gold standard for evaluating medical treatments
Recent examples: Banerjee, Duflo, and Kremer
Recent examples: Card, Angrist, and Imbens
Types of Experiments
- Lab experiments: Conducted in controlled environments
- Field experiments: Conducted in real-world settings
- Natural experiments: Use naturally occurring events as treatments
- Quasi-experiments: Use non-randomised designs
- Why are natural/quasi-experiments not as reliable as RCTs?
Correlation versus causation
Source Tyler Vigen
Correlation versus causation
Source Tyler Vigen
Correlation versus causation
Source Tyler Vigen
Correlation versus causation
From xkcd
David Hume and causality
- David Hume (1739): “We may define a cause to be an object followed by another, and where all the objects similar to the first are followed by objects similar to the second. Or in other words where, if the first object had not been, the second never had existed.”
- Three concepts worth noting:
- Temporal sequence: The cause must precede the effect
- Covariation: The cause and effect must vary together
- Non-spuriousness: The relationship must not be due to a third variable
Five principles of causality
- A more modern take involves 5 principles:
- Relationship: There must be a relationship between the cause and effect
- Direction (temporality): The cause must precede the effect
- Nonconfounding: The relationship must not be due to a third variable
- Mechanism: There must be a plausible mechanism linking the cause and effect
- Appropriate level of analysis: The relationship must happen (and be measured) at the appropriate level of analysis
Course objectives
- Design rigorous experiments with proper randomisation procedures and sample sizing calculations
- Create pre-analysis plans (PAP) and apply appropriate statistical methods for experimental analysis
- Produce reproducible reports using Quarto
- Evaluate experimental designs, identifying potential limitations
- Manage unexpected data challenges, such as attrition and non-compliance
- Understand ethical considerations in experimental design
- Develop analytical skills through practical problem sets and discussion sessions
Textbooks
Required: Alan Gerber and Donald Green - Field Experiments
Recommended: Douglas C. Montgomery - Design and Analysis of Experiments
Lots of additional readings included in the syllabus 😊
