Click on the image to access the course website! 🌐

https://danilofreire.github.io/qtm385/

Lots of additional readings included in the syllabus 😊

• Come here to the front of the room (one at a time!)
• You will look at my screen and answer a question
• Please don’t say the answer out loud and don’t give hints to others!
• Let’s see what happens! 😊

https://forms.gle/xUL39k7ngY2kXYJC7

• As researchers, we are interested in research questions about how the world works

• There are a number of different types of questions that we may want to answer. In academia, they are often divided into two broad categories:

  • Descriptive questions: Descriptions of a given phenomena: e.g., “how do teachers allocate their time during a school day?”
  • Causal questions: Questions about how \(X\) affects \(Y\): e.g., “Does providing vocational training to migrants improve their economic integration in the receiving country?”

• Then we can move on to questions about why? \(\rightarrow\) i.e., knowing the effect of a cause is necessary before moving on to understanding the causes of an effect.

• (Next sessions: more on about what we mean by causality and how experiments give us leverage to make causal claims.)

Click on the image to watch the video! 📺

• Operationalisation: The process of translating theoretical concepts into measurable variables.
  • Example: Turning the concept of “social isolation” into a measurable variable such as the frequency of social interactions
• Key Questions:
  • How will we measure our outcomes?
  • What indicators will be used to represent the underlying concept?
  • How will we manipulate the cause of interest?
  • What intervention or treatment will be implemented?
• Importance of Alignment:
  • The research design must align with the theoretical framework to ensure that the study is addressing the intended questions

• Similarly, we often cannot directly observe the true value of the outcome concept for most of the outcomes we are interested in

• Examples:

  • Correct answers to problems (indicators) for underlying mathematical aptitude (the actual phenomenon)
  • Days without food (indicators) for hunger (the actual phenomenon)
  • Reports of bribes (indicators) for corruption (the actual phenomenon)

• Moreover, the underlying outcome concept may be even under debate (e.g., democracy)

• If our indicators don’t measure the underlying concept that we’re interested in, then we may not be able to learn very much, even if we have an otherwise very sound experiment

https://doi.org/10.1371/journal.pmed.0020124

https://www.science.org/doi/10.1126/science.aac4716

https://www.journals.uchicago.edu/doi/abs/10.1086/734279?journalCode=jop

https://onlinelibrary.wiley.com/doi/full/10.1111/joes.12598

https://onlinelibrary.wiley.com/doi/full/10.1002/jrsm.1703

https://www.nature.com/articles/s41597-022-01143-6

• Pre-registration is the filing of your research design and hypotheses with a publicly-accessible repository. EGAP hosts one that you can use for free (currently on OSF.io using the EGAP registration form)

• Pre-registration does not preclude later exploratory analyses that were not stated in advance. You just have to clearly distinguish between the two

• Even if you will be submitting a paper with results rather than a design to an academic journal or you are primarily interested in a final report with findings for a policy audience, there are important advantages to you and to other researchers from pre-registering your research

  • You can learn about other research, completed and in progress; others can learn about yours. We can learn about studies that produced null results
  • It forces you to state your hypotheses and plan of analysis in advance of seeing the results, which limits \(p\)-hacking

• EGAP is a network of researchers and practitioners who conduct experiments to learn about effective governance and politics
• They have long been advocates of pre-registration and reproducible research, such as with their Metaketa initiative
• Disclosure: I am a member of EGAP and have been involved in several of their projects (including the courses on experimental design 😉)
• They have lots of useful resources for researchers interested in experimental methods, some of them we will be using in this course. More in https://egap.org/

• The EGAP research design form is a guide to help you think through the key components of your research design
• It shows the main parts of a pre-analysis plan, and it is a good blueprint for your own future experiments
• We will be using it in our course, and you can access it here or on our website
• We will have a few sessions dedicated to discussing pre-analysis plans, but I would like to introduce you to the form today (so you can start thinking about your final project!)

• Finally, we will briefly introduce the MIDA framework
• As we have seen, a good research design is a practical plan for research that makes the best use of available resources and produces a credible answer
• But how can we assess the quality of a research design before we implement it?
• Simulate it!
• Luckly, there is a package that does all the hard work for us: DeclareDesign
• Helps us be concrete about the stages of research design by allowing us to represent them in code, which then allows us to simulate the stages of research design in order to understand the properties of the statistical estimators and tests that we use.

• See https://declaredesign.org/

• Regardless of the method, research designs have four components

• MIDA:

  • M: Model (of how the world works)
  • I: Inquiry
  • D: Data strategy
  • A: Answer strategy

• Critical insight: Simulation of a research design teaches what answers a research design can find

• Working with simulated data before data collection helps prevent errors and oversights

• A model of how we think the world works, including:

  • \(T\)s and \(X\)s (treatments or focal causal variables like policy interventions and other background variables)
  • \(Y\)s (dependent variables)
  • Relations between variables (potential outcomes, functional forms, auxiliary variables and contexts)
  • Probability distribution over \(X\)s if not also over \(Y\)s.

• This is the theory!

  • Codified numerically

• The model is wrong by definition. If it were correct, you wouldn’t need to do the study

• But without a model, we don’t have a place to start to assess what can be learned

• An answerable question about the model:

• What is the effect of a treatment \(T\) on an outcome \(Y\) ?

• Usually a quantity of interest, some summary of the data:

  • Descriptive: What is the mean of \(Y\) in treatment, formally.
  • Causal: What would be the average difference of \(Y\) if we switched treatment to control? If we claimed that \(T\) cannot cause \(Y\), how much evidence do we have about this claim?
  • Quantity is the estimand or hypothesis

• Not all questions that we want to ask are answerable

  • And the range of inquiries we can ask are limited: how much can we learn from some summary quantity such as the average treatment effect (ATE)?

• Realise (generate) data on the set of variables (all \(X\)s, \(T\)s and \(Y\)s)

• A function of your model

• Includes both:

  • Sampling — how units arrive in your sample
  • Treatment assignment — what values of endogenous variables are revealed

• Given a realization of the data, generate an answer – an estimate of the quantity of interest (inquiry)

• This is your estimator or test:

  • Difference-in-means
  • \(t\)-test
  • Regression methods
  • etc.

• Answer is an estimate of the quantity of interest or \(p\)-value (inquiry/estimand/test)

• Two-arm trial: A common design in which units are randomly assigned to one of two conditions (treatment and control)
• Model: We have a treatment \(T\) that we think might affect an outcome \(Y\). We have \(N\) units, which we can sample in many ways (simple random sampling, stratified sampling, etc.). We can include background variables \(X\) that we think might affect \(Y\), but let’s keep it simple for now
• Inquiry: What is the effect of \(T\) on \(Y\)?
  • Defined by the average treatment effect (ATE): \(\text{ATE} = \frac{1}{N} \sum_{i=1}^N Y_i(1) - \frac{1}{N} \sum_{i=1}^N Y_i(0)\)
• Data: We randomly assign units to treatment and control, and we measure \(Y\) for each unit
• Answer: We estimate the ATE using a difference-in-means test

• This is a Directed acyclic graph (DAG) of the model
  • A DAG is a way to represent the model of the world, showing the relationships between variables
  • An outcome \(Y\) is affected by unknown factors \(U\) and a treatment \(Z\) (the authors use \(Z\) instead of \(T\))
  • The measurement procedure \(Q\) affects in the sense that it measures a latent outcome and records the measurement in a dataset
  • No arrows lead into \(Z\) because it is randomly assigned

• We will discuss the experimental ideal: the gold standard for causal inference
• But we will also see what to do when we cannot run an experiment (and still use the experimental logic to infer causality)
• Natural and quasi-experiments: the next best thing to a randomised controlled trial
• I will also post the first assignment on the course website and on Canvas
• Please read Blair et al. (2023) if you haven’t yet, as well as the required readings for next class
• … and that’s it for today! 😊🎉