Open Science tools

PSM2 UCL

Isabelle van der Vegt

19 Feb 2019

Today’s tutorial

  • Pre-registrations: why and how
  • Power analysis: how-to
  • Effect size conversions
  • R Markdown how-to

Your final project

  • Answer RQ(s) with dataset you are given
  • Requirements: pre-registration, reproducible code

Pre-registrations

What is a good/interesting scientific result?

What is a good/interesting scientific result?

  • “significant difference”
  • “x predicts y”
  • “p < 0.05”

The research process

  • Generate hypotheses
  • Design study & collect data
  • Analyse data
  • Interpret data

Compare

  • Generate hypotheses → Introduction
  • Design study & collect data → Method
  • Analyse data → Results
  • Interpret data → Discussion

The real research process

QRP 1: HARKing / post-hoc theorizing

  • Design study & collect data
  • Analyse data
  • Generate hypotheses
  • Interpet data → Look, a “good” result!

The real research process

QRP 2: significance chasing/p-hacking

  • Generate hypotheses
  • Design study & collect data
  • Analyse data
  • Collect MORE data
  • Interpret data → Nice, just as expected!

The real research process

QRP 3: selective reporting

  • Generate hypotheses
  • Design study & collect data
  • Analyse data
  • Exclude condition/data not in line with results
  • Interpret data → Wow, an interesting result!

The problem

  • Everyone wants interesting results → QRPs

The solution

Pre-registrations

Specifying your hypotheses, study design, and analysis plan BEFORE collecting/analysing data.

How?

  • Public, independent record
  • Timestamps
  • Explain deviations from pre-registered plans
  • Distinguish confirmatory and exploratory results in paper

Open Science Framework

  • Open a project
  • Complete pre-registration form (free text or template)
  • Upload additional files
  • Register the project, this ‘freezes’ it
  • Make public or share with specific people
  • Anonymous links

Register

5 minutes: osf.io