<div id="left"> <img src="img/DIL_logo_safespace_RGBFull Color.svg" alt="DIL logo" width="400"> </div> # Data validation February 10, 2023

## Data validation - Input: a clean and tidy data set - Objectives: 1. Ensuring that the data quality is as high as it can be 1. Documenting patterns that may bias our analysis - Output: - A data set that is very similar do the initial one, but may contain some corrections - Documentation about any data quality issues found in the data --- ## No data is safe from quality issues Note: - raise your hand if you ever found something in your data that simply did not make sense - raise your hand if you are in the habit of testing the quality of the data before analyzing it - raise your hand if your data processing code automatically flags issues in the data --- ## This session <ul> <li>Review what are common data issues to look up</li> <ul class="fragment fade-in"> <li>Data set keys</li> <li>Completeness</li> <li>Consistency</li> <li>Extreme values</li> </ul> <li class="fragment fade-in">Test some tools to automate data validation</li> </ul>

ID variable

## Exploring new data What is the first thing you look at when working with a new data set? <div class="fragment fade-in"> <ol> <li>What is the <b>unit of observation</b> in this data table?</li> <li>What column in the data <b>identifies</b> a unit in this data table?</li> </ol> </div> --- ## Data set key (ID variable) <ol> <li>Uniquely identifies observations</li> <li>Fully identifies observations</li> <div class="fragment fade-in"> <li>Anonymous</li> <li>Constant across a project</li> </div> </ol> --- ## Useful tools - Stata: - `isid` - `ieduplicates` (from `iefieldkit`) - `duplicates` - R: - `unique` - `nrow` - `n_distinct` (from `dplyr`) - `get_dupes` (from `janitor`) - `isKey` (from `dataReporter`)

Completeness

## Completeness - Missing values are common source of issues when handling data - When inspecting new data, check that the data is complete - Across time - Across space or groups - Across units of observation Note: - Skip patterns should normally be predictable. For example, you're only asked how many hours a week you work if you are employed - Do all the observations in one region show missing values for a particular variable? - Is a particular enumerator skipping entire survey sections? - Was some information only collected after a certain date? --- # Useful tools - Stata: - `missing()` - `misstable` - R: - [`naniar`](https://naniar.njtierney.com/) - `is.na` - `remove_empty` (from `janitor`)

Consistency

## Consistency Values in a data set should be consistent 1. For any given variable across observations 1. Across variables for any given observation 1. For related information across data tables 1. With pre-existing information and common sense <p class="highlight fragment fade-in"> This is the most subjective issue in the list, and requires knowledge of the data and of the subject matter </p> Note: Ask for examples for each case: 1. Yes/No and Y/N used for the same variable 1. Pregnant males 1. Mother says she has no children on mother data table, but is listed as a mother in the children table 1. Being 150 years-old is not reasonable, if we know there are only 10 states in a country, we should not have data for 8 different states --- # Useful tools - Stata: [`ipacheck`](https://github.com/PovertyAction/high-frequency-checks) - R: - [`validate`](https://data-cleaning.github.io/validate/) - [`pointblank`](https://rich-iannone.github.io/pointblank/)

Extreme values

## Extreme values - Identifying outliers can help us find incorrect values in our data - However, extreme values are not always mistakes - Still, even correctly measured outliers can affect our analysis - For example, by leading to results that are driven by only a few units in our sample <p class="highlight fragment fade-in"> So how should we handle them? </p> Note: I'm sorry to say I don't have an answer --- ## Food for thought <ul> <li>There is no exact rule for what is an outlier</li> <li class="highlight fragment fade-in">Separating outliers from mistakes often comes down to common sense</li> <ul class="highlight fragment fade-in"> <li>Can this valued be explained by a typo?</li> <li>Is there more information in the data we can use to confirm whether it's an error?</li> </ul> </ul> <p class="highlight fragment fade-in"> Finding and flagging this cases is more important than having a plan for how to deal with them </p>

When to validate data

## When to validate data <div class="highlight fragment fade-in"> As soon as you get your hands on a new data set! </div> <p> <ul class="fragment fade-in"> <li>If you are collecting primary data, do it often, ideally every day. Specially on the first few days of data collection.</li> <li>If you are receiving the data from a partner, do it after each data transfer</li> </ul> </p> Note: Discuss why we want to validate data in real time

Making corrections

## Making corrections Whether to make changes to values or not is a controversial point. <span class="fragment fade-in">But here are a few points where there is not as much controversy:</span> <div> <ul> <li class="fragment fade-in">If you do make corrections, make them through code and carefully document the process or reasoning that led you to find the corrected value.</li> <li class="fragment fade-in">You should check how sensitive your results are to data quality issues. This can be done by simulating noise or by testing what happens when you disregard observations with suspicious values.</li> </ul> </div> --- # Useful tools - Stata: `iecorrect` (from `iefieldkit`)

Exercise

## Making corrections Test some of the tools we listed before to answer the following questions: - Is the data set uniquely and fully identified? - Are there any issues with completeness? - Are there any inconsistencies in the data? - Are the potentially problematic outliers? <p class="highlight"> Can you write code that will flag issues <em>automatically</em>, rather than requiring interactive and manual inspections? </p>

## References - [How can we improve the quality of big data for development economics research? Experiences from traditional data collection can help!](https://blogs.worldbank.org/opendata/how-can-we-improve-quality-big-data-development-economics-research-experiences-traditional) - [Implementing data quality checks](https://worldbank.github.io/dime-data-handbook/processing.html#implementing-data-quality-checks)