Tidying data

Let's try something

Exercise 1

Write code to calculate the average age of children in the data using

data/raw/mother_survey.csv
data/tidy/child.csv

Write code to calculate the average age of mothers in the data using

data/raw/mother_long.csv
data/tidy/mother.rds
data/analysis/child.csv

Tidy data

Some semantics

Data can be acquired in all shapes and sizes
In our line of work, it is mostly stored as one or multiple data tables, like the ones we were just working on
Data tables organize information into rows and columns
This kind of data is often called tabular or rectangular data

Can you think of an example of non-tabular data?

Some semantics

A dataset is a collection of values
Every value belongs to a variable and an observation
A variable contains all values that measure the same underlying attribute (like height, temperature, duration) across units
An observation contains all values measured on the same unit (like a person, or a day, or a race) across attributes

## Tidy data

1.  Each **column** corresponds to one **variable**
                        2.  Each **row** corresponds to one **observation** 
                        3.  Each **data table** corresponds to one **unit of observation**

<div class="highlight">
                            <p class="fragment fade-in">
                                Every other format is "untidy"
                            </p>
                        </div>

---

## Tidy data

<ul>
                            <li class="fragment fade-in">Why is <code>mother_survey.csv</code> not tidy? <span class="fragment fade-in">Why is this data format useful?</span></li>
                            <li class="fragment fade-in">Why is <code>mother_long.csv</code> not tidy?</li>
                            <li class="fragment fade-in">Is <code>data/analysis/child.csv</code> tidy?</li>
                        </ul>

Tidying data

## Tidying data

How can we turn `mother_long.csv` into a tidy data set?

<ul>
                            <li class="fragment fade-in">By reshaping it so that variables are represented in columns, not rows</li>
                            <li class="fragment fade-in">
                                <span class="highlight">Reshaping</span> means transforming how values are organized 
                                into rows and columns in a data table in a way that the unit of observation represented by each row is changed</li>
                        </ul>

---

## Tidying data

How can we turn `mother_survey.csv` into a tidy data set?

<ol>
                            <li class="fragment fade-in">Create two separate data tables, one where the unit of observation is the mother, and one where it is the child</li>
                            <li class="fragment fade-in">Reshape the child-level data so that each observation is represented in a row, not in different columns</li>
                        </ol>

---

## Tidying data

Why are ID or key variables important when tidying data?

<ul>
                            <li class="fragment fade-in">Because they allow us to connect different data tables</li>
                        </ul>

---

## Some practical tips

- Name tidy data tables in a way that makes it easy to identify its unit of observation
                        - Keep documentation about the different units of observation in your data and how their respective data tables can be linked

Why tidy data?

## When to use tidy data?

<ul>
                            <li>As we've discussed, sometimes we can only perform the analysis we are interested in using untidy data</li>
                            <li class="fragment fade-in">Still, this format is extremely useful for other stages of the data work, including data cleaning and indicator construction</li>
                            <li class="fragment fade-in">If we are interested in analyzing data in multiple units of observations, 
                                we'll likely need analysis data sets that are as close to tidy as possible for each of these units</li>
                        </ul>

---
                        ## Why use tidy data?

<div class = "fragment fade-in">
                            <ul>
                                <li>Easier to change the format of a single column instead of multiple columns (in wide format)</li>
                                <li>Easier to handle data if all rows in each column have the same data type (in long format, that is oftern not the case)</li>
                                <li>Easier to calculate statistics when the unit of observation and unit of analysis are the same</li>
                                <li>Easier to create visualizations when each value into the data will be one data point in the graph</li>
                                <li>Easier to aggregate observations having tidy data as a starting point</li>
                            <ul>
                        </div>

Let's try something

Exercise 1

Tidy data

Some semantics

Some semantics

Tidying data

Why tidy data?

Questions?