---
title: "Session 3: Data Wrangling"
subtitle: "R for Stata Users"
author: "Luiza Andrade, Marc-Andrea Fiorina, Rob Marty, Maria Reyes Retana, Rony Rodriguez Ramirez, Luis Eduardo San Martin, Leonardo Viotti"
date: "The World Bank -- DIME | [WB Github](https://github.com/worldbank) <br> March 2024"
output:
  xaringan::moon_reader:
    css: ["libs/remark-css/default.css", 
          "libs/remark-css/metropolis.css", 
          "libs/remark-css/metropolis-fonts.css",
          "libs/remark-css/custom.css"]
    lib_dir: libs
    nature:
      ratio: "16:9"
      highlightStyle: github
      highlightLines: true
      countIncrementalSlides: false
---

```{r setup, include = FALSE}

# Load packages
library(knitr)
library(tidyverse)
library(hrbrthemes)
library(fontawesome)
library(here)
library(xaringanExtra)
library(countdown)

here::i_am("Presentations/03-data-wrangling.Rmd")

options(htmltools.dir.version = FALSE)
opts_chunk$set(
  fig.align = "center",
  fig.height = 4,
  dpi = 300,
  cache = T
  )

xaringanExtra::use_panelset()
xaringanExtra::use_webcam()
xaringanExtra::use_clipboard()
htmltools::tagList(
  xaringanExtra::use_clipboard(
    success_text = "<i class=\"fa fa-check\" style=\"color: #90BE6D\"></i>",
    error_text = "<i class=\"fa fa-times-circle\" style=\"color: #F94144\"></i>"
  ),
  rmarkdown::html_dependency_font_awesome()
)

xaringanExtra::use_logo(
  image_url = here("Presentations",
                   "img",
                   "lightbulb.png"),
  exclude_class = c("inverse", 
                    "hide_logo"),
  width = "50px"
)
```


# Table of contents

1. [Introduction](#intro)

2. [Exploring your data](#exploring)

3. [ID variables](#id)

4. [Wrangling your data](#wrangling)

4. [Create variables](#variables)

5. [Appending and marging](#bind)

6. [Saving a dataframe](#saving)

7. [Factor variables](#factor)

8. [Reshaping](#reshaping)


---
class: inverse, center, middle
name: intro

# Introduction

<html><div style='float:left'></div><hr color='#D38C28' size=1px width=1100px></html>

---

# Introduction

### Initial Setup

.panelset[

.panel[.panel-name[If You Created an RStudio Project in Session 2]
1. Go to the `dime-r-training-mar2024` folder that you created yesterday, and open the `dime-r-training-mar2024` R project that you created there.
]

.panel[.panel-name[If You Did Not Attend Session 2]

 1. Open RStudio.

 2. Type in the following lines, replacing “YOURFILEPATHHERE” (use forward slashes only: "/") with a file path where the file path where you will place this R project. 

```{r, eval=FALSE}
install.packages("usethis")
library(usethis)
usethis::use_course(
    "https://github.com/worldbank/dime-r-training/archive/main.zip",
    destdir = "YOURFILEPATHHERE"
)
```

 3\. In the console, type in the requisite number to delete the .zip file (we don't need it anymore)

 4\. A new RStudio environment will open. Use this for the session today. 
  
  
]

]

---

# Introduction

### Goals of this session

- To organize data in a manner that makes it easier to analyze and communicate.

--

### Things to keep in mind

- We'll take you through the same steps we've taken when we were preparing the datasets used in this course. 

- In most cases, your datasets won't be `tidy`. 

> **Tidy data**: A dataset is said to be tidy if it satisfies the following conditions: 

```{r, echo = FALSE, out.width="30%"}
knitr::include_graphics("img/tidy_data.png")
```



Therefore, messy data is any other arrangement of the data.

---

# Introduction

In this session, you'll be introduced to some basic concepts of data cleaning in R. We will cover: 

1. Exploring a dataset;
2. Creating new variables;
3. Filtering and subsetting datasets;
4. Merging datasets;
5. Dealing with factor variables;
6. Saving data.

> There are many other tasks that we usually perform as part of data cleaning that are beyond the scope of this session.


### Before we start, let's make sure we are ready: 

1. Start a fresh RStudio session.
2. Open the RStudio project you created yesterday.
3. Create a new R Script called `exercises-session3.R`

---

# Introduction: Packages

### Another important aspect to consider is R packages. Consider the following: 

.pull-left[

#### R is a new phone

```{r, echo = FALSE, out.width="30%"}
knitr::include_graphics("https://images-na.ssl-images-amazon.com/images/I/81F-QC1N5WL._AC_SY550_.jpg")
```
]

.pull-right[

#### R packages are apps on your phone

```{r, echo = FALSE, out.width="80%"}
knitr::include_graphics("https://www.apple.com/v/app-store/a/images/meta/og__c59t0pflacq6.png?202009232211")
```
]

---

# RECAP: Packages

### To install a package you can run the following command:

```{r, echo = TRUE, eval = FALSE, include = TRUE}

# To install
install.packages("dplyr")

```

* Unlike Stata, R packages need to be loaded in each R session that will use them. 
* That means that, for example, a function that comes from the `dplyr` package cannot be used if the package has not been installed and loaded first. 

### To load a package you can run the following command:

```{r, echo = TRUE, eval = FALSE, include = TRUE}

# To load
library(dplyr)
```

---

# RECAP: loading packages

```{r tverse, cache = FALSE, message = FALSE}
# If you haven't installed the packages uncomment the next line
# install.packages("tidyverse")
# install.packages("here")
# install.packages("janitor")
library(tidyverse)  # To wrangle data
library(here)       # A package to work with relative file paths
library(janitor)    # Additional data cleaning tools
```

> *Notes*: Remember you should always load your packages before your start coding.

---

# RECAP: File paths

The `here` package allows you to interact with your working directory. It will look for the closest R Project and set its location as the working directory. That's why it is important to set your RStudio project correctly. 

.pull-left[

The goal of this package is to:

- Easily reference your files in project-oriented workflows. 

Using `here`:

- Load the library.
- Use `here()` for relative file paths.

```{r, eval=FALSE}
  path <- here("data", "raw", "data-file.csv")
  df <- read.csv(path)
```
]



.pull-right[
```{r here, echo = FALSE, out.width = "85%"}
knitr::include_graphics("https://raw.githubusercontent.com/allisonhorst/stats-illustrations/main/rstats-artwork/here.png")
```
]

---

# RECAP: Loading a dataset in R

Before we start wrangling our data, let's read our dataset. In R, we can use the `read.csv` function from Base R, or `read_csv` from the `readr` package if we want to load a CSV file. For this exercise, we are going to use the World Happiness Report (2015-2018)

.exercise[

**Exercise 1:** Loading data using the `here` package:

Use either of the functions mentioned above and load the three WHR datasets from the `DataWork/DataSets/Raw/Un WHR` folder. Use the following notation for each dataset: `whrYY`, e.g. `WHR2015.csv` becomes the `whr15` dataset. 

]

`r countdown(minutes = 0, seconds = 45, left = 0, font_size = "2em")`

--

.solution[

**Solution:**

```{r readcsv, message = FALSE}
whr15 <- read_csv(here("DataWork", "DataSets", "Raw", "Un WHR", "WHR2015.csv")) 
whr16 <- read_csv(here("DataWork", "DataSets", "Raw", "Un WHR", "WHR2016.csv")) 
whr17 <- read_csv(here("DataWork", "DataSets", "Raw", "Un WHR", "WHR2017.csv"))
```
]

---

# The pipe %>% (or |>) operator

* "Piping" in R can be seen as "**chaining**." This means that we are invoking multiple method calls. 
* Every time you have invoked a method (a function) this return an object that then is going to be used in the next pipe.

.pull-left[

```{r, eval = FALSE}
rony %>% 
  wake_up(time = "5:30") %>% 
  get_out_of_bed() %>% 
  do_exercise() %>% 
  shower() %>% 
  get_dressed() %>% 
  eat(meal = "breakfast", coffee = TRUE) %>% 
  brush_teeth() %>%
  work(effort = "mininum")
```
]

.pull-right[

```{r, eval = FALSE}
  work(
    brush_teeth(
      eat(
        get_dressed(
          shower(
            do_exercise(
              get_out_of_bed(
                wake_up(rony, time = "5:30")
              ), 
            )
          )
        ), meal = "breakfast", coffee = TRUE
      )
    ), effort = "minimum"
  )

```

]

---

# The pipe %>% operator

From R for Data Science by Wickham & Grolemund:

.medium[

> Pipes are a powerful tool for clearly expressing a sequence of multiple operations. The point of the pipe is to help you write code in a way that is easier to read and understand. [...]  It focusses on verbs, not nouns. You can read this series of function compositions like it’s a set of imperative actions.

]

--

#### (only for 🤓 nerds:)

* The `%>%` pipe is part of the `magrittr` package. `R v4.1.0` adds a *native pipe* via `|>`. you could use it like

```{r, eval=FALSE}
whr15|> mean(variable, na.rm = T)
```

---
# Janitor package: The clean_names() function

The `clean_names()` function helps us when our variables names are pretty bad. For example, if we have a variable that is called **`GDP_per_CApita_2015`**, the `clean_names()` function will help us fix those messy names. 

> **Tip**: Pipe the `clean_names()` function after you load a dataset.

```{r clean_names, message = FALSE}
whr15 <- whr15 %>% 
  clean_names() #<<
whr16 <- whr16 %>% 
  clean_names()
whr17 <- whr17 %>% 
  clean_names()
```

However, if we want to to rename our variable manually, we could use: 

```{r manual_names, eval=FALSE}

whr15 <- whr15 %>% 
  rename( #<<
    var_newname = var_oldname #<<
  ) #<<

```

---
class: inverse, center, middle
name: exploring

# Exploring your data

<html><div style='float:left'></div><hr color='#D38C28' size=1px width=1100px></html>

---

# Exploring a data set

These are some useful functions from base R: 

* `View()`: open the data set.
* `class()`: reports object type of type of data stored.
* `dim()`: reports the size of each one of an object's dimension.
* `names()`: returns the variable names of a dataset.
* `str()`: general information on an R object.
* `summary()`: summary information about the variables in a data frame.
* `head()`: shows the first few observations in the dataset.
* `tail()`: shows the last few observations in the dataset.

Some other useful functions from the tidyverse:

* `glimpse()`: get a glimpse of your data.

---

# Load and show a dataset

We can just show our dataset using the name of the object; in this case, `whr15`. 

```{r show}
whr15
```

---

# Glimpse your data

Use `glimpse()` to get information about your variables (e.g., type, row, columns,)

```{r glimpse}
whr15 %>% 
  glimpse() #<<

```

---
class: inverse, center, middle
name: id

# Dimensions of your data

<html><div style='float:left'></div><hr color='#D38C28' size=1px width=1100px></html>

---

# Dimensions of your data

Let's see first how many columns and observations the dataset has: 

* **Dimensions of your data** (Rows and Columns): 

```{r}
dim(whr15) 

```

* **The number of distinct values of a particular variable**:

```{r eval = FALSE}

n_distinct(DATASET$variable, na.rm = TRUE)

```

--

The `$` sign is a subsetting operator. In R, we have three subsetting operators (`[[`, `[`, and `$`.). It is often used to access variables in a dataframe. 

The `n_distinct` function allows us to count the number of unique values of a variable length of a vector. We included `na.rm = TRUE`, so we don't count missing values.

---

# Dimensions of your data

.exercise[

**Exercise 2**: Identify distinct values of a variable in a dataset. Using the `n_distinct` function, can you tell how many unique values these variables in the `whr15` dataset have?

1. Country
2. Region
]

`r countdown(minutes = 1, seconds = 0, left = 0, font_size = "2em")`

--

.solution[
**Solution**: 
```{r}
n_distinct(whr15$country, na.rm = TRUE)
n_distinct(whr15$region, na.rm = TRUE)
```
]


---

# Dimensions of your data

We can also test whether the number of rows is equal to the number of distinct values in a specific variable as follows:


```{r}
nrow(whr15)
```

--

We can use the two functions (`nrow` and `n_distinct`) together to test if their result is the same.

```{r}
n_distinct(whr15$country, na.rm = TRUE) == nrow(whr15)

n_distinct(whr16$country, na.rm = TRUE) == nrow(whr16)

n_distinct(whr17$country, na.rm = TRUE) == nrow(whr17)
```

---

class: inverse, center, middle
name: wrangling

# Wrangling your data

<html><div style='float:left'></div><hr color='#D38C28' size=1px width=1100px></html>

---

# Wrangling vs Cleaning

.large[

**Cleaning:**

- Detecting and addressing inconsistencies in a dataset. Removing erroneous data from your data.

**Wrangling:**

- Translating raw data into a more useful form. Unifying messy and complex data.

]

---

# Wrangling your data

For data wrangling you will frequently use one tidyverse package called dplyr. 

* [`dplyr`](https://dplyr.tidyverse.org) is part of the [`tidyverse`](https://www.tidyverse.org) package family.

* You are ***highly encouraged*** to read through [Hadley Wickham's chapter](https://r4ds.had.co.nz/transform.html). It's clear and concise.

--

* Also check out this great "cheatsheet" [here](https://github.com/rstudio/cheatsheets/blob/master/data-transformation.pdf).

--


* The package is organized around a set of **verbs**, i.e. *actions* to be taken.

* We operate on `data.frames` or `tibbles` (*nicer looking* data.frames.)

--

* All *verbs* work as follows:

$$\text{verb}(\underbrace{\text{data.frame}}_{\text{1st argument}}, \underbrace{\text{what to do}}_\text{2nd argument})$$

--

* Alternatively you can (should) use the `pipe` operator `%>%`:

$$\underbrace{\text{data.frame}}_{\text{1st argument}} \underbrace{\text{ %>% }}_{\text{"pipe" operator}} \text{verb}(\underbrace{\text{what to do}}_\text{2nd argument})$$


---

# dplyr::filter

- The `filter` function is used to subset rows in a dataset. 

```{r filter}
whr15 %>% filter(region == "Western Europe")

```

---

# dplyr::filter

.exercise[
**Exercise 3:** Use `filter()` to extract rows in these regions: (1) Eastern Asia and (2) North America. Hint: use the **or** operator (`|`): 

]

`r countdown(minutes = 1, seconds = 0, left = 0, font_size = "2em")`

--

.solution[

**Solution:**

```{r, eval = FALSE}
whr15 %>% 
  filter(region == "Eastern Asia" | region == "North America") 
```

A more elegant approach would be to use the `%in%` operator (equivalent to `inlist()` in Stata): 

```{r, eval = FALSE}
whr15 %>% 
  filter(region %in% c("Eastern Asia", "North America")) 
```

]

---

# dplyr::filter

```{r echo=FALSE}
knitr::include_app("https://rrmaximiliano.shinyapps.io/learnr-app/")
```

---

# dplyr::filter missing cases

If you want to remove (or identify) missing cases for a specific variable, you can use `is.na()`. 

* This function returns a value of true and false for each value in a data set.
* If the value is *NA* the `is.na()` function returns TRUE, otherwise, it returns FALSE.
* In this way, we can check NA values that can be used for other functions.
* We can also negate the function using `!is.na()` which indicates that we want to return those observations with no missing values in a variable.

The function syntax in a pipeline is as follows:

```{r eval = FALSE}
DATA %>% 
  filter(
    is.na(VAR) #<<
  )
```

**What are we returning here?**

--

The observations with missing values for the variable VAR.

---

# dplyr::filter missing cases

Let's try filtering the `whr15` data. Let's keep those observations that have information per region, i.e., no missing values.

```{r missings}
whr15 %>% 
  filter(!is.na(region)) %>% #<<
  head(5)

```

> Notice that we are negating the function, i.e., `!is.na()`
<br> In case we want to keep the observations that contains missing information we will only use `is.na()`. 

---

# dplyr::filter  missing cases

```{r echo=FALSE}
knitr::include_app("https://rrmaximiliano.shinyapps.io/learnr-app/")
```

---
class: inverse, center, middle
name: variables

# Creating new variables

<html><div style='float:left'></div><hr color='#D38C28' size=1px width=1100px></html>

---

# Creating new variables

We use `dplyr::mutate()` to create new variables. For example:

```{r mut, eval=FALSE}
whr15 %>%
  mutate(#<<
    hap_hle = happiness_score * health_life_expectancy  #<<
  ) #<<
```

This will add a new variable called `hap_hle` which is the interaction of happiness score and health life expectancy.

---

# Creating new variables: Dummy variables

```{r include = TRUE, eval = FALSE}
whr15 %>%
  mutate(happiness_score_6 = (happiness_score > 6))
```

**What do you think is happening to this variable?**

--

This new variables contains either `TRUE` or `FALSE`. To have it as a numeric variable (1 or 0, respectively), we include the `as.numeric()` function. 

--

```{r include = TRUE, eval = FALSE}
whr15 %>%
  mutate(happiness_score_6 = as.numeric((happiness_score > 6)))
```

--

Finally, instead of using a random number, such as 6, we can do the following: 

```{r include = TRUE, eval = FALSE}
whr15 %>%
  mutate(
    happiness_high_mean = as.numeric((happiness_score > mean(happiness_score, na.rm = TRUE)))
  ) 
```

---

# Creating variables by groups

In R, we can use `dplyr::group_by()` before we mutate to group an estimation. For example, we are going to pipe the following functions:

1. Group our data by the `region` variable.
2. Create a variable that would be the mean of `happiness_score` by each region.
3. Select the variables `country, region, happiness_score, mean_hap`. 

.panelset[
.panel[.panel-name[Example]
```{r eval = FALSE}
DATASET %>% 
  group_by(GROUPING VARIABLE) %>% #<<
  mutate(
    NAME OF NEW VAR = mean(VARIABLE, na.rm = TRUE)
  ) %>% 
  select(VAR1, VAR2, VAR3, VAR4) 
```

]

.panel[.panel-name[With variables]
```{r eval = FALSE}
whr15 %>% 
  group_by(region) %>%
  mutate(
    mean_hap = mean(happiness_score, na.rm = TRUE)
  ) %>% 
  select(country, region, happiness_score, mean_hap) 
```
]

.panel[.panel-name[Output]
```{r echo = FALSE}
whr15 %>% 
  group_by(region) %>%
  mutate(
    mean_hap = mean(happiness_score, na.rm = TRUE)
  ) %>% 
  select(country, region, happiness_score, mean_hap) %>% 
  head(7)
```
]
]

---

# Creating multiple variables at the same time

We can create multiple variables in an easy way. Let's imagine that we want to estimate the mean value for the variables: `happiness_score`, `health_life_expectancy`, and `trust_government_corruption` .

#### How we can do it? 

* We can use the function `across()`. <br> Syntax: `across(VARS that you want to transform, FUNCTION to execute)`. 
* `across()` should be always use inside `summarise()` or `mutate()`.

.panelset[
.panel[.panel-name[Across]

```{r across, eval = FALSE}
vars <- c("happiness_score", "health_life_expectancy", "trust_government_corruption")

whr15 %>%
  group_by(region) %>%
  summarize(
    across( #<<
      all_of(vars), mean #<<
    ) #<<
  ) %>%
  head(3)

```

]

.panel[.panel-name[Output]

```{r across2, echo = FALSE}
vars <- c("happiness_score", "health_life_expectancy", "trust_government_corruption")

whr15 %>%
  group_by(region) %>%
  summarize(
    across( #<<
      all_of(vars), mean #<<
    ) #<<
  ) %>% 
  head(3)

```
]
]

---

# Creating variables

.exercise[

**Exercise 4:** Create a variable called `year` that equals to the year of each dataframe using the `mutate()`. Remember to assign it to the same dataframe.

`r countdown(minutes = 1, seconds = 0, top = 0, font_size = "2em")`

]

--

.solution[

**Solution:**

```{r}
whr15 <- whr15 %>% 
  mutate(year = 2015)

whr16 <- whr16 %>% 
  mutate(year = 2016)

whr17 <- whr17 %>% 
  mutate(year = 2017)
```

]

---

# Creating variables

```{r echo=FALSE}
knitr::include_app("https://rrmaximiliano.shinyapps.io/learnr-app/")
```

---
class: inverse, center, middle
name: bind

# Appending and merging data sets

<html><div style='float:left'></div><hr color='#D38C28' size=1px width=1100px></html>

---

# Appending dataframes

Now that we can identify the observations, we can combine the data set. Here are two functions to append objects by row

```{r eval = FALSE}
rbind(df1, df2, df3) # The base R function

bind_rows(df1, df2, df3) # The dplyr function, making some improvements to base R
```
.exercise[
**Exercise 5:** Append data sets. Use the function `bind_rows` to append the three WHR datasets:

`r countdown(minutes = 0, seconds = 45, bottom = 0, font_size = "2em")`

]

--

.solution[

**Solution:**
```{r bind_rows, eval = FALSE}

bind_rows(whr15, whr16, whr17)

```

]

--

.notes[

**Notes**
* One of the problems with binding rows like this is that, sometimes, some columns are not *fully* compatible.

]

---

# Appending dataframes

```{r echo=FALSE}
knitr::include_app("https://rrmaximiliano.shinyapps.io/learnr-app/")
```

---

# Appending and merging data sets

.exercise[

.medium[

**Exercise 6:** Fixing our variables and appending the data frames correctly.

]

*Exercise 6a*

* Load the R data set `regions.RDS` from `DataWork/DataSets/Raw/Un WHR` using the `read_rds` function.

`r countdown(minutes = 0, seconds = 45, left = 0, font_size = "2em")`

]

--

.solution[

**Solution:**

```{r regions}
regions <- read_rds(here("DataWork", "DataSets", "Raw", "Un WHR", "regions.RDS"))

```

]

---

# Appending and merging data sets

We can use the `dplyr::left_join()` function to merge two dataframes. The function syntax is: `left_join(a_df, another_df, by = c("id_col1"))`. 

> A left join takes all the values from the first table, and looks for matches in the second table. If it finds a match, it adds the data from the second table; if not, it adds missing values. It is the equivalent of `merge, keep(master matched)` in Stata.

```{r, echo = FALSE, out.width="70%"}
knitr::include_graphics("img/left_join.png")
```

---

# Appending and merging data sets

.exercise[


**Exercise 6b:** Join the dataframes: `regions` and `whr17`. 

]

`r countdown(minutes = 1, seconds = 0, left = 0, font_size = "2em")`

--

.pull-left[

.solution[

**Solution:** 

```{r}
whr17 <- whr17 %>% #<<
  left_join(regions, by = "country") %>% #<<
  select(country, region, everything()) 
```

]

]

.pull-right[

.notes[

**Notes:**

Look at the `everything()` function. It takes all the variables from the dataframe and put them after country and region. In this way, select can be use to **order** columns! 

]

]

---

# Appending and merging data sets

.exercise[

**Exercise 6c:** Check if there is any other countries in `whr17` without region info:

* Only use pipes %>%
* And `filter()`
* Do not assign it to an object.


]

`r countdown(minutes = 1, seconds = 0, top = 0, font_size = "2em")`


--

.solution[

**Solution:**

```{r}
whr17 %>% 
  filter(is.na(region))
  
```

]

---

# So we ended up with two countries with NAs

This is due to the name of the countries. The regions dataset doesn't have "Taiwan Province of China" nor "Hong Kong S.A.R., China" but "Taiwan" and "Hong Kong."

#### How do you think we should solve this? 

--

* My approach would be to: 

1. fix the names of these countries in the `whr17` dataset (a data cleaning task) and; 
2. merge (left_join) it with the regions dataset.


Appendix: [case_when and mutate](#case_when) for more information.

---

# Appending and merging data sets

Finally, let's keep those relevant variables first and bind those rows. 

.exercise[

**Exercise 7:** Bind all rows and create a panel called: `whr_panel`. 

* Select the variables: `country`, `region`, `year`, `happiness_rank`, `happiness_score`, `economy_gdp_per_capita`, `health_life_expectancy`, `freedom` for each df, i.e., `whr15`, `whr16`, `whr17`. 
* Use `rbind()`

`r countdown(minutes = 1, seconds = 0, top = 0, font_size = "2em")`

]

--

.solution[

**Solution:**

```{r}
vars_to_keep <- c("country", "region", "year", "happiness_rank", 
              "happiness_score", "economy_gdp_per_capita", 
              "health_life_expectancy", "freedom")

whr15 <- select(whr15, all_of(vars_to_keep))
whr16 <- select(whr16, all_of(vars_to_keep))
whr17 <- select(whr17, all_of(vars_to_keep))

whr_panel <- rbind(whr15, whr16, whr17)    # or bind_rows
```

]

---

# Appending and merging data sets

There are other types of joins in the `dplyr` package. We won't get into detail, but here are some examples. 

```{r, echo = FALSE, out.width="80%"}
knitr::include_graphics("img/join_types.png")
```

You can also check [this chapter](https://r4ds.hadley.nz/joins.html), which is very clear. 


---

class: inverse, center, middle
name: saving

# Saving a dataset

<html><div style='float:left'></div><hr color='#D38C28' size=1px width=1100px></html>

---

# Saving a dataset

* The dataset you have is the same data set we’ve been using for earlier sessions, so we can save it now.
* To save a dataset we can use the `write_csv` function from the `readr` package, or `write.csv` from base R. 

The function takes the following syntax: 

`write_csv(x, file, append = FALSE, row.names = FALSE, na = "")`: 

* `x:` the object (usually a data frame) you want to export to CSV
* `file:` the file path to where you want to save it, including the file name and the format (“.csv”)

---

# Saving a dataset

.exercise[

**Exercise 8:** Save the dataset as csv format in the "Final" folder with the name `whr_panel_**YOUR INITIALS**.csv`

* Use `write_csv()`
* Use `here()`

`r countdown(minutes = 1, seconds = 0, top = 0, font_size = "2em")`

]

--

.solution[

**Solution:**

```{r}
write_csv(
  whr_panel, here("DataWork", "DataSets", "Final", "whr_panel_MA.csv")
)

```

]



.notes[

* The problem with CSVs is that they cannot differentiate between `strings` and `factors`
* They also don’t save factor orders
* Data attributes (which are beyond the scope of this training, but also useful to document data sets) are also lost.

]

---

# Saving a dataset

The R equivalent of a `.dta` file is a `.rds` file. It can be saved and loaded using the following commands:

* `write_rds(object, file = "")`: Writes a single R object to a file.

* `read_rds(file)`: Load a single R object from a file. 


```{r}
# Save the data set

write_rds(
  whr_panel, 
  here("DataWork", "DataSets", "Final", "whr_panel_MA.Rds")
)

```

---
class: inverse, center, middle

# And that's it for this session. Join us tomorrow!! 

---
class: inverse, center, middle
name: appendix

# Appendix

<html><div style='float:left'></div><hr color='#D38C28' size=1px width=1100px></html>

---

# Missing values in R


#### Quick Note: 

.pull-left[
* Missings in R are treated differently than in Stata. They are represented by the NA symbol.
* Impossible values are represented by the symbol NaN which means 'not a number.' 
* R uses the same symbol for character and numeric data.
]

.pull-right[
* NA is not a string or a numeric value, but an indicator of missingness.
* NAs are contagious. This means that if you compare a number with NAs you will get NAs. 
* Therefore, always remember the `na.rm = TRUE` argument if needed.
]


---

# Other relevant functions: slice, subset, select

.panelset[
.panel[.panel-name[Arrange]

`Arrange`: allows you to order by a specific column.

```{r arrange}
whr15 %>% 
  arrange(region, country) %>% 
  head(5)

```

]

.panel[.panel-name[Slice]

`Slice`: allows you to select, remove, and duplicate rows. 

```{r slice}
whr15 %>% 
  slice(1:5) # to select the first 5 rows

```

You can also use `slice_head` and `slice_tail` to select the first or last rows respectively. Or `slice_sample` to randomly draw n rows.

]

.panel[.panel-name[Select]

`Select`: allows you to select specific columns.

```{r select}
whr15 %>% 
  select(region, country, happiness_rank)
```

]

.panel[.panel-name[Combining functions]

`Select`: allows you to specific columns.

```{r mix}
whr15 %>% 
  arrange(region, country) %>%                        # Sort by region and country
  filter(!is.na(region)) %>%                          # Filter those non-missing obs for region if any
  select(country, region, starts_with("happin")) %>%  # Select country, year, and vars that stars with happin
  slice_head()                                        # Get the first row

```

]

]


---

# Using ifelse when creating a variable

We can also create a dummy variable with the `ifelse()` function. The way we use this function is as: `ifelse(test, yes, no)`. We can also use another function called `case_when()`.

```{r}
whr15 %>% 
  mutate(
    latin_america_car = ifelse(region == "Latin America and Caribbean", 1, 0)
  ) %>% 
  arrange(-latin_america_car) %>% 
  head(5)
```

---
name: case_when

# Using case_when() to update a variable

Recall the problem we have with regions in the `whr17` data. We can fix it as follows:

```{r, eval=FALSE}
whr17 <- whr17 %>% 
  mutate(
    country = case_when(
      country == "Hong Kong S.A.R., China" ~ "Hong Kong", 
      country == "Taiwan Province of China" ~ "Taiwan",
      TRUE ~ country
    )
  )

whr17 %>%
  left_join(regions, by = "country") %>% 
  rename(region = region.y) %>% 
  select(-region.x) %>% 
  select(country, region, everything()) %>% 
  filter(is.na(region))
```
---
class: inverse, center, middle
name: factor

# Factor variables

<html><div style='float:left'></div><hr color='#D38C28' size=1px width=1100px></html>

---

# Factor variables

* When we imported this data set, we told R explicitly to not read strings as factor.
* We did that because we knew that we’d have to fix the country names.
* The region variable, however, should be a factor.

```{r}
str(whr_panel$region)
```

---

# Factor variables

To create a factor variable, we use the `factor()` function (or `as_factor()` from the `forcats` package).

* `factor(x, levels, labels)` : turns numeric or string vector `x` into a factor vector.
* `levels`: a vector containing the possible values of `x`.
* `labels`: a vector of strings containing the labels you want to apply to your factor variable
* `ordered`: logical flag to determine if the levels should be regarded as ordered (in the order given).

--

If your categorical variable does not need to be ordered, and your string variable already has the label you want, making the conversion is quite easy.


---

# Factor variables

.exercise[

**Extra exercise:** Turn a string variable into a factor.

* Use the mutate function to create a variable called region_cat containing a categorical version of the region variable.
* TIP: to do this, you only need the first argument of the factor function.

]

--

.solution[

**Solution:**

```{r}
whr_panel <- mutate(whr_panel, region_cat = factor(region)) 
```

]

--

And now we can check the class of our variable.

```{r}
class(whr_panel$region_cat)
```

---
class: inverse, center, middle
name: reshaping

# Reshaping a dataset

<html><div style='float:left'></div><hr color='#D38C28' size=1px width=1100px></html>

---

# Reshaping a dataset

Finally, let's try to reshape our dataset using the tidyverse functions. No more `reshape` from Stata. We can use `pivot_wider` or `pivot_longer`. Let's assign our wide format panel to an object called whr_panel_wide. 

.panelset[
.panel[.panel-name[Long to Wide]

```{r echo = FALSE, include = FALSE, message = FALSE, eval = TRUE}
whr_panel_wide <- whr_panel %>% 
  select(country, region, year, happiness_score) %>% 
  pivot_wider(
    names_from = year, 
    values_from = happiness_score
  ) 
```


```{r}
whr_panel %>% 
  select(country, region, year, happiness_score) %>% 
  pivot_wider(
    names_from = year, 
    values_from = happiness_score
  ) %>% 
  head(3)
```
]

.panel[.panel-name[Wide to Long]
```{r echo = FALSE, include = FALSE, message = FALSE, eval = TRUE}
whr_panel_long <- whr_panel_wide %>% 
  pivot_longer(
    cols = `2015`:`2017`,  
    names_to = "year",
    values_to = "happiness_score" 
  )

```

```{r}
whr_panel_wide %>% 
  pivot_longer(
    cols = `2015`:`2017`,  
    names_to = "year",
    values_to = "happiness_score" 
  ) %>% 
  head(3)
```
] 
]

---
class: inverse, center, middle

# Thank you!!

---
exclude: true

```{R, pdfs, include = F, eval = F}
pagedown::chrome_print("03-data-wrangling.html", output = "03-data-wrangling.pdf")

# Or
source("https://git.io/xaringan2pdf")
xaringan_to_pdf("Presentations/03-data-wrangling.html")
```
