02:30
Tidying and Data manipulation with tidyverse
Bending data with tidyverse to answer your questions
Masumbuko Semba
2024-03-07
The concept
- The concept is based on five itmes—plan, simulate, acquire, explore and share
Learning Agenda
- Get familiar with R and Rstudio
- Data structure and data types
- Reading and writing data in Rstudio
- Tidying and Data manipulation with tidyverse
- Plotting and Visualization
- Descriptive Statistics
- Inferential Statistics
- Modelling and simulation
- Spatial Handling and Analysis
Introduction
Introduction to Data Manipulation
- Manipulating vectors through reordering and subsetting
- Advanced analyses require the use of tidy data
- Data frames become the preferred data storage
Importance of Data Frames
- Data frames facilitate the organization of data
- R put more Emphasis on working with data frames
- Data frame combine more than one vector
- Each vector contain one data type
- Leading to tidy data
01:30
Tidy data
What is Tidy Data?
- Tidy data is a specific structure that simplifies data manipulation and analysis.
- We ought to learn the best practices,
- We also need to know the benefits of organizing our data in a tidy format
Tidy Data Characteristic
- Definition of Tidy Data: Each row represents one observation, and columns represent variables for each observation
- Rows represent single observations
- Columns represent single Variables
- Emphasis on clarity and structure in organizing data tables
Tidy data quotes
“Happy families are all alike; every unhappy family is unhappy in its own way.” —- Leo Tolstoy
“Tidy datasets are all alike, but every messy dataset is messy in its own way.” —- Hadley Wickham
Example of tidy data
species
island
year
sex
bill_length_mm
Adelie
Torgersen
2,007
male
39.1
Adelie
Torgersen
2,007
female
39.5
Adelie
Torgersen
2,007
female
40.3
Adelie
Torgersen
2,007
NA
NA
Adelie
Torgersen
2,007
female
36.7
Adelie
Torgersen
2,007
male
39.3
Adelie
Torgersen
2,007
female
38.9
Adelie
Torgersen
2,007
male
39.2
Adelie
Torgersen
2,007
NA
34.1
Adelie
Torgersen
2,007
NA
42.0
1–10 of 344 rows
Show
...
Benefits of Tidy Data
- Easier data cleaning and manipulation
- Efficient analysis
- Improved collaboration
- Clearer communication
The Tidyverse
Tidyverse: Your Tidy Data Toolkit
The Tidyverse …
Loading required package: tidyverseWarning: package 'ggplot2' was built under R version 4.3.1Warning: package 'purrr' was built under R version 4.3.1Warning: package 'dplyr' was built under R version 4.3.2Warning: package 'stringr' was built under R version 4.3.2Warning: package 'lubridate' was built under R version 4.3.1── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.1.4 ✔ readr 2.1.4
✔ forcats 1.0.0 ✔ stringr 1.5.1
✔ ggplot2 3.4.4 ✔ tibble 3.2.1
✔ lubridate 1.9.3 ✔ tidyr 1.3.0
✔ purrr 1.0.2 ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors02:30
Tidy
Tidy can mean neat and well-organized, clean and orderly in appearance, methodical and efficient, or sufficient and adequate depending on the context.
Tidy data
- Most statistical datasets are data frames made up of rows and columns.
- The columns represent variables
- The rows represent records or observations
- Each observation is among the common data types
- Each variable store one type of data type
Tidy data ….
- Reports indicate that 80% of data analysis time is spent on cleaning and preparing data.
- Organizing datasets for easier analysis and plotting.
- Data arrives with repeated observations in a single row.
- Memory-saving or inherent structure makes ‘wide’ format appealing.
- Convert between wide and long formats as needed (Semba and Peter 2020).
Tidy data ….
- Structuring data frames is a significant challenge in statistical analysis, visualization, and modeling.
- Tidy data implies organizing data consistently for easy analysis and visualization.
- Data preparation in R involves thoughtful consideration of the desired data frame structure.
- Clarifying the purpose of each row and column ensures consistent and clear data manipulation.
tidy data….
There are basically three principles that we can follow to make a tidy dataset.
- Each variable must have its own a column,
- Each observation must have its own row, and
- Each cell must have its own value.
Tip
The tidyr package is designed to structure to make data frame tidy
tidy data….
Many datasets that you receive are untidy and will require some work on your end. There are several reasons why a dataset is messy.
- Often times the people who created the dataset aren’t familiar with the principles of tidy data.
- Most datasets are messy is that data is often organized to facilitate something other than analysis.
- Data entry is perhaps the most common reason for organizing data in untidy format.
tidy data
We are going to look for the function that are regularly used to tidy the data frames. These inlude:
- Gathering from wide format
- Spreading from long format
- Separating columns
- Uniting columns
01:30
The Pipe operator: %>%
- The pipe operator
%>%allows to combine multiple operations in R into a single sequential chain of actions - The %>% operator, improve code readability and simplifying data manipulation and analysis workflows
01:30
Wide format
Wide format organize data where each observation is represented by a single row, and each variable has its own column.
- Suitable for small datasets with a limited number of variables.
- Easy to read and interpret.
- Commonly used for summary statistics and aggregate data.
Long format
Long format organize data where observations are represented by multiple rows and columns represent variabels.
- Suitable for large datasets with varying numbers of observations and variables.
- Facilitates easier data manipulation and analysis.
- Supports complex hierarchical structures.
Tidying and Data manipulation with tidyverse Bending data with tidyverse to answer your questions Masumbuko Semba 2024-03-07