• pandas is the main library for data manipulation in Python 🐼
• We will use it a lot in this course (and in your life as a data scientist!)
• It is built on top of numpy and matplotlib, and has a gazillion functions to work with data 😁
• If you use R already, think about it as the dplyr of Python
  • A list of equivalences between dplyr and pandas
• We will learn more about it in the next classes

• The apply function is used to apply a function to a dataset
  • (This course is full of surprises, isn’t it? 😄)
• It is a method of a pandas DataFrame
• It can be used with built-in functions, custom functions, or lambda functions
  • df.apply(function)
• You can apply functions to rows or columns
  • df.apply(function, axis=0) applies the function to each column (default)
  • df.apply(function, axis=1) applies the function to each row

• Let’s do a quick exercise

# Create an empty DataFrame
data = pd.DataFrame()

# Add variables
data["age"] = [18,29,15,32,6]
data["num_underage_siblings"] = [0,0,1,1,0]
data["num_adult_siblings"] = [1,0,0,1,0]

display(data)

• Now let’s define some functions

# The first two functions return True/False depending on age constraints
# The third function returns the sum of two numbers
# The fourth function returns a string with the age bracket

fn_iseligible_vote = lambda age: age >= 18

fn_istwenties = lambda age: (age >= 20) & (age < 30)

fn_sum = lambda x,y: x + y

def fn_agebracket(age):
    if (age >= 18):
        status = "Adult"
    elif (age >= 10) & (age < 18):
        status = "Adolescent"
    else:
        status = "Child"
    return(status)

• Now let’s apply the functions to the data["age"] column

data["can_vote"]    = data["age"].apply(fn_iseligible_vote)
data["in_twenties"] = data["age"].apply(fn_istwenties)
data["age_bracket"] = data["age"].apply(fn_agebracket)

display(data)

• You can also create a new variable using the apply function

# Creating a new variable
data["new_var"] = data["age"].apply(lambda age: age >= 18)

display(data)

• You can also delete a variable using the drop function

data = data.drop(columns = ["new_var"])

display(data)

• The map function is used to apply a function to a list, an array, or a series
  • A series is a single column of a pandas DataFrame
• In pandas, map works very similarly to the apply function, and they are interchangeable when working with series
• map can be faster than apply for simple functions, but apply is more flexible as it can be used with DataFrames (many columns)
• However, if you are using regular lists (e.g., list01 = [1,2,3]), you should use map instead of apply
  • apply is not a built-in Python function

data["age_bracket01"] = data["age"].map(fn_agebracket)

display(data[["age","age_bracket01"]])

data["age_bracket02"] = data["age"].apply(fn_agebracket)

display(data[["age","age_bracket02"]])

• Write a lambda function checking whether num_siblings \(\ge 1\)
• Add a variable to the dataset called has_siblings
• Assign True/False to this variable using apply()
• Appendix 02

• First, we will load the packages we need

import pandas as pd
import numpy as np

• Then, we will load the dataset

circuits = pd.read_csv("data_raw/circuits.csv")

# Or download it from the internet
# circuits = pd.read_csv("https://raw.githubusercontent.com/danilofreire/qtm151-summer/main/lectures/lecture-08/data_raw/circuits.csv")
display(circuits.head(2))

• The dataset contains information about F1 circuits, such as its name, location, latitude, longitude, and more
• You can find more information about the dataset here
• The dataset is available in the course’s GitHub repository here
  • Or you can download it using the command above
• Let’s see how the codebook looks like
• More information about Formula 1 here

• We can use the function query() to reference subattributes of a variable
  • query() is a method of a pandas DataFrame and it has many useful functions!
  • We will use it more in the future!
• Here we will combine query() with pd.unique() to extract a list of non-numeric values
• The pd.unique() function will return a list of unique values in a variable

# Extract a list of non-numeric values
# The pd.unique() function extracts unique values from a list
# Check each value in the alt column to see if it is not numeric
# True if it is not numeric, False if it is numeric
subset = circuits.query("alt.str.isnumeric() == False")
list_unique = pd.unique(subset["alt"])
print(list_unique)

['\\N' '-7']

• The replace function is used to replace values in a variable
• The syntax is dataframe["variable"].replace(list_old, list_new)
• More information about the function can be found here

# "list_old" encodes values we want to change
# From the list_unique, we see that the values we want to change are '\N' and '-7'
# "list_new" encodes the values that will replace the old
list_old = ['\\N','-7']
list_new = [np.nan, -7]

# This command replaces the values of the "alt" column
circuits["alt"] = circuits["alt"].replace(list_old, list_new)

• After the cleaning process is done, you may want to store the dataset again
• It’s strongly recommended to do this in a separate file from the original
• Use to_csv() to save the dataset as a .csv file

circuits.to_csv("data_clean/circuits_clean.csv", index=False)

• Use .replace() with the “country” column
• Replace “UK” with “United Kingdom”
• Appendix 04

• What is the column type of “lat” or “lng”?
• Does it have any string variables?
• Can we use str.isnumeric() here?
• Appendix 05

• Imagine that we want to recode the alt variable into an interval

\[x_{bin} = \begin{cases} "A" &\text{ if } x_1 < x \le x_2 \\ "B" &\text{ if } x_2 < x \le x_3 \end{cases} \]

• We can use the pd.cut() function to do this
• The syntax is df["new_variable"] = pd.cut(df["variable"], bins = [x1, x2, x3], labels = ["A", "B"])
• Where bins are the intervals and labels are the new values
• More information about the function can be found here

# Recode the "alt" variable into an interval
bins_x = [0, 2500, 5000]
labels_x = ["Between 0 and 2500",
            "Between 2500 and 5000"]

circuits["bins_alt"] = pd.cut(circuits["alt_numeric"],
                              bins = bins_x,
                              right = True,
                              labels = labels_x)
np.random.seed(2014)
display(circuits.sample(5))

def modify_x():
    global x
    x = x + 5

x = 1

# Now, running the function 
# will permanently increase x by 5.

modify_x()
print(x)
modify_x()
print(x)

6
11

def fn_square(x):
    global y
    y = x**2
    return(y)

x = 5
y = -5

print("Example 1:")
print(fn_square(x = 10))
print(x)
print(y)

Example 1:
100
5
100

Back to exercise 01

• Write a lambda function checking whether num_siblings \(\ge 1\)
• Add a variable to the dataset called has_siblings
• Assign True/False to this variable using apply()

fn_has_siblings = lambda num_siblings: num_siblings >= 1

data["has_siblings"] = data["num_adult_siblings"].apply(fn_has_siblings)

display(data[["num_adult_siblings","has_siblings"]])

Back to exercise 02

• Read the car dataset data_raw/features.csv
• Create a function that tests whether mpg \(\ge\) 29
• Add a variable mpg_above_29 which is True/False if mpg \(\ge\) 29
• Store the new dataset to data_clean/features.csv

data_raw = pd.read_csv("data_raw/features.csv")

data_raw["mpg_above_29"] = data_raw["mpg"].apply(lambda mpg: mpg >= 29)

display(data_raw[["mpg","mpg_above_29"]])

data_raw.to_csv("data_clean/features.csv", index = False)

Back to exercise 03

• Use .replace() with the “country” column
• Replace “UK” with “United Kingdom”

# Replace "UK" with "United Kingdom"
circuits["country"] = circuits["country"].replace("UK", "United Kingdom")
display(circuits[["country"]])

Back to exercise 04

# Check the data type of the "lat" column
print(circuits["lat"].dtype)

# Check if the variable "lat" is numeric
# print(circuits["lat"].str.isnumeric())
# It will raise an error because "lat" is a float

# Check if the variable "lng" is numeric
# print(circuits["lng"].str.isnumeric())
# It will also raise an error

float64

Back to exercise 05