• Dived a bit deeper into time series data
• Saw how to plot (multiple) trends in the same plot
• Learned how to calculate growth rates and normalise data in different ways
• Learned how to use the diff() function to calculate differences between consecutive observations and the shift() function to shift the data
• Saw how to use query() to filter data

• Today we will work with text data in Python
• Text data is as popular as ever, thanks to social media, LLMs, and the like
• We use text data to estimate sentiment, classify documents, ideal points of politicians, and much more
• Text is usually messy and unstructured, so we need to clean it before we can use it
• We will learn how to clean text data and how to use them for analysis

• We will use some pandas funcions today
• But there are several libraries that are worth cheking out too, such as nltk, spaCy, and gensim
• We will also introduce the concept of regular expressions, which allow you to search for patterns in text

• As usual, let’s start by importing the necessary libraries and the data we will use today
• The data are about congressional bills in the US

import pandas as pd

# Load the data
bills_actions = pd.read_csv("data_raw/bills_actions.csv")
bills_actions.dtypes

Congress        int64
bill_number     int64
bill_type      object
action         object
main_action    object
category       object
member_id       int64
dtype: object

• Let’s have a quick look at the data

bills_actions.head()

Counting categories

• A simple way to start working with text data is to count the number of words in a text or dataset
• Let’s see how many categories we have in the category column

bills_actions["category"].nunique()

9

• We can also use value_counts() to see the frequency of each category

bills_actions["category"].value_counts()

category
amendment                       1529
house bill                       902
senate bill                      514
house resolution                 234
senate resolution                 60
house joint resolution            22
house concurrent resolution       20
senate concurrent resolution      14
senate joint resolution            8
Name: count, dtype: int64

Subset text categories

• Here we are only interested in bills. So let’s use query() to subset the data
• We select all entries in the column called category which have values contain in list_categories
  • in is used to test whether a word belongs to a list
  • @ is the syntax to reference global variables that are defined in the global environment

list_categories = ["house bill","senate bill"]
bills = bills_actions.query('category in @list_categories')

# Verify that the code worked:
bills["category"].value_counts()

category
house bill     902
senate bill    514
Name: count, dtype: int64

• The str attribute allows us to access string methods, and there are a lot of them
• Here we will use str.contains() to check if a sentence contains a specific word
• Let’s see how many bills contain the word senator

# Check if the action contains the word "Senator"
bool_contains = bills["action"].str.contains("Senator")

# Check the result
bool_contains.head()

3    True
4    True
5    True
6    True
7    True
Name: action, dtype: bool

# Calculate the proportion 
bool_contains.mean()

0.3199152542372881

• Let’s double-check the result

bills[bills["action"].str.contains("Senator")]

• Regular expressions (“regex”) are a flexible tool to search for patterns in text
• Regex is a language in itself, and it is used in many programming languages
• They are indeed very powerful, but can also be very complex
• Here we will see just the basics of regex, but it is worth learning more about it (if you have the courage!)
• An example of a very simple regex used to validate email addresses: ^([a-zA-Z0-9_\-\.]+)@([a-zA-Z0-9_\-\.]+)\.([a-zA-Z]{2,5})$
• It means “a string that starts with a sequence of letters, numbers, underscores, hyphens, and dots, followed by an @, followed by another sequence of letters, numbers, underscores, hyphens, and dots, followed by a dot and a sequence of letters with length between 2 and 5” 😅

• Now let’s combine the str.contains() method with a regular expression
• Python handles regex with the re module, which is part of the standard library
• Let’s import it too

import re

• We can use the str.findall() method to find all occurrences of a pattern in a string
• This is similar to str.contains(), but it returns the actual matches instead of a boolean (True/False) when a match is found
• We will search for all occurrences of the word Amdt followed by a dot and a number or a non-digit character (e.g., Amdt.1, Amdt.2!, Amdt.3., Amdt.4@, Amdt.5a, etc)

• The regex pattern is Amdt\.\d+
  • Amdt is the word we are looking for
  • \. is used to escape the dot, which is a special character in regex
  • \d is used to match any digit
  • + is used to match one or more occurrences of the previous character
  • \D+ is used to match any non-digit character
• Note the r before the string, which is used to indicate a raw string (to avoid Python from interpreting the backslashes)
• Let’s see the result

amendments["action"].str.findall(r'Amdt\.\d+\D').head()

0     [Amdt.1274 ]
1     [Amdt.2698 ]
2     [Amdt.2659 ]
8     [Amdt.2424 ]
11    [Amdt.1275 ]
Name: action, dtype: object

• There are several special characters in regex. Examples:
• . = any character except a newline
• \d = digit
• \D = non-digit character
• \w = any word character (alphanumeric character plus underscore)
• \W = any non-word character
• \s = any whitespace character
• \S = any non-whitespace character
• + = one or more occurrences of the previous character

• Quantifiers are used to specify how many occurrences of a character we want to match
• * = zero or more occurrences of the previous character
• ? = zero or one occurrence of the previous character
• {n} = exactly n occurrences of the previous character
• {n,} = at least n occurrences of the previous character
• {n,m} = between n and m occurrences of the previous character
• ^ = start of a string
• $ = end of a string
• [] = a set of characters
• | = or
• () = group
• Enough for today! 😅

• Match any characters (including none) before Amdt followed by non-whitespace

amendments["action"].str.findall(r".*Amdt\S*").head()

0     [S.Amdt.1274]
1     [S.Amdt.2698]
2     [S.Amdt.2659]
8     [S.Amdt.2424]
11    [S.Amdt.1275]
Name: action, dtype: object