\n",
"\n",
""
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "361d390a",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"from os.path import basename, exists\n",
"\n",
"def download(url):\n",
" \"\"\"Download a file if it isn't already here, and return its filename.\"\"\"\n",
" filename = basename(url)\n",
" if not exists(filename):\n",
" from urllib.request import urlretrieve\n",
"\n",
" local, _ = urlretrieve(url, filename)\n",
" print(\"Downloaded \" + str(local))\n",
" return filename\n",
"\n",
"download('https://github.com/porttack/working-in-python/raw/v3/working_in_python.py');\n",
"download('https://github.com/AllenDowney/ThinkPython/raw/v3/diagram.py');\n",
"\n",
"import working_in_python\n",
"# apcsp:begin type=\"note\" chapter=\"08\"\n",
"working_in_python.enable_docstring_reminders()\n",
"# apcsp:end"
]
},
{
"cell_type": "markdown",
"id": "9d97603b",
"metadata": {},
"source": [
"# Strings and Regular Expressions\n",
"\n",
"Strings are not like integers, floats, and booleans. A string is a **sequence**, which means it contains multiple values in a particular order.\n",
"In this chapter we'll see how to access the values that make up a string, and we'll use functions that process strings.\n",
"\n",
"We'll also use regular expressions, which are a powerful tool for finding patterns in a string and performing operations like search and replace.\n",
"\n",
"As an exercise, you'll have a chance to apply these tools to a word game called Wordle."
]
},
{
"cell_type": "markdown",
"id": "1280dd83",
"metadata": {},
"source": [
"## A string is a sequence\n",
"\n",
"A string is a sequence of characters. A **character** can be a letter (in almost any alphabet), a digit, a punctuation mark, or white space.\n",
"\n",
"You can select a character from a string with the bracket operator.\n",
"This example statement selects character number 1 from `fruit` and\n",
"assigns it to `letter`:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "9b53c1fe",
"metadata": {},
"outputs": [],
"source": [
"fruit = 'banana'\n",
"letter = fruit[1]"
]
},
{
"cell_type": "markdown",
"id": "a307e429",
"metadata": {},
"source": [
"The expression in brackets is an **index**, so called because it *indicates* which character in the sequence to select.\n",
"But the result might not be what you expect.\n",
""
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "2cb1d58c",
"metadata": {},
"outputs": [],
"source": [
"letter"
]
},
{
"cell_type": "markdown",
"id": "57c13319",
"metadata": {},
"source": [
"The letter with index `1` is actually the second letter of the string.\n",
"An index is an offset from the beginning of the string, so the offset of the first letter is `0`."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "4ce1eb16",
"metadata": {},
"outputs": [],
"source": [
"fruit[0]"
]
},
{
"cell_type": "markdown",
"id": "57d8e54c",
"metadata": {},
"source": [
"You can think of `'b'` as the 0th letter of `'banana'` -- pronounced \"zero-eth\".\n",
"\n",
"The index in brackets can be a variable."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "11201ba9",
"metadata": {},
"outputs": [],
"source": [
"i = 1\n",
"fruit[i]"
]
},
{
"cell_type": "markdown",
"id": "9630e2e7",
"metadata": {},
"source": [
"Or an expression that contains variables and operators."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "fc4383d0",
"metadata": {},
"outputs": [],
"source": [
"fruit[i+1]"
]
},
{
"cell_type": "markdown",
"id": "939b602d",
"metadata": {},
"source": [
"But the value of the index has to be an integer -- otherwise you get a `TypeError`."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "aec20975",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"\n",
"fruit[1.5]"
]
},
{
"cell_type": "markdown",
"id": "3f0f7e3a",
"metadata": {},
"source": [
"As we saw in Chapter 1, we can use the built-in function `len` to get the length of a string."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "796ce317",
"metadata": {},
"outputs": [],
"source": [
"n = len(fruit)\n",
"n"
]
},
{
"cell_type": "markdown",
"id": "29013c47",
"metadata": {},
"source": [
"To get the last letter of a string, you might be tempted to write this:\n",
""
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "3ccb4a64",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"\n",
"fruit[n]"
]
},
{
"cell_type": "markdown",
"id": "b87e09bd",
"metadata": {},
"source": [
"But that causes an `IndexError` because there is no letter in `'banana'` with the index 6. Because we started counting at `0`, the six letters are numbered `0` to `5`. To get the last character, you have to subtract `1` from `n`:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "2cf99de6",
"metadata": {},
"outputs": [],
"source": [
"fruit[n-1]"
]
},
{
"cell_type": "markdown",
"id": "3c79dcec",
"metadata": {},
"source": [
"But there's an easier way.\n",
"To get the last letter in a string, you can use a negative index, which counts backward from the end. "
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "3dedf6fa",
"metadata": {},
"outputs": [],
"source": [
"fruit[-1]"
]
},
{
"cell_type": "markdown",
"id": "5677b727",
"metadata": {},
"source": [
"The index `-1` selects the last letter, `-2` selects the second to last, and so on."
]
},
{
"cell_type": "markdown",
"id": "8392a12a",
"metadata": {},
"source": [
"## String slices\n",
"\n",
"A segment of a string is called a **slice**.\n",
"Selecting a slice is similar to selecting a character."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "386b9df2",
"metadata": {},
"outputs": [],
"source": [
"fruit = 'banana'\n",
"fruit[0:3]"
]
},
{
"cell_type": "markdown",
"id": "5cc12531",
"metadata": {},
"source": [
"The operator `[n:m]` returns the part of the string from the `n`th\n",
"character to the `m`th character, including the first but excluding the second.\n",
"This behavior is counterintuitive, but it might help to imagine the indices pointing *between* the characters, as in this figure:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "05f9743d",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"from diagram import make_binding, Element, Value\n",
"\n",
"binding = make_binding(\"fruit\", ' b a n a n a ')\n",
"elements = [Element(Value(i), None) for i in range(7)]"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "b09d8356",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"from diagram import diagram, adjust\n",
"from matplotlib.transforms import Bbox\n",
"\n",
"width, height, x, y = [1.35, 0.54, 0.23, 0.39]\n",
"\n",
"ax = diagram(width, height)\n",
"bbox = binding.draw(ax, x, y)\n",
"bboxes = [bbox]\n",
"\n",
"def draw_elts(x, y, elements):\n",
" for elt in elements:\n",
" bbox = elt.draw(ax, x, y, draw_value=False)\n",
" bboxes.append(bbox)\n",
"\n",
" x1 = (bbox.xmin + bbox.xmax) / 2\n",
" y1 = bbox.ymax + 0.02\n",
" y2 = y1 + 0.14\n",
" handle = plt.plot([x1, x1], [y1, y2], ':', lw=0.5, color='gray')\n",
" x += 0.105\n",
" \n",
"draw_elts(x + 0.48, y - 0.25, elements)\n",
"bbox = Bbox.union(bboxes)\n",
"# adjust(x, y, bbox)"
]
},
{
"cell_type": "markdown",
"id": "c391abbf",
"metadata": {},
"source": [
"For example, the slice `[3:6]` selects the letters `ana`, which means that `6` is legal as part of a slice, but not legal as an index.\n",
"\n",
"\n",
"If you omit the first index, the slice starts at the beginning of the string."
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "00592313",
"metadata": {},
"outputs": [],
"source": [
"fruit[:3]"
]
},
{
"cell_type": "markdown",
"id": "1bd7dcb1",
"metadata": {},
"source": [
"If you omit the second index, the slice goes to the end of the string:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "01684797",
"metadata": {},
"outputs": [],
"source": [
"fruit[3:]"
]
},
{
"cell_type": "markdown",
"id": "4701123b",
"metadata": {},
"source": [
"If the first index is greater than or equal to the second, the result is an **empty string**, represented by two quotation marks:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "c7551ded",
"metadata": {},
"outputs": [],
"source": [
"fruit[3:3]"
]
},
{
"cell_type": "markdown",
"id": "d12735ab",
"metadata": {},
"source": [
"An empty string contains no characters and has length 0.\n",
"\n",
"Continuing this example, what do you think `fruit[:]` means? Try it and\n",
"see."
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "b5c5ce3e",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"fruit[:]"
]
},
{
"cell_type": "markdown",
"id": "918d3dd0",
"metadata": {},
"source": [
"## Strings are immutable\n",
"\n",
"It is tempting to use the `[]` operator on the left side of an\n",
"assignment, with the intention of changing a character in a string, like this:\n",
""
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "69ccd380",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"\n",
"greeting = 'Hello, world!'\n",
"greeting[0] = 'J'"
]
},
{
"cell_type": "markdown",
"id": "df3dd7d1",
"metadata": {},
"source": [
"The result is a `TypeError`.\n",
"In the error message, the \"object\" is the string and the \"item\" is the character\n",
"we tried to assign.\n",
"For now, an **object** is the same thing as a value, but we will refine that definition later.\n",
"\n",
"The reason for this error is that strings are **immutable**, which means you can't change an existing string.\n",
"The best you can do is create a new string that is a variation of the original."
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "280d27a1",
"metadata": {},
"outputs": [],
"source": [
"new_greeting = 'J' + greeting[1:]\n",
"new_greeting"
]
},
{
"cell_type": "markdown",
"id": "2848546f",
"metadata": {},
"source": [
"This example concatenates a new first letter onto a slice of `greeting`.\n",
"It has no effect on the original string."
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "8fa4a4cf",
"metadata": {},
"outputs": [],
"source": [
"greeting"
]
},
{
"cell_type": "markdown",
"id": "49e4da57",
"metadata": {},
"source": [
"## String comparison\n",
"\n",
"The relational operators work on strings. To see if two strings are\n",
"equal, we can use the `==` operator."
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "b754d462",
"metadata": {},
"outputs": [],
"source": [
"word = 'banana'\n",
"\n",
"if word == 'banana':\n",
" print('All right, banana.')"
]
},
{
"cell_type": "markdown",
"id": "e9be6097",
"metadata": {},
"source": [
"Other relational operations are useful for putting words in alphabetical\n",
"order:"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "44374eb8",
"metadata": {},
"outputs": [],
"source": [
"def compare_word(word):\n",
" if word < 'banana':\n",
" print(word, 'comes before banana.')\n",
" elif word > 'banana':\n",
" print(word, 'comes after banana.')\n",
" else:\n",
" print('All right, banana.')"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "a46f7035",
"metadata": {},
"outputs": [],
"source": [
"compare_word('apple')"
]
},
{
"cell_type": "markdown",
"id": "b66f449a",
"metadata": {},
"source": [
"Python does not handle uppercase and lowercase letters the same way\n",
"people do. All the uppercase letters come before all the lowercase\n",
"letters, so:"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "a691f9e2",
"metadata": {},
"outputs": [],
"source": [
"compare_word('Pineapple')"
]
},
{
"cell_type": "markdown",
"id": "f9b916c9",
"metadata": {},
"source": [
"To solve this problem, we can convert strings to a standard format, such as all lowercase, before performing the comparison.\n",
"Keep that in mind if you have to defend yourself against a man armed with a Pineapple."
]
},
{
"cell_type": "markdown",
"id": "531069f1",
"metadata": {},
"source": [
"## String methods\n",
"\n",
"Strings provide methods that perform a variety of useful operations. \n",
"A method is similar to a function -- it takes arguments and returns a value -- but the syntax is different.\n",
"For example, the method `upper` takes a string and returns a new string with all uppercase letters.\n",
"\n",
"Instead of the function syntax `upper(word)`, it uses the method syntax `word.upper()`."
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "fa6140a6",
"metadata": {},
"outputs": [],
"source": [
"word = 'banana'\n",
"new_word = word.upper()\n",
"new_word"
]
},
{
"cell_type": "markdown",
"id": "1ac41744",
"metadata": {},
"source": [
"This use of the dot operator specifies the name of the method, `upper`, and the name of the string to apply the method to, `word`.\n",
"The empty parentheses indicate that this method takes no arguments.\n",
"\n",
"A method call is called an **invocation**; in this case, we would say that we are invoking `upper` on `word`."
]
},
{
"cell_type": "markdown",
"id": "2a13d4ef",
"metadata": {
"tags": []
},
"source": [
"## Writing files\n",
"\n",
"String operators and methods are useful for reading and writing text files.\n",
"As an example, we'll work with the text of *Dracula*, a novel by Bram Stoker that is available from Project Gutenberg ()."
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "e3f1dc18",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import os\n",
"\n",
"if not os.path.exists('pg345.txt'):\n",
" !wget https://www.gutenberg.org/cache/epub/345/pg345.txt"
]
},
{
"cell_type": "markdown",
"id": "963dda79",
"metadata": {},
"source": [
"I've downloaded the book in a plain text file called `pg345.txt`, which we can open for reading like this:"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "bd2d5175",
"metadata": {},
"outputs": [],
"source": [
"reader = open('pg345.txt')"
]
},
{
"cell_type": "markdown",
"id": "b5d99e8c",
"metadata": {},
"source": [
"In addition to the text of the book, this file contains a section at the beginning with information about the book and a section at the end with information about the license.\n",
"Before we process the text, we can remove this extra material by finding the special lines at the beginning and end that begin with `'***'`.\n",
"\n",
"The following function takes a line and checks whether it is one of the special lines.\n",
"It uses the `startswith` method, which checks whether a string starts with a given sequence of characters."
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "b9c9318c",
"metadata": {},
"outputs": [],
"source": [
"def is_special_line(line):\n",
" return line.startswith('*** ')"
]
},
{
"cell_type": "markdown",
"id": "2efdfe35",
"metadata": {},
"source": [
"We can use this function to loop through the lines in the file and print only the special lines."
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "a9417d4c",
"metadata": {},
"outputs": [],
"source": [
"for line in reader:\n",
" if is_special_line(line):\n",
" print(line.strip())"
]
},
{
"cell_type": "markdown",
"id": "07fb5992",
"metadata": {},
"source": [
"Now let's create a new file, called `pg345_cleaned.txt`, that contains only the text of the book.\n",
"In order to loop through the book again, we have to open it again for reading.\n",
"And, to write a new file, we can open it for writing."
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "f2336825",
"metadata": {},
"outputs": [],
"source": [
"reader = open('pg345.txt')\n",
"writer = open('pg345_cleaned.txt', 'w')"
]
},
{
"cell_type": "markdown",
"id": "96d881aa",
"metadata": {},
"source": [
"`open` takes an optional parameters that specifies the \"mode\" -- in this example, `'w'` indicates that we're opening the file for writing.\n",
"If the file doesn't exist, it will be created; if it already exists, the contents will be replaced.\n",
"\n",
"As a first step, we'll loop through the file until we find the first special line."
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "d1b286ee",
"metadata": {},
"outputs": [],
"source": [
"for line in reader:\n",
" if is_special_line(line):\n",
" break"
]
},
{
"cell_type": "markdown",
"id": "1989d5a1",
"metadata": {},
"source": [
"The `break` statement \"breaks\" out of the loop -- that is, it causes the loop to end immediately, before we get to the end of the file.\n",
"\n",
"When the loop exits, `line` contains the special line that made the conditional true."
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "b4ecf365",
"metadata": {},
"outputs": [],
"source": [
"line"
]
},
{
"cell_type": "markdown",
"id": "9f28c3b4",
"metadata": {},
"source": [
"Because `reader` keeps track of where it is in the file, we can use a second loop to pick up where we left off.\n",
"\n",
"The following loop reads the rest of the file, one line at a time.\n",
"When it finds the special line that indicates the end of the text, it breaks out of the loop.\n",
"Otherwise, it writes the line to the output file."
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "a99dc11c",
"metadata": {},
"outputs": [],
"source": [
"for line in reader:\n",
" if is_special_line(line):\n",
" break\n",
" writer.write(line)"
]
},
{
"cell_type": "markdown",
"id": "c07032a4",
"metadata": {},
"source": [
"When this loop exits, `line` contains the second special line."
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "dfd6b264",
"metadata": {},
"outputs": [],
"source": [
"line"
]
},
{
"cell_type": "markdown",
"id": "0c30b41c",
"metadata": {},
"source": [
"At this point `reader` and `writer` are still open, which means we could keep reading lines from `reader` or writing lines to `writer`.\n",
"To indicate that we're done, we can close both files by invoking the `close` method."
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "4eda555c",
"metadata": {},
"outputs": [],
"source": [
"reader.close()\n",
"writer.close()"
]
},
{
"cell_type": "markdown",
"id": "d5084cdc",
"metadata": {},
"source": [
"To check whether this process was successful, we can read the first few lines from the new file we just created."
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "5e1e8c74",
"metadata": {},
"outputs": [],
"source": [
"for line in open('pg345_cleaned.txt'):\n",
" line = line.strip()\n",
" if len(line) > 0:\n",
" print(line)\n",
" if line.endswith('Stoker'):\n",
" break"
]
},
{
"cell_type": "markdown",
"id": "34c93df3",
"metadata": {},
"source": [
"The `endswith` method checks whether a string ends with a given sequence of characters."
]
},
{
"cell_type": "markdown",
"id": "fcdb4bbf",
"metadata": {},
"source": [
"## Find and replace\n",
"\n",
"In the Icelandic translation of *Dracula* from 1901, the name of one of the characters was changed from \"Jonathan\" to \"Thomas\".\n",
"To make this change in the English version, we can loop through the book, use the `replace` method to replace one name with another, and write the result to a new file.\n",
"\n",
"We'll start by counting the lines in the cleaned version of the file."
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "63ebaafb",
"metadata": {},
"outputs": [],
"source": [
"total = 0\n",
"for line in open('pg345_cleaned.txt'):\n",
" total += 1\n",
" \n",
"total"
]
},
{
"cell_type": "markdown",
"id": "8ba9b9ca",
"metadata": {},
"source": [
"To see whether a line contains \"Jonathan\", we can use the `in` operator, which checks whether this sequence of characters appears anywhere in the line."
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "9973e6e8",
"metadata": {},
"outputs": [],
"source": [
"total = 0\n",
"for line in open('pg345_cleaned.txt'):\n",
" if 'Jonathan' in line:\n",
" total += 1\n",
"\n",
"total"
]
},
{
"cell_type": "markdown",
"id": "27805245",
"metadata": {},
"source": [
"There are 199 lines that contain the name, but that's not quite the total number of times it appears, because it can appear more than once in a line.\n",
"To get the total, we can use the `count` method, which returns the number of times a sequence appears in a string."
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "02e06ff1",
"metadata": {},
"outputs": [],
"source": [
"total = 0\n",
"for line in open('pg345_cleaned.txt'):\n",
" total += line.count('Jonathan')\n",
"\n",
"total"
]
},
{
"cell_type": "markdown",
"id": "68026797",
"metadata": {},
"source": [
"Now we can replace `'Jonathan'` with `'Thomas'` like this:"
]
},
{
"cell_type": "code",
"execution_count": 41,
"id": "1450e82c",
"metadata": {},
"outputs": [],
"source": [
"writer = open('pg345_replaced.txt', 'w')\n",
"\n",
"for line in open('pg345_cleaned.txt'):\n",
" line = line.replace('Jonathan', 'Thomas')\n",
" writer.write(line)"
]
},
{
"cell_type": "markdown",
"id": "57ba56f3",
"metadata": {},
"source": [
"The result is a new file called `pg345_replaced.txt` that contains a version of *Dracula* where Jonathan Harker is called Thomas."
]
},
{
"cell_type": "code",
"execution_count": 42,
"id": "a57b64c6",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"total = 0\n",
"for line in open('pg345_replaced.txt'):\n",
" total += line.count('Thomas')\n",
"\n",
"total"
]
},
{
"cell_type": "markdown",
"id": "cc9af187",
"metadata": {},
"source": [
"## Regular expressions\n",
"\n",
"If we know exactly what sequence of characters we're looking for, we can use the `in` operator to find it and the `replace` method to replace it.\n",
"But there is another tool, called a **regular expression** that can also perform these operations -- and a lot more.\n",
"\n",
"To demonstrate, I'll start with a simple example and we'll work our way up.\n",
"Suppose, again, that we want to find all lines that contain a particular word.\n",
"For a change, let's look for references to the titular character of the book, Count Dracula.\n",
"Here's a line that mentions him."
]
},
{
"cell_type": "code",
"execution_count": 43,
"id": "a6069027",
"metadata": {},
"outputs": [],
"source": [
"text = \"I am Dracula; and I bid you welcome, Mr. Harker, to my house.\""
]
},
{
"cell_type": "markdown",
"id": "d4fd6d11",
"metadata": {},
"source": [
"And here's the **pattern** we'll use to search."
]
},
{
"cell_type": "code",
"execution_count": 44,
"id": "e3c19abe",
"metadata": {},
"outputs": [],
"source": [
"pattern = 'Dracula'"
]
},
{
"cell_type": "markdown",
"id": "268f647c",
"metadata": {},
"source": [
"A module called `re` provides functions related to regular expressions.\n",
"We can import it like this and use the `search` function to check whether the pattern appears in the text."
]
},
{
"cell_type": "code",
"execution_count": 45,
"id": "db588abb",
"metadata": {},
"outputs": [],
"source": [
"import re\n",
"\n",
"result = re.search(pattern, text)\n",
"result"
]
},
{
"cell_type": "markdown",
"id": "e17f6731",
"metadata": {},
"source": [
"If the pattern appears in the text, `search` returns a `Match` object that contains the results of the search.\n",
"Among other information, it has a variable named `string` that contains the text that was searched."
]
},
{
"cell_type": "code",
"execution_count": 46,
"id": "924524a6",
"metadata": {},
"outputs": [],
"source": [
"result.string"
]
},
{
"cell_type": "markdown",
"id": "a8eab0f6",
"metadata": {},
"source": [
"It also provides a method called `group` that returns the part of the text that matched the pattern."
]
},
{
"cell_type": "code",
"execution_count": 47,
"id": "c72b860c",
"metadata": {},
"outputs": [],
"source": [
"result.group()"
]
},
{
"cell_type": "markdown",
"id": "b6962a7d",
"metadata": {},
"source": [
"And it provides a method called `span` that returns the index in the text where the pattern starts and ends."
]
},
{
"cell_type": "code",
"execution_count": 48,
"id": "7c2f556c",
"metadata": {},
"outputs": [],
"source": [
"result.span()"
]
},
{
"cell_type": "markdown",
"id": "8f1e5261",
"metadata": {},
"source": [
"If the pattern doesn't appear in the text, the return value from `search` is `None`."
]
},
{
"cell_type": "code",
"execution_count": 49,
"id": "d5242ef6",
"metadata": {},
"outputs": [],
"source": [
"result = re.search('Count', text)\n",
"print(result)"
]
},
{
"cell_type": "markdown",
"id": "d5ed33ff",
"metadata": {},
"source": [
"So we can check whether the search was successful by checking whether the result is `None`."
]
},
{
"cell_type": "code",
"execution_count": 50,
"id": "18c09b63",
"metadata": {},
"outputs": [],
"source": [
"result == None"
]
},
{
"cell_type": "markdown",
"id": "a08e38f6",
"metadata": {},
"source": [
"Putting all that together, here's a function that loops through the lines in the book until it finds one that matches the given pattern, and returns the `Match` object."
]
},
{
"cell_type": "code",
"execution_count": 51,
"id": "fedb7d95",
"metadata": {},
"outputs": [],
"source": [
"def find_first(pattern):\n",
" for line in open('pg345_cleaned.txt'):\n",
" result = re.search(pattern, line)\n",
" if result != None:\n",
" return result"
]
},
{
"cell_type": "markdown",
"id": "96570515",
"metadata": {},
"source": [
"We can use it to find the first mention of a character."
]
},
{
"cell_type": "code",
"execution_count": 52,
"id": "d7cbe2c2",
"metadata": {},
"outputs": [],
"source": [
"result = find_first('Harker')\n",
"result.string"
]
},
{
"cell_type": "markdown",
"id": "3f687fdc",
"metadata": {},
"source": [
"For this example, we didn't have to use regular expressions -- we could have done the same thing more easily with the `in` operator.\n",
"But regular expressions can do things the `in` operator cannot.\n",
"\n",
"For example, if the pattern includes the vertical bar character, `'|'`, it can match either the sequence on the left or the sequence on the right.\n",
"Suppose we want to find the first mention of Mina Murray in the book, but we are not sure whether she is referred to by first name or last.\n",
"We can use the following pattern, which matches either name."
]
},
{
"cell_type": "code",
"execution_count": 53,
"id": "96c64f83",
"metadata": {},
"outputs": [],
"source": [
"pattern = 'Mina|Murray'\n",
"result = find_first(pattern)\n",
"result.string"
]
},
{
"cell_type": "markdown",
"id": "8bea66a6",
"metadata": {},
"source": [
"We can use a pattern like this to see how many times a character is mentioned by either name.\n",
"Here's a function that loops through the book and counts the number of lines that match the given pattern."
]
},
{
"cell_type": "code",
"execution_count": 54,
"id": "d0d2e926",
"metadata": {},
"outputs": [],
"source": [
"def count_matches(pattern):\n",
" count = 0\n",
" for line in open('pg345_cleaned.txt'):\n",
" result = re.search(pattern, line)\n",
" if result != None:\n",
" count += 1\n",
" return count"
]
},
{
"cell_type": "markdown",
"id": "0e753a5b",
"metadata": {},
"source": [
"Now let's see how many times Mina is mentioned."
]
},
{
"cell_type": "code",
"execution_count": 55,
"id": "d7e8c5b4",
"metadata": {},
"outputs": [],
"source": [
"count_matches('Mina|Murray')"
]
},
{
"cell_type": "markdown",
"id": "780c9fab",
"metadata": {},
"source": [
"The special character `'^'` matches the beginning of a string, so we can find a line that starts with a given pattern."
]
},
{
"cell_type": "code",
"execution_count": 56,
"id": "be63c5b0",
"metadata": {},
"outputs": [],
"source": [
"result = find_first('^Dracula')\n",
"result.string"
]
},
{
"cell_type": "markdown",
"id": "332bad2e",
"metadata": {},
"source": [
"And the special character `'$'` matches the end of a string, so we can find a line that ends with a given pattern (ignoring the newline at the end)."
]
},
{
"cell_type": "code",
"execution_count": 57,
"id": "37595ac5",
"metadata": {},
"outputs": [],
"source": [
"result = find_first('Harker$')\n",
"result.string"
]
},
{
"cell_type": "markdown",
"id": "d4b22b6e",
"metadata": {},
"source": [
"## String substitution\n",
"\n",
"Bram Stoker was born in Ireland, and when *Dracula* was published in 1897, he was living in England.\n",
"So we would expect him to use the British spelling of words like \"centre\" and \"colour\".\n",
"To check, we can use the following pattern, which matches either \"centre\" or the American spelling \"center\"."
]
},
{
"cell_type": "code",
"execution_count": 58,
"id": "18237bea",
"metadata": {},
"outputs": [],
"source": [
"pattern = 'cent(er|re)'"
]
},
{
"cell_type": "markdown",
"id": "35abfd7d",
"metadata": {},
"source": [
"In this pattern, the parentheses enclose the part of the pattern the vertical bar applies to.\n",
"So this pattern matches a sequence that starts with `'cent'` and ends with either `'er'` or `'re'`."
]
},
{
"cell_type": "code",
"execution_count": 59,
"id": "ce65805f",
"metadata": {},
"outputs": [],
"source": [
"result = find_first(pattern)\n",
"result.string"
]
},
{
"cell_type": "markdown",
"id": "e5703c18",
"metadata": {},
"source": [
"As expected, he used the British spelling.\n",
"\n",
"We can also check whether he used the British spelling of \"colour\".\n",
"The following pattern uses the special character `'?'`, which means that the previous character is optional."
]
},
{
"cell_type": "code",
"execution_count": 60,
"id": "af770664",
"metadata": {},
"outputs": [],
"source": [
"pattern = 'colou?r'"
]
},
{
"cell_type": "markdown",
"id": "beed9a7b",
"metadata": {},
"source": [
"This pattern matches either \"colour\" with the `'u'` or \"color\" without it."
]
},
{
"cell_type": "code",
"execution_count": 61,
"id": "ed67bde7",
"metadata": {},
"outputs": [],
"source": [
"result = find_first(pattern)\n",
"line = result.string\n",
"line"
]
},
{
"cell_type": "markdown",
"id": "1a31f179",
"metadata": {},
"source": [
"Again, as expected, he used the British spelling.\n",
"\n",
"Now suppose we want to produce an edition of the book with American spellings.\n",
"We can use the `sub` function in the `re` module, which does **string substitution**."
]
},
{
"cell_type": "code",
"execution_count": 62,
"id": "52dd938c",
"metadata": {},
"outputs": [],
"source": [
"re.sub(pattern, 'color', line)"
]
},
{
"cell_type": "markdown",
"id": "04a80fc6",
"metadata": {},
"source": [
"The first argument is the pattern we want to find and replace, the second is what we want to replace it with, and the third is the string we want to search.\n",
"In the result, you can see that \"colour\" has been replaced with \"color\"."
]
},
{
"cell_type": "code",
"execution_count": 63,
"id": "d2e309a2",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"# I used this function to search for lines to use as examples\n",
"\n",
"def all_matches(pattern):\n",
" for line in open('pg345_cleaned.txt'):\n",
" result = re.search(pattern, line)\n",
" if result:\n",
" print(line.strip())"
]
},
{
"cell_type": "code",
"execution_count": 64,
"id": "3d8b2a9f",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"# Here's the pattern I used (which uses some features we haven't seen)\n",
"\n",
"names = r'(?`, which indicates that the results should be written to a file rather than displayed."
]
},
{
"cell_type": "code",
"execution_count": 67,
"id": "d4c92501",
"metadata": {},
"outputs": [],
"source": [
"!head pg345_cleaned.txt > pg345_cleaned_10_lines.txt"
]
},
{
"cell_type": "markdown",
"id": "3fc851f8",
"metadata": {},
"source": [
"By default, `!head` reads the first 10 lines, but it takes an optional argument that indicates the number of lines to read."
]
},
{
"cell_type": "code",
"execution_count": 68,
"id": "8f6606dd",
"metadata": {},
"outputs": [],
"source": [
"!head -100 pg345_cleaned.txt > pg345_cleaned_100_lines.txt"
]
},
{
"cell_type": "markdown",
"id": "24871c78",
"metadata": {},
"source": [
"This shell command reads the first 100 lines from `pg345_cleaned.txt` and writes them to a file called `pg345_cleaned_100_lines.txt`.\n",
"\n",
"Note: The shell commands `!head` and `!tail` are not available on all operating systems.\n",
"If they don't work for you, we can write similar functions in Python.\n",
"See the first exercise at the end of this chapter for suggestions."
]
},
{
"cell_type": "markdown",
"id": "c842524d",
"metadata": {},
"source": [
"## Glossary\n",
"\n",
"**sequence:**\n",
" An ordered collection of values where each value is identified by an integer index.\n",
"\n",
"**character:**\n",
"An element of a string, including letters, numbers, and symbols.\n",
"\n",
"**index:**\n",
" An integer value used to select an item in a sequence, such as a character in a string. In Python indices start from `0`.\n",
"\n",
"**slice:**\n",
" A part of a string specified by a range of indices.\n",
"\n",
"**empty string:**\n",
"A string that contains no characters and has length `0`.\n",
"\n",
"**object:**\n",
" Something a variable can refer to. An object has a type and a value.\n",
"\n",
"**immutable:**\n",
"If the elements of an object cannot be changed, the object is immutable.\n",
"\n",
"**invocation:**\n",
" An expression -- or part of an expression -- that calls a method.\n",
"\n",
"**regular expression:**\n",
"A sequence of characters that defines a search pattern.\n",
"\n",
"**pattern:**\n",
"A rule that specifies the requirements a string has to meet to constitute a match.\n",
"\n",
"**string substitution:**\n",
"Replacement of a string, or part of a string, with another string.\n",
"\n",
"**shell command:**\n",
"A statement in a shell language, which is a language used to interact with an operating system."
]
},
{
"cell_type": "markdown",
"id": "8ce7473f",
"metadata": {},
"source": [
"\n",
"**See also:** the [full vocabulary glossary](https://python.porttack.com/alignment/ap-vocabulary-glossary.html) collects every term in this book, alphabetized, alongside the complete AP CSP exam vocabulary list.\n",
""
]
},
{
"cell_type": "markdown",
"id": "4306e765",
"metadata": {},
"source": [
"## Exercises"
]
},
{
"cell_type": "code",
"execution_count": 69,
"id": "18bced21",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"# This cell tells Jupyter to provide detailed debugging information\n",
"# when a runtime error occurs. Run it before working on the exercises.\n",
"\n",
"%xmode Verbose"
]
},
{
"cell_type": "code",
"execution_count": 70,
"id": "772f5c14",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"download('https://raw.githubusercontent.com/AllenDowney/ThinkPython/v3/words.txt');"
]
},
{
"cell_type": "code",
"execution_count": 71,
"id": "8650dec2",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"from doctest import run_docstring_examples\n",
"\n",
"def run_doctests(func):\n",
" run_docstring_examples(func, globals(), name=func.__name__)"
]
},
{
"cell_type": "markdown",
"id": "20dcbbb3",
"metadata": {},
"source": [
"### Exercise\n",
"\n",
"See if you can write a function that does the same thing as the shell command `!head`.\n",
"It should take as arguments the name of a file to read, the number of lines to read, and the name of the file to write the lines into.\n",
"If the third parameter is `None`, it should display the lines rather than write them to a file.\n",
"\n",
"Don't use a `with` statement or a `try` statement -- we haven't covered them yet."
]
},
{
"cell_type": "code",
"execution_count": 72,
"id": "75b12538",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "24f6aac3",
"metadata": {
"tags": []
},
"source": [
"You can use the following examples to test your function."
]
},
{
"cell_type": "code",
"execution_count": 73,
"id": "9cbc19cd",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"head('pg345_cleaned.txt', 10)"
]
},
{
"cell_type": "code",
"execution_count": 74,
"id": "19de7df0",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"head('pg345_cleaned.txt', 100, 'pg345_cleaned_100_lines.txt')"
]
},
{
"cell_type": "code",
"execution_count": 75,
"id": "242f7ba6",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!tail pg345_cleaned_100_lines.txt"
]
},
{
"cell_type": "markdown",
"id": "adb78357",
"metadata": {},
"source": [
"### Exercise\n",
"\n",
"\"Wordle\" is an online word game where the objective is to guess a five-letter word in six or fewer attempts.\n",
"Each attempt has to be recognized as a word, not including proper nouns.\n",
"After each attempt, you get information about which of the letters you guessed appear in the target word, and which ones are in the correct position.\n",
"\n",
"For example, suppose the target word is `MOWER` and you guess `TRIED`.\n",
"You would learn that `E` is in the word and in the correct position, `R` is in the word but not in the correct position, and `T`, `I`, and `D` are not in the word.\n",
"\n",
"As a different example, suppose you have guessed the words `SPADE` and `CLERK`, and you've learned that `E` is in the word, but not in either of those positions, and none of the other letters appear in the word.\n",
"Of the words in the word list, how many could be the target word?\n",
"Write a function called `check_word` that takes a five-letter word and checks whether it could be the target word, given these guesses."
]
},
{
"cell_type": "code",
"execution_count": 76,
"id": "2a37092e",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "87fdf676",
"metadata": {},
"source": [
"You can use any of the functions from the previous chapter, like `uses_any`."
]
},
{
"cell_type": "code",
"execution_count": 77,
"id": "8d19b6ce",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"def uses_any(word, letters):\n",
" for letter in word.lower():\n",
" if letter in letters.lower():\n",
" return True\n",
" return False"
]
},
{
"cell_type": "markdown",
"id": "63593f1b",
"metadata": {
"tags": []
},
"source": [
"You can use the following loop to test your function."
]
},
{
"cell_type": "code",
"execution_count": 78,
"id": "9bbf0b1c",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"for line in open('words.txt'):\n",
" word = line.strip()\n",
" if len(word) == 5 and check_word(word):\n",
" print(word)"
]
},
{
"cell_type": "markdown",
"id": "d009cb52",
"metadata": {},
"source": [
"### Exercise\n",
"\n",
"Continuing the previous exercise, suppose you guess the work `TOTEM` and learn that the `E` is *still* not in the right place, but the `M` is. How many words are left?"
]
},
{
"cell_type": "code",
"execution_count": 79,
"id": "925c7aa9",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 80,
"id": "3f658f3a",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "c1d0f892",
"metadata": {},
"source": [
"### Exercise\n",
"\n",
"*The Count of Monte Cristo* is a novel by Alexandre Dumas that is considered a classic.\n",
"Nevertheless, in the introduction of an English translation of the book, the writer Umberto Eco confesses that he found the book to be \"one of the most badly written novels of all time\".\n",
"\n",
"In particular, he says it is \"shameless in its repetition of the same adjective,\" and mentions in particular the number of times \"its characters either shudder or turn pale.\"\n",
"\n",
"To see whether his objection is valid, let's count the number number of lines that contain the word `pale` in any form, including `pale`, `pales`, `paled`, and `paleness`, as well as the related word `pallor`. \n",
"Use a single regular expression that matches any of these words.\n",
"As an additional challenge, make sure that it doesn't match any other words, like `impale`."
]
},
{
"cell_type": "markdown",
"id": "742efd98",
"metadata": {
"tags": []
},
"source": [
"The following cell downloads the book from Project Gutenberg ."
]
},
{
"cell_type": "code",
"execution_count": 81,
"id": "9a74be13",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import os\n",
"\n",
"if not os.path.exists('pg1184.txt'):\n",
" !wget https://www.gutenberg.org/cache/epub/1184/pg1184.txt"
]
},
{
"cell_type": "markdown",
"id": "881c2f99",
"metadata": {
"tags": []
},
"source": [
"The following cell runs a function that reads the file from Project Gutenberg and writes a file that contains only the text of the book, not the added information about the book."
]
},
{
"cell_type": "code",
"execution_count": 82,
"id": "946c63d2",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"def clean_file(input_file, output_file):\n",
" reader = open(input_file)\n",
" writer = open(output_file, 'w')\n",
"\n",
" for line in reader:\n",
" if is_special_line(line):\n",
" break\n",
"\n",
" for line in reader:\n",
" if is_special_line(line):\n",
" break\n",
" writer.write(line)\n",
" \n",
" reader.close()\n",
" writer.close()\n",
"\n",
"clean_file('pg1184.txt', 'pg1184_cleaned.txt')"
]
},
{
"cell_type": "code",
"execution_count": 83,
"id": "08294921",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 84,
"id": "3eb8f83f",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 85,
"id": "b6bffe8a",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "7db56337",
"metadata": {
"tags": []
},
"source": [
"By this count, these words appear on `223` lines of the book, so Mr. Eco might have a point."
]
},
{
"cell_type": "markdown",
"id": "a7f4edf8",
"metadata": {
"tags": []
},
"source": [
"\n",
"---\n",
"\n",
"**Working in Python** — modified by Eric Brown for a high school Computer Science Principles class. Source and modifications: [github.com/porttack/working-in-python](https://github.com/porttack/working-in-python)\n",
"\n",
"---\n",
"\n",
"\n",
"[Think Python: 3rd Edition](https://allendowney.github.io/ThinkPython/index.html)\n",
"\n",
"Copyright 2024 [Allen B. Downey](https://allendowney.com)\n",
"\n",
"Code license: [MIT License](https://mit-license.org/)\n",
"\n",
"Text license: [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-nc-sa/4.0/)\n",
"\n",
"\n",
"---\n",
"\n",
"## Standards alignment\n",
"\n",
"**AP CSP:** [3.4 Strings](https://python.porttack.com/alignment/apcsp-standards-reference.html#T-3.4) and [3.14 Libraries](https://python.porttack.com/alignment/apcsp-standards-reference.html#T-3.14) (Big Idea 3, 30–35% of the exam) — both carry real weight here, not just a mention. Also [1.4 Identifying and Correcting Errors](https://python.porttack.com/alignment/apcsp-standards-reference.html#T-1.4) (Big Idea 1, 10–13%), headers only.\n",
"**California 9-12:** [9-12.AP.17](https://python.porttack.com/alignment/ca-cs-standards-reference.html#S-9-12.AP.17)\n",
"**CSTA 2026:** [HS-DAT-DC-23](https://python.porttack.com/alignment/csta2026-standards-reference.html#T-HS-DAT-DC-23) and [HS-PRO-RD-17](https://python.porttack.com/alignment/csta2026-standards-reference.html#T-HS-PRO-RD-17). Also [HS-PRO-PD-13](https://python.porttack.com/alignment/csta2026-standards-reference.html#T-HS-PRO-PD-13), headers only.\n",
"**CA CTE (ICT):** [C4.7](https://python.porttack.com/alignment/ca-ict-anchor-standards-reference.html#T-C4.7), [C4.9](https://python.porttack.com/alignment/ca-ict-anchor-standards-reference.html#T-C4.9) (Pathway C), and [5.12](https://python.porttack.com/alignment/ca-ict-anchor-standards-reference.html#T-5.12) (Anchor Standard 5, Problem Solving). Also [C5.6](https://python.porttack.com/alignment/ca-ict-anchor-standards-reference.html#T-C5.6), headers only.\n",
"\n",
"This chapter treats a string as a sequence: indexed, sliced, compared, and searched — the exam's full string content (3.4), reached for the first time. Regular expressions are this book's deepest library example yet (3.14, California's AP.17), and cleaning up messy text with `re.sub` is exactly CSTA's data-preparation standard. Searching with `re.search` and testing its result against `None` is CTE's boolean-logic-underlies-searching standard in practice, and reading and writing files rounds out CTE's construct list.\n",
"\n",
"**Indexing:** this chapter's slices (`fruit[0:3]`) are 0-based and exclude the end index. The exam's pseudocode is 1-based and includes both endpoints.\n",
""
]
}
],
"metadata": {
"celltoolbar": "Tags",
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.11"
}
},
"nbformat": 4,
"nbformat_minor": 5
}