Regular Expressions

When we scraped web pages, we kept hitting the same wall: the piece we wanted was buried in a wall of text. A page has the one phone number we need and a thousand characters we do not; a downloaded file has the ten lines that matter and a hundred that do not. What we want is a way to say find me every piece of text shaped like a phone number, not one exact string but a whole family of strings that follow a pattern. That is what a regular expression (a regex) is: a small language for describing patterns in text, so the computer can find, extract, or replace every string that matches.

You have already written patterns like this once, back in the shell. When you typed ls *.txt, the * was a pattern that matched every filename ending in .txt, and topic_0?_* matched topic_00_... through topic_09_.... Those shell patterns are called globs, and a regex is the same idea turned up to eleven: a richer pattern language that works on any text, not only filenames.

Regex has a well-earned reputation for looking like someone fell asleep on the keyboard.

The syntax is dense, but it is not deep. It is a handful of small pieces you can learn one at a time, and that is exactly how we will learn them.

From globs to regex

In a glob, two characters did most of the work. * matched any run of characters, and ? matched exactly one character, so the glob ?.txt matched a.txt but not ab.txt. Regex keeps both ideas but respells them, and that respelling is the first trap: the same symbols mean different things here.

Here is the translation for the two you already know:

• The glob ? (any one character) becomes . in regex.
• The glob * (any run of characters) becomes .* in regex.

So the shell glob *.txt becomes the regex .*\.txt. Notice we had to write the literal dot as \., because a bare . now means any character, and putting a backslash in front of a metacharacter turns off its special meaning. Everything past this point is the vocabulary globs never had: ways to say a digit, three of these, this or that. The payoff is that a regex can find a phone number or an email address, which no glob can do.

Searching for a pattern

Python’s regexes live in the re module, which ships with Python, so there is nothing to install. The workhorse is re.search(pattern, text), which looks for the pattern anywhere inside the text. We start with the simplest possible pattern, a plain literal string with no special characters, which just asks does this substring appear?:

>>> import re
>>> re.search("cat", "the cat sat on the mat")
<re.Match object; span=(4, 7), match='cat'>
>>> re.search("dog", "the cat sat on the mat")
>>>

Read the two results. When the pattern is found, re.search hands back a match object that reports where it matched: span=(4, 7) says the match runs from index 4 up to (but not including) 7, and match='cat' shows the exact text it found. When the pattern is not found, re.search returns None, which is why the second line printed nothing at all.

That match-or-None behavior is what makes re.search easy to drop into an if, because a match object is truthy and None is falsy:

>>> if re.search("cat", "the cat sat"):
...     print("found it")
...
found it

So the first job a regex does, is there anything shaped like this in here?, is already a one-liner.

The building blocks

A literal pattern only finds text you could have found with in or str.find. Everything interesting comes from metacharacters: symbols that stand for a category of character rather than for themselves. The most useful one is \d, which matches any single digit:

>>> re.search(r"\d", "order #42 shipped")
<re.Match object; span=(7, 8), match='4'>

The pattern \d matched the 4 at index 7, the first digit, since re.search stops at the first match. Notice the r in front of the string: r"\d" is a raw string, which tells Python to leave the backslash alone instead of reading \d as an escape sequence. Always write regex patterns as raw strings; it is the convention, and it heads off a whole class of backslash bugs.

One digit is rarely what we want; we usually want the whole number. For that we need a quantifier, a symbol that says how many of the previous thing to match. The quantifier + means one or more, so \d+ grabs the longest run of digits:

>>> re.search(r"\d+", "order #42 shipped")
<re.Match object; span=(7, 9), match='42'>

Now the match is 42, spanning indices 7 to 9, because + kept going as long as it saw digits. Here are the pieces worth memorizing, and it is a short list:

• Character classes stand for a category: \d a digit, \w a word character (letter, digit, or underscore), \s whitespace, and . any character at all.
• Custom classes in square brackets match any one character you list: [aeiou] matches a vowel, [a-z] any lowercase letter, and [^0-9] any character that is not a digit (a leading ^ inside the brackets negates the class).
• Quantifiers say how many of the thing before them: + is one or more, * is zero or more, ? is zero or one (optional), {3} is exactly three, and {3,5} is between three and five.
• Anchors pin the match to a position: ^ means the start of the string and $ the end, so ^\d+$ matches text that is all digits and nothing else.

Two quick examples pin these down. A custom class pulls every vowel out of a word, and ? makes a character optional, which is how one pattern matches two spellings of the same word:

>>> re.findall(r"[aeiou]", "regular")
['e', 'u', 'a']
>>> re.findall(r"colou?r", "color and colour")
['color', 'colour']

The u? says a u may or may not be here, so colou?r matches color and colour both, which is the kind of thing you simply cannot say with a glob. (Those examples used re.findall, which we meet properly next.)

Extracting every match

re.search finds the first match; the everyday scraping job is to find all of them. That is re.findall(pattern, text), which returns a plain list of every matching substring, no match objects, just the strings. Suppose we scraped a contact page and have this line of text, and we want every phone number in it:

>>> text = "call 909-621-8000 or 213-555-0199 today"
>>> re.findall(r"\d{3}-\d{3}-\d{4}", text)
['909-621-8000', '213-555-0199']

Read the pattern left to right, because it is built entirely from pieces we just met: \d{3} is three digits, then a literal -, then \d{3}, another -, and \d{4} for the last four digits. That is the exact shape of a US phone number, and findall returned both of them as a list, ready to loop over or write to a file. This is the payoff: a pattern you can read like a sentence, doing in one line what would take a page of if statements and string slicing.

Often we do not want the whole match, only one piece of it, like the area code. Wrap the piece you want in parentheses to make a group, and findall returns just the group:

>>> re.findall(r"(\d{3})-\d{3}-\d{4}", text)
['909', '213']

The parentheses around (\d{3}) mark the area code as the part we care about; the rest of the pattern still has to match, but findall now hands back only what was inside the parentheses. Groups are how you turn matching into extracting: match the whole shape, then capture the slice you want.

The same tools pull email addresses out of the same kind of scraped text. An email is some word characters, an @, more word characters, a dot, and a bit more, which is almost a direct transcription into regex:

>>> re.findall(r"\w+@\w+\.\w+", "reach me at ada@example.com or grace@navy.mil")
['ada@example.com', 'grace@navy.mil']

Notice the \. again: we want a literal dot before com, so we escape it, because a bare . would match any character and happily accept adaXexample. This pattern is deliberately loose, since real email addresses are a swamp of edge cases, but for pulling addresses off a page you scraped, loose and readable beats perfect and unreadable.

Cleaning text with substitution

Finding is half the job; the other half is changing what you find. re.sub(pattern, replacement, text) replaces every match with the replacement string and returns the new text. The classic use is redaction, scrubbing every phone number out of a document before you share it:

>>> re.sub(r"\d{3}-\d{3}-\d{4}", "[redacted]", text)
'call [redacted] or [redacted] today'

Both numbers are gone, each replaced by [redacted], and the rest of the line is left untouched. This is the boring-task automation the whole course is about: the same three-line pattern that found the phone numbers now removes them, and it would run just as happily over a thousand files as over this one string. For a gentler, full tour of these functions, the Automate the Boring Stuff regex chapter and the Python re docs are the two references to keep open.

Why LLMs stumble

There is a reason this topic sits right after we built things with large language models. If you paste a chunk of text into an LLM and ask for the regex that extracts the dates, it will often nail it, and just as often hand you a pattern that looks right but matches the wrong things. Regex is one of the tasks where a confident-looking answer is most likely to be subtly wrong, so the habit is to test the pattern rather than trust it.

The better question is why regex is hard for a model that can write whole essays. The honest answer is math. Working out what a pattern can and cannot match, and how long that takes to check, is the subject of formal-languages and complexity theory: the kind of question you meet in an upper-division CS course, not in a coding tutorial.

That is a running joke of this course, and it is also true: coding is not the same thing as computer science. You can get remarkably far writing code, and an LLM can get further still, but the questions about why code behaves the way it does, which regexes are fast and which are effectively impossible, are answered with math and not with more typing. You do not need any of that theory to use regex today; you need it the day you want to know why the tool has the shape it does.

Looking forward

One last honest note, and it may be the most useful thing on this page. Nobody memorizes regex, and nobody reads a dense one at a glance.

The professional workflow is to build a pattern against real text and watch what it catches, using a tester like regex101.com, which highlights every match as you type and explains each piece of the pattern in plain English. Write the pattern there, paste in the text you actually have, and adjust until the highlights are exactly the strings you want; then copy the pattern into your re.findall.

You now have the whole core: re.search to test whether a pattern is present, re.findall to extract every match, re.sub to replace them, and the small vocabulary of classes, quantifiers, and anchors that patterns are built from. Whenever you scrape a page, this is how you turn a wall of text into the phone numbers, prices, and links you actually came for. Next class we look at the tidy formats, JSON and its neighbors, that let a site hand you its data already structured, so that sometimes, mercifully, you do not need a regex at all.