Lab: Redacted

The reading handed you the vocabulary of regular expressions: re.search to ask is it in here?, re.findall to pull out every match, re.sub to replace what you find, and the small alphabet of classes, quantifiers, and anchors that patterns are built from. This lab is where that vocabulary becomes muscle memory. You will write about a dozen patterns against messy, real-looking text until each one does exactly what its doctests demand.

The job splits neatly in two, and the split is the whole point. Half of these functions are offense: the data you want is buried in a wall of text, and you write a pattern to extract it, the phone number on the contact page, the hashtags in a tweet, the area code hiding inside the phone number. The other half are defense: the data you want gone is buried in a wall of text, and you write a pattern to redact it, the email addresses, the Social Security numbers, all but the last four digits of a card. It is the same move you made when you cracked passwords, and the same instinct you will need when we reach SQL injection: the person who can find the pattern is the person who can also scrub it.

There is a short bridge from this lab to the docchat project. That project’s grep tool loops over every line of a file and calls re.search(pattern, line) to decide whether to keep it, which means the patterns you drill here are literally the ones a user will type at the chat> prompt to search their documents. Project 3 was assigned two weeks ago, so if grep felt like magic then, this is the lab where the magic turns into \d{3}-\d{3}-\d{4}.

One habit before you start, and it is the single most useful thing on this page. Do not write regex in your head, and do not trust the regex an LLM hands you. The reading said it and it bears repeating: regex is the task where a confident-looking answer is most likely to be subtly wrong. The professional workflow is to build and watch: open regex101.com, paste in the text you actually have, and type your pattern while the site highlights every match live and explains each piece in plain English. Adjust until the highlights are exactly the strings you want, then copy the pattern into your function. Test the pattern; do not trust it.

Starter code: github.com/rtealwitter/lab-regex

The starter is a single file, lab_regex.py, holding about a dozen short functions whose bodies are blank. Each has a one-line description and a set of doctests; your job is to write the body so that every doctest passes. Work one function at a time and run the tests from your terminal:

$ python3 -m doctest lab_regex.py

Silence means every test passed. While a test is failing, the command prints the example, what it Expected, and what it Got, so you always know exactly which pattern to fix next. Write every pattern as a raw string, r"\d+", never "\d+", which is the convention and heads off a whole class of backslash bugs. None of these functions is more than a few minutes of work once the pattern is right; the work is getting the pattern right.

Warm-up: search, findall, sub

The three functions in the warm-up exist to lock in the difference between the three verbs, using the simplest pattern there is, \d, a single digit, so that nothing distracts from what each function returns.

• has_digit(text) uses re.search to answer a yes/no question. re.search returns a match object when it finds the pattern and None when it does not; a match object is truthy and None is falsy, so wrapping it in bool(...) turns it into the True/False the doctest wants. This is the exact shape grep uses to decide whether a line matches.
• find_numbers(text) uses re.findall to return a list of every run of digits. The quantifier + (one or more) is what makes \d+ grab the whole number 10 instead of the two separate digits 1 and 0.
• mask_digits(text) uses re.sub to replace every digit with an X and hand back the new string.

Three verbs, three return types: a bool, a list, a string. Once you can feel that difference, the rest of the lab is just fancier patterns plugged into the same three functions.

🖼️ Meme: the three-way Spider-Man pointing meme, the three Spider-Men labeled re.search, re.findall, and re.sub, all pointing at the same raw string r"\d+".

Extraction

Now the patterns get real. Extraction is the scraper’s daily bread: the page has the one thing you want and a thousand characters you do not, and you write a pattern shaped like the thing.

find_phone_numbers is the pattern straight from the reading, \d{3}-\d{3}-\d{4}, read left to right as three digits, dash, three digits, dash, four digits. find_urls is a looser one, where https? makes the trailing s optional so a single pattern catches both http:// and https://.

find_hashtags and find_mentions add the move that turns matching into extracting: a capture group. You match the whole shape, #\w+, but wrap the part you actually want in parentheses, #(\w+), and re.findall hands back only what was inside them, the tag without its #. These two are where the recurring dataset of this course earns its keep:

>>> find_hashtags('Big rally tonight! #MAGA #America #MAGA')
['MAGA', 'America', 'MAGA']

Notice the list keeps duplicates and keeps order, findall reports every match as it appears, it does not deduplicate. area_code uses the same capture-group idea through re.search instead: one phone number goes in, and .group(1) reads out the piece the parentheses marked.

One warning that is really the whole lesson of the reading in miniature. Your mention pattern, @(\w+), will happily “find a handle” inside an email address, because ada@example.com also contains an @ followed by word characters. Your doctests do not include that trap, so they will pass, but it is exactly the kind of quiet wrongness that makes people say a regex “works” when it does not. Build it on regex101 against text that contains both a mention and an email, and watch what lights up.

🖼️ Meme: a triumphant gold miner holding one gleaming nugget over a mountain of dirt, captioned “when re.findall finally returns something that isn’t [].”

Redaction

Finding is offense; scrubbing is defense, and it is the same three functions pointed the other way. Redaction is re.sub with a pattern shaped like the thing you want gone, and it is the boring-but-critical task every organization that handles personal data has to automate: strip the PII out of a document before it leaves the building.

redact_emails replaces the email shape from the reading with the literal string [REDACTED]. redact_ssns matches a US Social Security number, \d{3}-\d{2}-\d{4}, and mind the 2 in the middle, an SSN is not a phone number, then masks it as XXX-XX-XXXX. mask_last_four is the receipt-printer classic: keep only the last four digits of a sixteen-digit card and star out the rest.

>>> redact_ssns('SSN 123-45-6789 on file')
'SSN XXX-XX-XXXX on file'

Here is the sobering half. A redactor is only as good as its pattern: every format you forgot to match is a leak. If your SSN pattern misses the ones someone typed with spaces instead of dashes, those numbers sail straight through into the “safe to share” copy. This is why real redaction pipelines are tested against piles of adversarial examples, and why “the LLM said this regex catches all of them” is not a sentence you get to trust. Test the pattern; do not trust it.

🖼️ Meme: a document with a thick black bar over every single line, captioned “my search history, now GDPR-compliant thanks to re.sub.”

Validation

The last function flips the question one more time. Extraction and redaction ask is the pattern in here somewhere?, the job re.search and re.findall are built for. Validation asks something stricter: is this entire string nothing but the pattern? That is what anchors are for. ^ pins the match to the start of the string and $ to the end, so ^...$ matches only when the whole string, start to finish, is the shape you described.

is_valid_email is the one anchored function, and the anchors are the entire assignment. Without them, re.search finds a match anywhere, so a string like ada@example.com and some junk would look valid, there is a perfectly good email sitting at the front of it:

>>> is_valid_email('ada@example.com')
True
>>> is_valid_email('ada@example.com and some extra text')
False

The second call must be False, and the only thing standing between you and a bug is the $ that refuses to let the match stop before the end of the string. Anchored validation is the front door of every login form and comment box on the internet, and getting it wrong is how bad input gets in, precisely the crack that SQL injection, later in the course, pries open. Validate the whole thing, or you have not validated it at all.

Submitting

This lab uses the same loop as every doctest lab in the course. Work until the terminal falls silent:

$ python3 -m doctest lab_regex.py

Silence means every doctest passed; any output names the function, the Expected value, and what it Got, so you always know the next pattern to fix. Then upload your work, which is always the two steps of committing and then pushing:

$ git add lab_regex.py
$ git commit -m 'complete lab_regex.py'
$ git push

Your fork ships with a GitHub Actions workflow that reruns these exact doctests on every push. Enable Actions on your fork, and after your next push the tests badge flips from red to green once all of them pass. That green badge is the whole grade: there is no partial credit, so keep filling in patterns and resubmitting until every function is done and the badge is green, then submit the URL of your fork on Gradescope.

And when you finish and immediately forget every pattern you just wrote, that is not a bug. Nobody memorizes regex; the skill is not recalling the pattern but building it against real text until the highlights are right, and you just did that a dozen times.