Lab: Selector Golf

This is the second of this week’s two labs; the other is Analyzing Trump Tweets. That one drills the JSON half of scraping, where the data already arrives as neat lists and dictionaries. This one drills the other half, the messy half: pulling data out of a web page nobody formatted for you, using Beautiful Soup and the CSS selectors you learned in week one.

Back in the CSS reading we made a promise: the selectors you were using to color your own pages, things like div > .price and a[href], were the very selectors you would later use to extract data from pages you did not write. This lab is where that promise pays off. By the end you will have written the exact function Project 2 is built around, the one that turns a page of listings into a clean list of dictionaries.

Download: lab_selectors.py and the practice page sample_listings.html

Reduce, reuse, recycle: the selectors you wrote to make prices teal in Project 0 are the exact ones you’ll write to make prices yours here. Same syntax, different intentions.

From coloring to counting

Every function in this lab has the same first move. You take an HTML string, hand it to BeautifulSoup, and get back a soup object you can search with .select:

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'html.parser')
soup.select('.price')      # a list of every tag matching the selector

.select takes a CSS selector string, the same kind you wrote in your stylesheets, and returns a list of every tag that matches, top to bottom. The selector chooses which tags, and once you have the list you loop over it to pull out what you want. Nothing about the selector language changed on the trip from CSS to Python; .select('div > a') here matches exactly what div > a { ... } styled there.

Download sample_listings.html into the same folder as your code. It is a small, made-up search-results page, five items deep, each with a title, a price, a condition, and a link; three of the five advertise free shipping. It stands in for the eBay page you will scrape for real in Project 2, small enough that you can hold the whole thing in your head. Open it in your browser and in your editor, and keep it in front of you for the rest of the lab.

Making soup

Loading the page is two lines you have basically seen before, one to read the file into a string and one to parse it:

>>> from bs4 import BeautifulSoup
>>> html = open('sample_listings.html').read()
>>> soup = BeautifulSoup(html, 'html.parser')

That first line is the file-reading open(...).read() from the files unit, and html is now an ordinary str. Everything after it is Beautiful Soup’s job. Ask the soup for all the prices and read each one’s text:

>>> for tag in soup.select('.price'):
...     print(tag.text)
...
$49.99
$12.50
$8.99
$23.00
$34.95

Five prices, in page order, because the page has five items. .select did the finding; .text pulled the words out from between each tag’s opening and closing.

Two lines of setup and every selector you already know starts working on a page you didn’t write. There is no catch. (There is always a catch. It’s whitespace, and we’ll get to it.)

Selector golf

Before you extract anything, build the one habit that makes scraping feel less like guessing: given a selector and a page, predict how many tags it matches, then check yourself. It is the same drill as the selector quiz from the CSS unit, where you counted matches on paper; there you confirmed your guess in the browser console with document.querySelectorAll, and here you confirm it in Python with len:

>>> len(soup.select('.price'))
5

Play a round of selector golf on sample_listings.html. For each selector below, predict the count first, then run len(soup.select(...)) and see if you were right. No peeking at the page’s source until you have committed to a number.

soup.select('.item')            # every listing
soup.select('.badge')           # every badge, of any kind
soup.select('.free-shipping')   # just the free-shipping badges
soup.select('.sponsored')       # careful with this one
soup.select('#results > li')    # direct children of the results list
soup.select('.item + .item')    # each item that follows another item
soup.select('li.item > a')      # the title link inside each item

Two of these are traps, both carried straight over from CSS. .sponsored matches two elements, not one: the sponsored <li> and the <span class="badge sponsored"> inside it, because a class can live on any tag and this page happens to use the name twice. And .item + .item matches four, not five, because the very first item has no item before it to follow. If your prediction missed either, that is the lab working; the miss is where the learning is.

There are only two hard problems in web scraping: naming CSS classes, cache invalidation, and off-by-one selector counts.

The first two functions in lab_selectors.py, count_matches and select_texts, are this exact skill written down: one returns len(soup.select(selector)), the other returns the .text of every match. Write them and you have the core scraping move; everything else is a variation on it.

From tags to fields

The point of scraping is never the tags themselves, it is the data inside them, and two operations pull out almost everything. tag.text gives you the words between a tag’s opening and closing, and indexing with a key like tag['href'] gives you an attribute’s value:

>>> link = soup.select('a.title')[0]
>>> link.text
'Mechanical Keyboard, RGB, Brown Switches'
>>> link['href']
'/itm/101'

select_attr, extract_titles, and first_link are built out of exactly those two moves. One trap to watch, the whitespace we promised you: real pages put newlines and indentation inside their tags, so tag.text often comes back padded with spaces. Calling .strip() on it is a cheap, reliable habit, and it is why the extractors in this lab strip their text.

Prices need one extra step, and it is a discipline Project 2 will grade you on. A price reads off the page as the string '$49.99', but you must store money as a whole number of cents, as an int, never a float. So '$49.99' becomes the integer 4999: strip the $, and turn the dollars into cents.

>>> dollars = '$49.99'.replace('$', '')
>>> int(round(float(dollars) * 100))
4999

That is what extract_prices_cents returns, a list of integers.

$19.99 stored as a float is 19.990000000000002 waiting to ruin someone’s checkout. Store money as a whole number of cents, or the auditors will find you.

One dict per item

Now assemble the pieces into the shape a real program wants. A price by itself is useless; what you want is this item’s name next to this item’s price next to this item’s link, one record per listing. That record is a dictionary, and the whole page is a list of them.

The move is a loop over the items, and inside each item a smaller .select to find that item’s own title and price:

soup = BeautifulSoup(html, 'html.parser')
listings = []
for item in soup.select('.item'):
    title = item.select('.title')[0]
    price = item.select('.price')[0]
    listings.append({
        'name': title.text.strip(),
        'price_cents': ...,     # the price as an int of cents
        'url': title['href'],
    })

Notice that item.select('.title') searches within one listing, not the whole page, which is how the loop keeps each item’s fields from bleeding into the next. That is extract_listings, the last and most important function in the file, and it is worth seeing what you have built once it passes:

>>> extract_listings(open('sample_listings.html').read())
[{'name': 'Mechanical Keyboard, RGB, Brown Switches', 'price_cents': 4999, 'url': '/itm/101'}, ...]

A list of clean dictionaries, one per item, each with a name, a price in cents, and a link. That is not a warm-up for Project 2; it is Project 2’s core, minus the parts around the edges.

A scraper is just a for loop with strong opinions about .text.

There, the eBay project asks you to add status, shipping, free_returns, and items_sold to each dictionary, to walk ten pages instead of one string, and to hand the finished list to the json library to save. But the shape is this one exactly: .select the items, loop, pull each field with .text and [...], build a dict, collect the dicts. count_free_shipping is a small taste of the extra fields; extract_listings is the spine. Get these passing and the project stops looking like a wall and starts looking like homework you have already done once.

Submitting

This lab is auto-graded, and it uses the loop every doctest lab in this course shares. Open lab_selectors.py, fill in one function body at a time, and run the doctests from your terminal until the command falls silent:

$ python3 -m doctest lab_selectors.py

Silence means every test passed. While tests are still failing, the command prints each failing example with what it Expected and what it Got, so you always know which function to fix next; add -v if you want to watch the passing tests tally up too. Write, run, read the failures, fix, and repeat until it goes quiet.

Then submit the way you always do, by committing the finished file to your repository and pushing it to GitHub:

$ git add lab_selectors.py
$ git commit -m 'complete lab_selectors.py'
$ git push

Your repository runs the same doctests automatically on every push, and the tests badge turns from red to green once they all pass. That green badge is the whole grading mechanism: the tests either pass or they do not, and you resubmit until they do. When the badge is green, submit your repository’s URL on Gradescope.