Project 2: Scraping eBay

Dilbert comic: a marketing manager says he doesn't see why engineers get paid more than marketing professionals; Dilbert replies that engineers designed and built every important part of modern civilization while all marketing did was misrepresent it; the manager insists 'you need both,' and Dilbert answers 'you really don't.'

Your second project is to scrape a real, live website, eBay, and turn its search results into clean, structured data. You’ll write a command-line program that takes a search term, downloads the first pages of eBay results, pulls out each listing’s name, price, shipping, and a few other fields, and saves everything to a JSON file you can actually work with.

The clearest way to think about this project is as a compiler, the same way we thought about the markdown compiler. There, we converted Markdown into HTML. Here, we convert eBay’s HTML into JSON. That is what most programming projects turn out to be: taking data in one format and converting it into another, more useful one. The comic is a joke, but the point stands: this project is real engineering, building a small tool that reliably turns a messy webpage into data you can use.

Walkthrough (optional). This project is adapted from Mike Izbicki’s course, and he recorded a roughly two-hour pair-programming video that builds most of the code alongside you. Pairing with a more experienced programmer is one of the best ways to level up, so it’s worth watching if you get stuck. As you follow along, keep two questions in front of you: how do you debug code when it doesn’t do what you expect, and how do you structure code with doctests so you can prove each piece works?

Due: Wednesday, October 21, see the schedule.

Learning objectives

  1. understand how web scraping works
  2. complete a Python project from scratch
  3. integrate your Python knowledge with what you know about HTML and JSON

Instructions

The project has three parts: write the scraper, run it on a few real searches, and publish the result as a GitHub repository.

Part 1: the scraper. Create a Python file named ebay-dl.py. It should:

  1. Use the argparse library to read a search term from the command line.
  2. Use the requests library to download the first 10 pages of results for that search term.
  3. Use bs4 (Beautiful Soup) to extract every item in the search results.
  4. Build a Python list of the extracted items, where each entry is a dictionary with the following keys:
    1. name: the name of the item.
    2. price: the price of the item in cents, stored as an integer. Never use floats to store monetary values, because floats can’t be represented exactly in computers. If several prices are listed (for example, $54.99 to $79.99), you may pick either one.
    3. status: a string stating the item’s condition, such as "Brand New", "Refurbished", or "Pre-owned".
    4. shipping: the cost of shipping in cents, stored as an integer; if the item has free shipping, this value is 0.
    5. free_returns: a boolean value for whether the item has free returns.
    6. items_sold: the number of items sold, as an integer.
  5. Use the json library to save the list as a JSON file named SEARCH_TERM.json, where SEARCH_TERM is replaced by the search term passed in on the command line.

Note: Not every listing on eBay will have an entry for each of the fields above. If there is no entry, your dictionary must still contain the associated key, and the value should be None.

Note: eBay actively tries to block scrapers, so a bare requests.get may return a “robot check” page instead of real results. Set a browser-like User-Agent header (search the requests docs for how), space your requests out, and scrape gently, you only need a few searches. If a page still comes back blocked, wait and try again or switch networks.

Part 2: run it on real searches. Run your ebay-dl.py file on three search terms of your choice, generating three different JSON files. At least one of these search terms must contain a space (for example, drill press, stuffed animal, or claremont mckenna).

Part 3: publish it. Create a GitHub repository that:

  1. contains your ebay-dl.py file,
  2. contains your three JSON files, and
  3. contains a README.md file explaining:
    1. what your ebay-dl.py file does,
    2. how to run your ebay-dl.py file, using Markdown code block(s) (and not inline code) to show the exact commands that generate the three JSON files in your repo, and
    3. a link to this project page.

Grading rubric

This project is worth 18 points. Your grade starts at 18/18, and you lose the listed points for each checkbox you do not complete.

Parts 2 and 3 (running on three searches and publishing the repo with its README.md) are how you package and submit the project; the points above are earned on the Part 1 criteria, checked against the JSON your published repo produces.

Extra credit

  1. Modify ebay-dl.py so that it accepts a new command-line flag, --csv. Whenever this flag is specified, the output file should be saved in CSV format instead of JSON format.
  2. Generate three CSV files in addition to the three JSON files, and include them in your repo.
  3. Update your README.md file to include instructions and examples for using the --csv flag.

Submission

Submit a link to your GitHub repository on Gradescope.