Project 1: Markdown Compiler

Comic titled 'The only valid measurement of code quality: WTFs/minute.' Two doors: one labeled 'good code' with a couple of 'WTF' notes, the other labeled 'bad code' covered in 'WTF' notes.

Almost nobody writes HTML by hand anymore. Instead we write programs that generate the HTML for us, and run those programs whenever a page needs to change. In this project you’ll write one of those programs: a Python program that reads a Markdown file and compiles it into HTML.

Markdown is a much smaller language than HTML, and you have already read a great deal of it. Reddit comments are written in Markdown, data-science reports get written in R Markdown, and every README.md and issue on GitHub, including the ones for your labs, is GitHub-flavored Markdown that GitHub compiles to HTML before showing it to you. Your program will do the same job on a smaller scale: read Markdown like # Header and **bold** and write out the matching <h1> and <b> tags. This is where the string slicing, the find method, and the loops-over-characters from the Python readings start paying off on a problem people actually get paid to solve.

A financial aside

GitHub runs one of these compilers on every page: each README.md and every issue is Markdown that GitHub renders to HTML for you. Roughly what did that feature cost to build? Say an entry-level engineer takes about a week, and entry-level engineers at GitHub make around $148k a year; that is about $3000 of salary for the same program you’re writing for homework. Was it a good deal for GitHub? They have well over 100 million users, so the feature cost a small fraction of a cent per user, paid once, and GitHub then takes in a few dollars per user per month. The distance between what a feature costs to build and what it brings in once it is running for millions of people is a big part of why software engineers get paid what they do.

One more reason to take this seriously: employers really do read your GitHub profile when they hire, and a working compiler with green test badges is a good thing for them to find there.

Fork and clone the starter repository: github.com/rtealwitter/markdown-compiler. All of your work happens inside your fork of this repo.

Due: Wednesday, October 7, see the schedule.

Learning objectives

  1. understand the Markdown language
  2. understand string manipulation in Python
  3. understand how compilers translate from one programming language into another

Instructions

The starter code is a package of small functions, and, just like in lab, every function comes with its own doctests. Solve it the way you solve a lab: make the doctests pass one function at a time instead of trying to write the whole compiler at once. Splitting a big problem into small, separately testable pieces is one of the real skills this project is teaching, and it is one of the things that separates senior programmers from junior ones.

Step 1: make the tests pass. Your repository ships with three GitHub Actions, and right now they all fail. Fix the code in markdown_compiler/ until each one passes:

  • doctests runs every function’s doctests with python -m doctest. This is the bulk of the work: each function in markdown_compiler/util/line_functions.py and markdown_compiler/__init__.py has doctests that show exactly what it should return for a given input.
  • flake8 checks that your code is clean and free of style errors (unused variables, inconsistent indentation, and the like).
  • command_line installs your package and runs the markdown-compiler command from end to end.

You do not need to edit the Actions themselves, but reading them is worth your time: they are short, and they show you exactly how your code is being tested. One thing the command_line test needs is the --add_css flag wired up in markdown_compiler/__main__.py; there is a FIXME comment in that file showing you where.

Step 2: generate a page and screenshot it. Once the tests pass, run your compiler on the example document, both without and with CSS:

$ markdown-compiler --input_file=example/README.md
$ markdown-compiler --input_file=example/README.md --add_css

Each command writes an HTML file next to the input, at example/README.html. The second command overwrites the first, so open the file in your browser and take a screenshot after each run. Your repository’s own README.md already has two image tags in it, one pointing at example/example.png and one at example/example-css.png, waiting for exactly these screenshots. Save the plain screenshot at example/example.png and the CSS one at example/example-css.png, then commit them, so they show up on your repository’s front page on GitHub.

The --add_css flag adds two stylesheets to the generated page, so the same Markdown produces a very different-looking result. That flexibility, one source and many looks, is a big reason people generate HTML from Markdown instead of writing it by hand.

Don’t wait until every function is finished to run these commands. Run them as you complete each function and watch the output improve; a broken function is much easier to spot in a half-rendered page than in a wall of doctest failures.

Grading rubric

This project is worth 16 points. Your repository has three GitHub Actions (doctests, flake8, and command_line), each worth 4 points, and the two screenshots are worth 4 points together:

You only earn the screenshot points if the HTML shown in the screenshots is correct.

Extra credit

Right now the compiler has no support for Markdown lists. For example, the Markdown

1. this
2. is
3. a
4. list

currently compiles to

1. this 2. is 3. a 4. list

instead of

<ol><li>this</li><li>is</li><li>a</li><li>list</li></ol>

For 1 point of extra credit, so the maximum score is 17/16, add list support:

  1. Add 3 doctests to the compile_lines function that contain Markdown lists. These should be good doctests that reasonably test that the list functionality works; come talk to us if you’d like feedback on whether yours are good ones.
  2. Modify your code so that these doctests pass.

Submission

Submit on Gradescope:

  1. the URL to your GitHub repository.

If you completed the extra credit, say so in your submission so we know to grade it.