JSON & Alternatives

Two of the tools we have already built reach out into the world and bring data back. The scraper from web scraping pulls pages off the internet with requests.get, and the LLM programs behind docchat sent a prompt to an API and got an answer. Neither one handed us back a Python dictionary. Both handed us back text, in a format called JSON, and we called json.loads on it to turn that text into something we could use. Today we give that step the full treatment.

A running program keeps its data in memory as dicts, lists, numbers, and strings. The moment that data has to leave the program, to be saved to a file, sent across a network, or read by a different program written in a different language, it has to be flattened into plain text that anything can parse. JSON (JavaScript Object Notation) is the text format almost everyone agreed to use for that job. It is the closest thing programming has to a common language for shipping structured data around, and by the end of today you will read and write it without thinking.

The shape of JSON

The reason JSON felt familiar when we met it while scraping is that it is almost exactly Python’s dicts and lists, written out as text. Here is a small JSON document, the same kind our scraper pulled back: a list of tweets, each one an object with a text and a username:

[
    {"text": "hello", "username": "Trump"},
    {"text": "world", "username": "Obama"},
    {"text": "hola",  "username": "Obama"}
]

the square brackets are a list (JSON calls it an array), each pair of curly braces is a dict (JSON calls it an object), and the keys and text are strings in double quotes. If that looks like Python, it should, because the mapping between the two is direct:

  • an object {...} becomes a Python dict
  • an array [...] becomes a Python list
  • a string, always in double quotes, becomes a str
  • a number becomes an int or a float
  • true and false become True and False
  • null becomes None

The differences are small and worth memorizing, because each one is a way to write JSON that Python will refuse to read:

  • strings and keys use double quotes, never single
  • the booleans and null are lowercase: true, not True
  • there is no trailing comma after the last item
  • there are no comments anywhere

Hold onto that last one, because it is most of the reason the alternative formats exist.

Reading JSON

JSON almost always arrives as a string: the body of a web response, or the contents of a file. Turning that string into Python objects is one function, json.loads, which you can read as “load string”:

>>> import json
>>> jsontext = '''
... [
...     {"text": "hello", "username": "Trump"},
...     {"text": "world", "username": "Obama"},
...     {"text": "hola",  "username": "Obama"}
... ]
... '''
>>> data = json.loads(jsontext)
>>> type(data)
<class 'list'>
>>> data[0]
{'text': 'hello', 'username': 'Trump'}
>>> data[0]['username']
'Trump'

json.loads parsed the text and handed us an ordinary Python list of dicts. From there nothing about it is special: we index it with [0], look up keys with ['username'], and loop over it like any other list. Notice that Python printed the value back with single quotes; once the data is inside Python it is a normal str, and the double-quote rule only ever applied to the text form.

This is the exact move the tweets dataset from web scraping is built on: a folder of master_*.json files, each holding a list of tweets, that you load and then count. Counting how many came from one account is a one-liner once the JSON is Python:

>>> sum(1 for tweet in data if tweet['username'] == 'Obama')
2

When the JSON lives in a file instead of a string, use json.load (no s) on an open file, the same way we read text files in strings and files:

with open('tweets.json') as f:
    data = json.load(f)

The naming is worth saying out loud once: the functions with an s, like loads, work on strings; the ones without, like load, work on open files.

Writing JSON

The reverse direction is just as common: you have Python data and you want to save it or send it somewhere. json.dumps (“dump string”) turns Python objects back into a JSON string:

>>> config = {'title': 'My Blog', 'port': 8080, 'debug': True, 'tags': ['python', 'web']}
>>> print(json.dumps(config, indent=2))
{
  "title": "My Blog",
  "port": 8080,
  "debug": true,
  "tags": [
    "python",
    "web"
  ]
}

json.dumps walked the dict and produced the text form, turning True into true on the way out, exactly reversing what reading did. The indent=2 argument is the difference between a readable block and a single unbroken line; leave it off and you get compact JSON with no spaces, which is what you send over a network when you want to save bytes.

That gives you a debugging trick you will use all term. Any time you are unsure what shape some nested data actually has, print it as indented JSON and look:

print(json.dumps(data, indent=2))

It works on anything built from dicts, lists, strings, numbers, and booleans, and it is the fastest way to see the structure of a response you did not write yourself.

To write to a file instead of a string, use json.dump (no s) with an open file:

with open('config.json', 'w') as f:
    json.dump(config, f, indent=2)

Same rule as before, the s means string, now running in the writing direction. One value is worth watching on the way out. Python’s None becomes JSON null, and nothing else does:

>>> print(json.dumps({'author': None}))
{"author": null}

The lingua franca

We keep running into JSON because almost every system that hands data to another system hands it over as JSON. When your program asked the LLM a question in docchat, the API did not reply with plain English; it replied with a JSON object, and the model’s answer was one string buried a few keys deep inside it. Pulling that string out is the same indexing you just practiced:

reply = response['choices'][0]['message']['content']

the reply parsed into nested dicts and lists, and we dug down through the keys to the one piece we wanted.

That is the pattern for the rest of the course. The docchat and agents projects are loops of sending JSON to a model and reading JSON back out. When we build the Twitter clone’s backend, the server will answer the browser in JSON too. JSON is worth knowing cold because it is the format nearly everything you build from here on will speak.

YAML

JSON is built for programs talking to programs, and it starts to grate the moment a human has to write it by hand. No comments, double quotes on every single key, and a syntax error if you leave one comma after the last item: fine for a machine, tedious for a person editing a settings file. So for configuration that people edit, the world reached for friendlier formats, and the most common is YAML.

Here is the same config as before, written as YAML:

# a comment, at last
title: My Blog
port: 8080
debug: true
tags:
  - python
  - web

no braces and no quotes, one key: value per line, nesting shown by indentation the way Python shows blocks, and a list written as - bullets. That # comment is the feature JSON refuses to have, and it is most of why config files reach for YAML. Loading it into Python needs a third-party library, pyyaml, and the function is yaml.safe_load:

>>> import yaml
>>> config = yaml.safe_load(open('config.yaml'))
>>> config
{'title': 'My Blog', 'port': 8080, 'debug': True, 'tags': ['python', 'web']}

The same dict we started with, now round-tripped through a friendlier text form. Every JSON document is also, almost, valid YAML, which is the joke this meme is making:

'Change My Mind' meme: a man seated at a table behind a sign that reads 'YAML is just Python-like JSON.'

It is close to true. YAML is a superset that adds comments, indentation, and a few conveniences on top of the same object, array, string, and number model.

Where do you meet it? The GitHub Actions file that auto-grades your labs is YAML, and so are Docker Compose and Kubernetes config. Its friendliness does come at a price: YAML tries to guess the types of your values, and sometimes it guesses wrong. The classic surprise is that the bare word no is read as the boolean False:

>>> yaml.safe_load('country: no')
{'country': False}

This is known as the Norway problem, because a file listing country codes turns the code NO into False with no error to warn you. When YAML surprises you like this, fall back on the trick from earlier: load the file and print(json.dumps(config, indent=2)) to see the values Python actually built.

TOML

YAML bought comments at the cost of some guessing. TOML (Tom’s Obvious Minimal Language) is another format aimed at configuration, one that trades YAML’s guesswork for stricter, more predictable rules. The same config one more time, as TOML:

title = "My Blog"
port = 8080
debug = true
tags = ["python", "web"]

key = value with an equals sign, strings quoted like in JSON, and [section] headers (which this small example did not need) for grouping related keys. If it reminds you of an old .ini file, that is the family it grew out of. Python 3.11 added a TOML reader to the standard library, tomllib, and it behaves just like json.load:

>>> import tomllib
>>> with open('config.toml', 'rb') as f:
...     config = tomllib.load(f)
>>> config
{'title': 'My Blog', 'port': 8080, 'debug': True, 'tags': ['python', 'web']}

Note the 'rb': tomllib reads raw bytes rather than text. The standard library only reads TOML; writing it back out needs a third-party package like tomli-w. You meet TOML most often as Python’s own pyproject.toml, the file every modern Python project uses to declare its dependencies, and as Rust’s Cargo config, and across a growing pile of developer tools.

TOML advertises itself as obvious and minimal, right there in the name. Critics enjoy pointing out that each new version adds more features, until the minimal format starts to look about as involved as the ones it set out to replace:

Trojan horse meme: the wooden horse is labeled 'TOML v1.1' and the soldiers hidden inside it are labeled 'JSON'.

The real point under the joke is a useful one: no data format stays simple forever, and “minimal” is a promise every one of them eventually strains.

CSV

All three formats so far can nest, holding a list inside a dict inside a list, as deep as you like. Plenty of data is not nested at all. When it is a plain table of rows and columns, the kind of thing that belongs in a spreadsheet, the right format is the oldest one here: CSV, comma-separated values. Our tweets have no nesting, so they flatten into CSV cleanly, one row per tweet:

text,username
hello,Trump
world,Obama
hola,Obama

The first line names the columns, and every line after it is one row, with a comma between the fields. That is the entire format, which is exactly its appeal: any spreadsheet program opens it, and the file stays tiny. Python reads and writes it with the built-in csv module, and csv.DictReader hands back one dict per row, keyed by the header:

>>> import csv
>>> with open('tweets.csv', newline='') as f:
...     for row in csv.DictReader(f):
...         print(row)
{'text': 'hello', 'username': 'Trump'}
{'text': 'world', 'username': 'Obama'}
{'text': 'hola', 'username': 'Obama'}

CSV cannot replace JSON, and the reason is that it cannot nest. There is no way to put a list inside a cell, so the moment your data has real structure, like a tweet carrying a list of hashtags, CSV forces you to flatten it or give up, and you reach back for JSON.

Which format when

Four formats, and they mostly do not compete; each one owns a job:

  • JSON for data moving between programs: API responses, scraper output, anything crossing a network. The default for machine-to-machine.
  • YAML for configuration a human edits, when the tool you are feeding expects it: GitHub Actions, Docker, Kubernetes.
  • TOML for configuration a human edits in the Python and Rust worlds: pyproject.toml and its neighbors.
  • CSV for flat tables headed to a spreadsheet or a data-analysis library.

The thing to notice is that the choice almost never depends on what your data is. Back in web scraping we saw that how you load data depends only on where it lives, not on what you plan to do with it. The same logic picks the format here: it depends on who reads the data next, another program, a person, or a spreadsheet, and underneath they are all just text. This week’s quiz is on these config formats, open-note and on paper like the others, and the practice quiz, with its answers, is the best way to study for it.

Looking forward

You can now move structured data in and out of your programs in whatever format the receiver expects, which is most of what “working with data” means from day to day. Every JSON file, though, shares one limit: to find anything in it, you load the whole file into memory and loop. That is fine for a few thousand tweets and hopeless for a few million. Next class we meet the tool built for exactly that problem, the database, and its language SQL, where the data stays on disk and you ask for the rows you want instead of reading all of them. The tweets we counted by hand today are about to become a single query.