parsarium

Chapter 03

Controls and receipts

What it means for a parser to be verified, what a harness records, and how to read a run receipt.

6 min read5 quiz questions3 exercises

Before a parser enters Parsarium, it has to read a valid input and report the right thing about it. That input is the control, and the record of what happened is the receipt. Together they tell you that the target is wired up correctly before you spend hours fuzzing it.

Why start with a valid input

A harness that rejects every input looks very robust: it never crashes. It also tests nothing. The same happens when the harness calls the wrong function, opens the wrong file, or catches every error before the parser runs. A valid control with a known expected output catches all of these problems in one step. If the harness reports width 2 and height 1 for red-green.png, the bytes reached the parser and the parser understood them.

What verification checks

  • A pinned release. The exact version, installed from a lock file or a pinned image, with source hashes.
  • A documented input API. The function the project documents for parsing this format. Detecting a MIME type or copying bytes does not count.
  • A valid control. A small file in controls/<format>/, identified by its SHA-256.
  • An observed result. Format-specific fields such as dimensions, entry names, page counts, tags, or header values, compared with the expected output.

Until all four pass, a package is a lead, not an entry. Leads are kept in the catalog data with the reason they are still open.

The verification receipt

Every runnable target has a verification.json next to its Dockerfile. It records when the harness was verified, the Docker image ID, the toolchain versions, the hash of every project file, the observed output, and how many lines of parser source the control covered. If any project file changes, its hash no longer matches and the harness refuses to run until it is verified again.

The run receipt

When you run a file through a target, you get a receipt like this:

{
  "schema_version": 1,
  "project": "png/python/pillow",
  "input": "my.png",
  "status": "ok",
  "case_sha256": "…",
  "case_bytes": 71,
  "image_id": "sha256:…",
  "exit_code": 0,
  "duration_seconds": 0.412,
  "stdout_tail": "…",
  "stderr_tail": ""
}

The status field describes the harness, not the parser:

StatusMeaning
okThe harness finished. The parser may have accepted or rejected the input; its output is in stdout_tail.
target_nonzeroThe process exited with an error. Worth replaying.
timeoutThe run took longer than the limit (60 seconds by default).
input_too_largeThe input is over 1 MiB, the harness limit.
infrastructure_errorDocker or the harness setup failed before the parser ran.

Keeping these apart is useful. A rejection is normal parser behaviour. A non-zero exit or a timeout is something to look at.

Key ideas

  • A valid control with a known expected output proves that bytes reach the parser and that the parser understands them.
  • A package becomes a target only after four checks: pinned release, documented input API, valid control, and observed result. Before that it is a lead.
  • The run receipt's status describes the harness, not the parser. Acceptance is in the observed output.
  • Verification receipts pin file hashes, so a changed harness has to be verified again before it runs.

Check yourself

Answer all five to see your score

A run receipt says status: ok. What do you know?

  1. The parser accepted the input.
  2. The harness finished. Whether the parser accepted the input is in its output.
  3. The input is a valid file.

ok only means the harness completed. A clean rejection is also ok; the parser's answer is in stdout_tail.

Why does every target need a valid control before fuzzing?

  1. It makes fuzzing faster.
  2. It shows that bytes reach the parser and are understood. A harness that rejects everything would look very robust.
  3. AFL++ refuses to start without one.

Without a control you cannot tell a robust parser from a broken harness. The control is the cheapest way to rule out the second.

A package shows up in an npm search for "png" but has not passed the control yet. What is it in Parsarium?

  1. A target
  2. A lead
  3. A family

Search hits stay leads until all four checks pass. Leads are kept in the cell's data with the reason they are still open.

You edit the harness file of a project. What happens on the next run?

  1. Its hash no longer matches the receipt, so it has to be verified again.
  2. Nothing; the receipt is only informative.
  3. Docker silently rebuilds the image.

The verification receipt lists the hash of every project file. Runs check these hashes, which keeps results tied to the harness that was verified.

You run a 2 MiB file through a target. What status do you get?

  1. timeout
  2. target_nonzero
  3. input_too_large

Harnesses accept inputs up to 1 MiB. Larger inputs are refused before the parser runs.

Exercises

Exercise 3.1Warm-up

Run the stb_image target on its own control and on a file that is not a PNG at all.

$ parsarium show png c stb --json
$ parsarium run png c stb "$PARSARIUM/controls/png/red-green.png" --results runs/stb-control
$ parsarium run png c stb "$PARSARIUM/controls/zip/hello.zip" --results runs/stb-zip

Open both receipts and compare status, exit_code, and the output tails. Where does stb_image say what it thought of the ZIP file?

Hint

Receipts are written as <sha256>.json inside the --results directory.

Solution

The control receipt shows the dimensions and pixels in stdout_tail. The ZIP receipt shows stb_image's rejection, either in the output tails or as a non-zero exit, depending on how the harness reports errors. The important habit is to look at the output, not only at status.

Exercise 3.2Practice

Open the verification receipts of two PNG targets in different languages and compare them field by field.

cat "$PARSARIUM/projects/png/c/stb/verification.json"
cat "$PARSARIUM/projects/png/python/pillow/verification.json"

What does each toolchain list contain? How many parser source lines did each control cover? Can you say which parser is better tested from these numbers?

Hint

Look for toolchain, output, and parser_source_lines_covered.

Solution

At the time of writing the stb_image control covers 435 parser lines and the Pillow control 2,097. That does not make Pillow better tested: the two code bases are different sizes, written in different languages, and the line counts come from different coverage tools. The numbers are useful for each target on its own, for example to check that a new input reaches more of the same parser.

Exercise 3.3Challenge

Pick a PNG library you know that is not in the catalog. Write a minimal harness for it: read one file, call the documented parsing API, and print the width and height. Decide what it should print and which exit code it should use when the parser rejects the input, and when something unexpected happens.

Hint

Compare your draft with $PARSARIUM/projects/png/python/pillow/harness.py and adapter.py.

Solution

A good harness does four things. It calls the parser through the API a real user would use. It prints observable, format-specific fields. It treats a normal rejection as a normal outcome, so the exit code stays zero and the reason goes to the output. And it lets unexpected failures, such as crashes or errors that are not part of the parser's documented behaviour, surface as a non-zero exit, because those are the ones worth investigating.