Chapter 03
Controls and receipts
What it means for a parser to be verified, what a harness records, and how to read a run receipt.
Before a parser enters Parsarium, it has to read a valid input and report the right thing about it. That input is the control, and the record of what happened is the receipt. Together they tell you that the target is wired up correctly before you spend hours fuzzing it.
Why start with a valid input
A harness that rejects every input looks very robust: it never crashes. It also tests nothing. The same happens when the harness calls the wrong function, opens the wrong file, or catches every error before the parser runs. A valid control with a known expected output catches all of these problems in one step. If the harness reports width 2 and height 1 for red-green.png, the bytes reached the parser and the parser understood them.
What verification checks
- A pinned release. The exact version, installed from a lock file or a pinned image, with source hashes.
- A documented input API. The function the project documents for parsing this format. Detecting a MIME type or copying bytes does not count.
- A valid control. A small file in
controls/<format>/, identified by its SHA-256. - An observed result. Format-specific fields such as dimensions, entry names, page counts, tags, or header values, compared with the expected output.
Until all four pass, a package is a lead, not an entry. Leads are kept in the catalog data with the reason they are still open.
The verification receipt
Every runnable target has a verification.json next to its Dockerfile. It records when the harness was verified, the Docker image ID, the toolchain versions, the hash of every project file, the observed output, and how many lines of parser source the control covered. If any project file changes, its hash no longer matches and the harness refuses to run until it is verified again.
The run receipt
When you run a file through a target, you get a receipt like this:
{
"schema_version": 1,
"project": "png/python/pillow",
"input": "my.png",
"status": "ok",
"case_sha256": "…",
"case_bytes": 71,
"image_id": "sha256:…",
"exit_code": 0,
"duration_seconds": 0.412,
"stdout_tail": "…",
"stderr_tail": ""
}
The status field describes the harness, not the parser:
| Status | Meaning |
|---|---|
ok | The harness finished. The parser may have accepted or rejected the input; its output is in stdout_tail. |
target_nonzero | The process exited with an error. Worth replaying. |
timeout | The run took longer than the limit (60 seconds by default). |
input_too_large | The input is over 1 MiB, the harness limit. |
infrastructure_error | Docker or the harness setup failed before the parser ran. |
Keeping these apart is useful. A rejection is normal parser behaviour. A non-zero exit or a timeout is something to look at.
Key ideas
- A valid control with a known expected output proves that bytes reach the parser and that the parser understands them.
- A package becomes a target only after four checks: pinned release, documented input API, valid control, and observed result. Before that it is a lead.
- The run receipt's status describes the harness, not the parser. Acceptance is in the observed output.
- Verification receipts pin file hashes, so a changed harness has to be verified again before it runs.
Check yourself
Answer all five to see your score
A run receipt says status: ok. What do you know?
- The parser accepted the input.
- The harness finished. Whether the parser accepted the input is in its output.
- The input is a valid file.
ok only means the harness completed. A clean rejection is also ok; the parser's answer is in stdout_tail.
Why does every target need a valid control before fuzzing?
- It makes fuzzing faster.
- It shows that bytes reach the parser and are understood. A harness that rejects everything would look very robust.
- AFL++ refuses to start without one.
Without a control you cannot tell a robust parser from a broken harness. The control is the cheapest way to rule out the second.
A package shows up in an npm search for "png" but has not passed the control yet. What is it in Parsarium?
- A target
- A lead
- A family
Search hits stay leads until all four checks pass. Leads are kept in the cell's data with the reason they are still open.
You edit the harness file of a project. What happens on the next run?
- Its hash no longer matches the receipt, so it has to be verified again.
- Nothing; the receipt is only informative.
- Docker silently rebuilds the image.
The verification receipt lists the hash of every project file. Runs check these hashes, which keeps results tied to the harness that was verified.
You run a 2 MiB file through a target. What status do you get?
timeouttarget_nonzeroinput_too_large
Harnesses accept inputs up to 1 MiB. Larger inputs are refused before the parser runs.
Exercises
Exercise 3.1Warm-up
Run the stb_image target on its own control and on a file that is not a PNG at all.
$ parsarium show png c stb --json $ parsarium run png c stb "$PARSARIUM/controls/png/red-green.png" --results runs/stb-control $ parsarium run png c stb "$PARSARIUM/controls/zip/hello.zip" --results runs/stb-zip
Open both receipts and compare status, exit_code, and the output tails. Where does stb_image say what it thought of the ZIP file?
Hint
Receipts are written as <sha256>.json inside the --results directory.
Solution
The control receipt shows the dimensions and pixels in stdout_tail. The ZIP receipt shows stb_image's rejection, either in the output tails or as a non-zero exit, depending on how the harness reports errors. The important habit is to look at the output, not only at status.
Exercise 3.2Practice
Open the verification receipts of two PNG targets in different languages and compare them field by field.
cat "$PARSARIUM/projects/png/c/stb/verification.json" cat "$PARSARIUM/projects/png/python/pillow/verification.json"
What does each toolchain list contain? How many parser source lines did each control cover? Can you say which parser is better tested from these numbers?
Hint
Look for toolchain, output, and parser_source_lines_covered.
Solution
At the time of writing the stb_image control covers 435 parser lines and the Pillow control 2,097. That does not make Pillow better tested: the two code bases are different sizes, written in different languages, and the line counts come from different coverage tools. The numbers are useful for each target on its own, for example to check that a new input reaches more of the same parser.
Exercise 3.3Challenge
Pick a PNG library you know that is not in the catalog. Write a minimal harness for it: read one file, call the documented parsing API, and print the width and height. Decide what it should print and which exit code it should use when the parser rejects the input, and when something unexpected happens.
Hint
Compare your draft with $PARSARIUM/projects/png/python/pillow/harness.py and adapter.py.
Solution
A good harness does four things. It calls the parser through the API a real user would use. It prints observable, format-specific fields. It treats a normal rejection as a normal outcome, so the exit code stays zero and the reason goes to the output. And it lets unexpected failures, such as crashes or errors that are not part of the parser's documented behaviour, surface as a non-zero exit, because those are the ones worth investigating.