Chapter 06
Differential testing
Send the same bytes to many parsers and compare what they say. Where they disagree is where to look.
When many programs implement the same format, they can check each other. Send one input to all of them and compare the answers. If most parsers say the image is 2 by 1 and one says 3 by 1, at least one of them is worth a closer look. You do not need to know the correct answer in advance; the disagreement is the signal.
Why disagreements matter
Some disagreements are harmless: one parser is lenient about a missing field, another is strict. Others are security problems. If a proxy and a server parse the same HTTP request differently, an attacker can hide a second request inside the first. If a signature checker and an installer read different entries from the same ZIP, the checked file and the installed file are not the same. If a PDF shows different content in different viewers, a reviewer and a reader see different documents. In each case, both programs may look correct on their own.
Differential testing also finds plain bugs. A parser that accepts a file every other implementation rejects is often skipping a check.
Running one file through every parser
Parsarium has a command for this. It runs every verified harness for a format, optionally limited to one language, and saves one receipt per target plus a summary.json.
$ parsarium run-format png --list $ parsarium run-format png --file my.png --results runs/png-my-file --workers 4
summary.json lists each target with its status and the directory of its receipt. The status counts give a first view: how many finished, exited with an error, or timed out.
Comparing the answers
Status alone does not tell you whether a parser accepted the input, so the next step is to compare outputs. Harnesses print the fields they observed, and the field names differ slightly between targets, so pick the fields that matter for your question and normalise them. For images, width and height are a good start. A short script is enough:
python3 - runs/png-my-file <<'EOF'
import json, sys, pathlib, collections
root = pathlib.Path(sys.argv[1])
summary = json.loads((root / "summary.json").read_text())
groups = collections.defaultdict(list)
for row in summary["results"]:
receipts = list((root / row["receipt_dir"]).glob("*.json"))
tail = json.loads(receipts[0].read_text()).get("stdout_tail", "") if receipts else ""
groups[(row["status"], tail.strip()[-80:])].append(row["project"])
for (status, tail), projects in sorted(groups.items(), key=lambda g: -len(g[1])):
print(f"{len(projects):4} {status:15} {tail}")
for p in projects[:5]:
print(" ", p)
EOF
The largest group is usually the consensus. Small groups are the interesting ones.
Families again
When you count agreement, remember that wrappers share engines. Eight PHP packages agreeing with each other may be one decoder agreeing with itself. Group results by parser family before deciding how unusual a disagreement is.
Key ideas
- Sending one input to many implementations needs no correct answer in advance: disagreement is the signal.
- Some disagreements are leniency. Others are security problems when two parsers sit on the same path.
run-formatruns every harness for a format and saves one receipt per target plus a summary.- Status counts are a first view; the real comparison is on normalised output fields.
- Group by parser family before deciding how unusual a disagreement is.
Check yourself
Answer all five to see your score
Do you need to know the correct output before running a differential test?
- No. The disagreement between implementations is the signal.
- Yes, otherwise results cannot be interpreted.
- Only for binary formats.
That is the main strength of differential testing. You look at the minority answers, then decide who is right.
What causes HTTP request smuggling?
- A server that crashes on long headers
- A front end and a back end that disagree about where a request ends
- Requests that use HTTP/2
If one program uses Content-Length and the other Transfer-Encoding, the bytes one sees as a body the other sees as the start of a new request.
run-format reports ok for every target. Do all the parsers agree?
- Yes, all of them accepted the file.
- Yes, if they are in the same language.
- Not necessarily.
okmeans each harness finished; you still have to compare their outputs.
A clean rejection and a successful parse both give ok. The agreement is in the output fields.
Nine of ten PHP packages give the same answer. Is that strong consensus?
- Yes, nine independent votes
- Maybe not. If they share one engine, it is one answer repeated.
- No, PHP is always lenient.
Wrappers inherit their engine's behaviour. Group results by family to count independent opinions.
Why normalise field names before comparing outputs?
- Harnesses name the same field differently, so a naive comparison shows false disagreements.
- To make the receipts smaller
- Because JSON is case-sensitive
One PNG harness prints rgba0, another first. Pick the fields your question needs and map them to common names.
Exercises
Exercise 6.1Warm-up
Take wide-new-crc.png from exercise 2.2: a PNG that claims to be 3 pixels wide but carries data for 2. Send it to every PNG parser.
$ parsarium run-format png --file wide-new-crc.png --results runs/png-wide --workers 4
Group the results with the script in this chapter. How many parsers reject the file? Do any report width 3? Do the answers split along languages, or along parser families?
Hint
Start with status_counts in summary.json, then run the grouping script.
Solution
The exact split depends on the parsers, which is the point of running it. Look for three groups: parsers that reject the file because the image data is too short, parsers that accept it and report width 3 (filling or ignoring the missing pixels), and any that exit with an error or time out. The last group is worth a closer look. Then check the family column: if a group is made of one engine's wrappers, it is one behaviour, not many.
Exercise 6.2Practice
ZIP entries can be compressed with different methods. controls/zip/hello-zstd.zip uses Zstandard, a newer method that not every reader supports. Send it to every ZIP parser.
$ parsarium run-format zip --file "$PARSARIUM/controls/zip/hello-zstd.zip" --results runs/zip-zstd --workers 4
Which parsers can read the entry? How do the others fail: a clear error, a wrong result, or a crash?
Hint
Only one ZIP target in the catalog uses this file as its control. That tells you something about how many readers are expected to support it.
Solution
Expect most readers not to support Zstandard; the catalog's own zipfile-zstd target exists precisely because Python's standard zipfile needs an extension for it. The useful finding is how each reader fails. A clear "unsupported compression method" error is good behaviour. Returning garbage or an empty file without an error is the kind of disagreement that can matter in practice, for example when one tool checks an archive and another extracts it.
Exercise 6.3Challenge
Write an HTTP request with both a Content-Length and a Transfer-Encoding: chunked header, and send it to every HTTP/1.1 parser.
printf 'POST / HTTP/1.1\r\nHost: example.test\r\nContent-Length: 4\r\nTransfer-Encoding: chunked\r\n\r\n0\r\n\r\n' > cl-te.http
$ parsarium run-format http11 --file cl-te.http --results runs/http-cl-te --workers 4
Which header does each parser follow, and which ones reject the request?
Hint
Read section 6.1 of RFC 9112 first to know what the specification allows.
Solution
RFC 9112 lets a server either reject such a request or process it using Transfer-Encoding alone. Both are fine. A parser that uses Content-Length instead is the dangerous case, because if it sits in front of or behind a parser that follows Transfer-Encoding, the two disagree about where the request ends. Group your results into those three behaviours. Note that some targets parse responses rather than requests; leave those out of this comparison.