Chapter 09
Designing an experiment
A checklist for studies that use many parsers: question, targets, budgets, repetitions, metrics, and what to publish.
This last chapter puts the others together. It is the checklist I use before starting an experiment on many targets, written so I do not have to remember it each time.
Start with the question
Write the question in one sentence before choosing anything. "Does tool A reach more parser code than tool B on image formats?" leads to a different setup than "How many ZIP readers accept archives with overlapping entries?" The question tells you which targets, which inputs, and which measurements you need.
Choose and pin the targets
- Build the selection on the bench. Decide whether you count packages or families, and whether to include standard-library APIs.
- Save the manifest and note the selection ID and the Parsarium commit.
- Run the controls for the whole selection first. It takes minutes and confirms that every target works on your machine before the long runs start.
Fix the conditions
- Seeds. The same seed set for every tool. The controls are a reasonable default.
- Budget. Wall-clock time per target, or number of executions. State which.
- Repetitions. Several runs per configuration. Five is a minimum; ten is better when you can afford it.
- Hardware. CPU model, cores used, memory, and whether campaigns shared a machine.
Choose the metrics
| Metric | Good for | How to compare |
|---|---|---|
| Coverage | How far into the parser a tool gets | Within one target, across tools or over time |
| Unique bugs | What a tool finds | After deduplication by location and family |
| Time to first crash | How fast a tool finds a known bug | Median over repetitions |
| Acceptance | How parsers treat an input set | Across all targets of a format |
For coverage, a fair method is to replay each tool's final corpus through the same instrumented target and measure it the same way for every tool, instead of comparing the numbers each tool reports about itself.
Analyse
Report medians with the spread, not single runs. Use a statistical test such as Mann-Whitney U to compare tools on each target, and an effect size alongside the p-value. Report per target first and aggregate second; an average across languages can hide a tool that does well on C and poorly on everything else.
Publish
Share the manifest, the scripts you ran, and the run directories, or at least every summary.json, campaign.json, and afl-summary.json. Cite the Parsarium commit and the selection ID. With those, someone else can rebuild your exact target set and rerun the parts they care about.
Key ideas
- Write the question in one sentence before choosing targets, inputs, or metrics.
- Pin the selection and the commit, and run the controls before the long runs.
- Fix seeds, budget, repetitions, and hardware, and report all four.
- Compare per target first, with medians, spread, and a statistical test. Aggregate second.
- Publish the manifest, the scripts, and the run summaries.
Check yourself
Answer all five to see your score
Tool A has higher average coverage than tool B across 30 targets, from one run each. Is that enough to say A is better?
- Yes, 30 targets is a lot.
- Yes, if the difference is large.
- No. You need repetitions per target and a statistical test, and the average can hide per-target losses.
One run per target mixes the tool's effect with randomness, and an average across languages can hide a tool that only does well on C.
What is a Mann-Whitney U test used for here?
- Measuring coverage
- Comparing the results of two tools on one target without assuming a normal distribution
- Choosing the number of repetitions
Fuzzing results are rarely normally distributed. Mann-Whitney U compares two samples of runs; report an effect size with it.
How do you measure coverage fairly across tools?
- Replay each tool's final corpus through the same instrumented target
- Use the number each tool prints at the end
- Count the files in each queue
Tools measure coverage in different ways. One shared measurement makes the numbers comparable.
Why run the controls for the whole selection before the campaigns?
- To warm up the CPU
- To confirm every target works on your machine before hours of fuzzing
- Because the bench requires it
A broken image or a missing build is cheap to find in minutes and expensive to find after a day of campaigns.
What is the minimum to share so others can rebuild your setup?
- The paper
- The fuzzer's version
- The selection manifest with its ID, the Parsarium commit, and the run summaries
With those, someone can rebuild the target set and rerun the parts they care about.
Exercises
Exercise 9.1Warm-up
For each study idea below, write the question in one sentence and choose a bench selection: formats, languages, per-cell limit, and whether to use one target per family.
- Does structure-aware generation reach more code than byte mutation on binary formats?
- How many ZIP readers accept an archive whose central directory and local headers disagree?
- Do Python parsers handle malformed inputs more safely than C parsers?
Hint
Ask what you are counting in each case: code, behaviours, or languages.
Solution
- PNG, ZIP, MP4, WAV, MP3; one target per family; three per cell. You are comparing tools, so engines matter more than wrappers.
- ZIP only, all languages, all targets, standard APIs included. You are counting real-world behaviour, so every package counts; report families as well.
- All formats, Python and C only, all targets, then repeat with one per family. Language is the variable, so keep everything else the same and show both counts.
Exercise 9.2Practice
Read this evaluation paragraph and list everything a reviewer should ask about.
"We evaluated our fuzzer on 40 parsers from Parsarium for 24 hours. It found 12 crashes and achieved 18% more coverage than AFL++ on average, demonstrating its superiority."
Hint
Go through the checklist in this chapter: selection, seeds, budget, repetitions, metrics, statistics, artifacts.
Solution
- Which 40 parsers? No selection ID or commit.
- 24 hours in total or per target? How many in parallel, on what hardware?
- How many repetitions? Is 18% a median or a single run, and is it significant?
- How was coverage measured: each tool's own numbers, or one shared measurement? Bridged and compiled targets mixed?
- 12 crashes before or after deduplication? How many distinct bugs, in how many families? Were they reproduced and reported?
- Same seeds for both tools?
- "Superiority" on average can hide targets where AFL++ was better; per-target results are missing.
Exercise 9.3Challenge
Design a small study end to end: two formats, one target per family, five-minute campaigns, three repetitions. Build the selection on the bench, download the manifest and the report template, run the controls, then the campaigns. Fill in the setup lines of the report and add a table with the median corpus_count per target.
Hint
Reuse the collector from exercise 7.3, then compute medians per project with statistics.median.
Solution
A complete answer has five files: the manifest, the run script, the CSV from the collector, the filled report, and a short note on anything that did not go as planned. That is a small but complete experiment, and the same structure scales to hundreds of targets.