parsarium

Chapter 09

Designing an experiment

A checklist for studies that use many parsers: question, targets, budgets, repetitions, metrics, and what to publish.

6 min read5 quiz questions3 exercises

This last chapter puts the others together. It is the checklist I use before starting an experiment on many targets, written so I do not have to remember it each time.

Start with the question

Write the question in one sentence before choosing anything. "Does tool A reach more parser code than tool B on image formats?" leads to a different setup than "How many ZIP readers accept archives with overlapping entries?" The question tells you which targets, which inputs, and which measurements you need.

Choose and pin the targets

  • Build the selection on the bench. Decide whether you count packages or families, and whether to include standard-library APIs.
  • Save the manifest and note the selection ID and the Parsarium commit.
  • Run the controls for the whole selection first. It takes minutes and confirms that every target works on your machine before the long runs start.

Fix the conditions

  • Seeds. The same seed set for every tool. The controls are a reasonable default.
  • Budget. Wall-clock time per target, or number of executions. State which.
  • Repetitions. Several runs per configuration. Five is a minimum; ten is better when you can afford it.
  • Hardware. CPU model, cores used, memory, and whether campaigns shared a machine.

Choose the metrics

MetricGood forHow to compare
CoverageHow far into the parser a tool getsWithin one target, across tools or over time
Unique bugsWhat a tool findsAfter deduplication by location and family
Time to first crashHow fast a tool finds a known bugMedian over repetitions
AcceptanceHow parsers treat an input setAcross all targets of a format

For coverage, a fair method is to replay each tool's final corpus through the same instrumented target and measure it the same way for every tool, instead of comparing the numbers each tool reports about itself.

Analyse

Report medians with the spread, not single runs. Use a statistical test such as Mann-Whitney U to compare tools on each target, and an effect size alongside the p-value. Report per target first and aggregate second; an average across languages can hide a tool that does well on C and poorly on everything else.

Publish

Share the manifest, the scripts you ran, and the run directories, or at least every summary.json, campaign.json, and afl-summary.json. Cite the Parsarium commit and the selection ID. With those, someone else can rebuild your exact target set and rerun the parts they care about.

Key ideas

  • Write the question in one sentence before choosing targets, inputs, or metrics.
  • Pin the selection and the commit, and run the controls before the long runs.
  • Fix seeds, budget, repetitions, and hardware, and report all four.
  • Compare per target first, with medians, spread, and a statistical test. Aggregate second.
  • Publish the manifest, the scripts, and the run summaries.

Check yourself

Answer all five to see your score

Tool A has higher average coverage than tool B across 30 targets, from one run each. Is that enough to say A is better?

  1. Yes, 30 targets is a lot.
  2. Yes, if the difference is large.
  3. No. You need repetitions per target and a statistical test, and the average can hide per-target losses.

One run per target mixes the tool's effect with randomness, and an average across languages can hide a tool that only does well on C.

What is a Mann-Whitney U test used for here?

  1. Measuring coverage
  2. Comparing the results of two tools on one target without assuming a normal distribution
  3. Choosing the number of repetitions

Fuzzing results are rarely normally distributed. Mann-Whitney U compares two samples of runs; report an effect size with it.

How do you measure coverage fairly across tools?

  1. Replay each tool's final corpus through the same instrumented target
  2. Use the number each tool prints at the end
  3. Count the files in each queue

Tools measure coverage in different ways. One shared measurement makes the numbers comparable.

Why run the controls for the whole selection before the campaigns?

  1. To warm up the CPU
  2. To confirm every target works on your machine before hours of fuzzing
  3. Because the bench requires it

A broken image or a missing build is cheap to find in minutes and expensive to find after a day of campaigns.

What is the minimum to share so others can rebuild your setup?

  1. The paper
  2. The fuzzer's version
  3. The selection manifest with its ID, the Parsarium commit, and the run summaries

With those, someone can rebuild the target set and rerun the parts they care about.

Exercises

Exercise 9.1Warm-up

For each study idea below, write the question in one sentence and choose a bench selection: formats, languages, per-cell limit, and whether to use one target per family.

  1. Does structure-aware generation reach more code than byte mutation on binary formats?
  2. How many ZIP readers accept an archive whose central directory and local headers disagree?
  3. Do Python parsers handle malformed inputs more safely than C parsers?
Hint

Ask what you are counting in each case: code, behaviours, or languages.

Solution
  1. PNG, ZIP, MP4, WAV, MP3; one target per family; three per cell. You are comparing tools, so engines matter more than wrappers.
  2. ZIP only, all languages, all targets, standard APIs included. You are counting real-world behaviour, so every package counts; report families as well.
  3. All formats, Python and C only, all targets, then repeat with one per family. Language is the variable, so keep everything else the same and show both counts.

Exercise 9.2Practice

Read this evaluation paragraph and list everything a reviewer should ask about.

"We evaluated our fuzzer on 40 parsers from Parsarium for 24 hours. It found 12 crashes and achieved 18% more coverage than AFL++ on average, demonstrating its superiority."
Hint

Go through the checklist in this chapter: selection, seeds, budget, repetitions, metrics, statistics, artifacts.

Solution
  • Which 40 parsers? No selection ID or commit.
  • 24 hours in total or per target? How many in parallel, on what hardware?
  • How many repetitions? Is 18% a median or a single run, and is it significant?
  • How was coverage measured: each tool's own numbers, or one shared measurement? Bridged and compiled targets mixed?
  • 12 crashes before or after deduplication? How many distinct bugs, in how many families? Were they reproduced and reported?
  • Same seeds for both tools?
  • "Superiority" on average can hide targets where AFL++ was better; per-target results are missing.

Exercise 9.3Challenge

Design a small study end to end: two formats, one target per family, five-minute campaigns, three repetitions. Build the selection on the bench, download the manifest and the report template, run the controls, then the campaigns. Fill in the setup lines of the report and add a table with the median corpus_count per target.

Hint

Reuse the collector from exercise 7.3, then compute medians per project with statistics.median.

Solution

A complete answer has five files: the manifest, the run script, the CSV from the collector, the filled report, and a short note on anything that did not go as planned. That is a small but complete experiment, and the same structure scales to hundreds of targets.