parsarium

Chapter 07

Fuzzing at scale

From one AFL++ campaign to hundreds: seeds, budgets, parallel runs, repetitions, and keeping track of it all.

5 min read5 quiz questions3 exercises

One fuzzing campaign is a command. A hundred campaigns are a plan: how long each runs, how many run at once, where the results go, and how you will compare them afterwards. This chapter covers both.

One campaign

Every runnable target has an AFL++ overlay. A campaign has four steps:

$ parsarium afl build zip go stdlib_openreader
$ parsarium afl prepare zip go stdlib_openreader runs/zip-go-afl
$ parsarium afl run runs/zip-go-afl --seconds 3600
$ parsarium afl summary runs/zip-go-afl
  • afl build builds the instrumented image and checks that the control produces feedback.
  • afl prepare copies the seeds, pins the image ID, and writes plan.json with the exact command.
  • afl run checks the plan, the image, and the seed hashes, then runs AFL++ for the given time. --check runs only the checks. --resume continues an existing run.
  • afl summary lists the files in queue/, crashes/, and hangs/ with their hashes, and reads AFL++'s fuzzer_stats.

Each target also has its language's own fuzzing engine (libFuzzer, Go native fuzzing, Jazzer, Jazzer.js, Atheris, PHP-Fuzzer) through parsarium.py build and parsarium.py native. They are useful when you want to compare against the tool developers in that language already use.

Seeds

By default, a campaign starts from the target's valid control. That is a good minimal seed: it reaches the main parsing path. A richer seed corpus usually helps, especially for formats with many optional parts. You can add files with --seed or whole directories with --seed-dir:

$ parsarium fandango setup
$ parsarium fandango generate png c stb runs/stb-generated --count 100 --seed 42
$ parsarium afl prepare png c stb runs/stb-afl --seed-dir runs/stb-generated/generated

When you compare tools, give every tool the same seeds, so the comparison is about what the tool does after it starts.

Budgets

Work out the total time before you start. Each AFL++ instance uses one CPU core. With T targets, S seconds each, R repetitions, and P campaigns in parallel, the wall time is roughly T × S × R / P. A hundred targets for one hour each, repeated five times on sixteen cores, is a bit over 31 hours. Leave one or two cores free for Docker and the system, and check disk space: queues and logs grow with every campaign.

Bridged targets run fewer executions per second than compiled ones. If you use the same wall-clock budget for both, say so in your report; if you prefer to match executions instead, AFL++'s execs_done in fuzzer_stats gives you the count.

Running many campaigns

The bench generates a script that runs a selection one target at a time. To run several at once, write the target list to a file and use xargs:

cat > targets.txt <<'EOF'
png c stb
png rust png
zip go stdlib_openreader
zip python stdlib_zipfile
EOF

run_one() {
  set -- $1
  dir="runs/batch/$1-$2-$3"
  parsarium afl build "$1" "$2" "$3" &&
  parsarium afl prepare "$1" "$2" "$3" "$dir" &&
  parsarium afl run "$dir" --seconds 3600 &&
  parsarium afl summary "$dir" > /dev/null
}
export -f run_one
xargs -P 4 -I{} bash -c 'run_one "{}"' < targets.txt

Use parsarium projects --format zip to look up the slugs. Every campaign writes its own plan.json, campaign.json, and afl-summary.json, so a batch can be collected and compared later with a short script.

Repetitions

Fuzzing is random. Two campaigns with the same seed and budget can end with different coverage and find different crashes. A single run per target says little about a tool. Run each configuration several times, report the median and the spread, and use a statistical test when you claim one tool is better. Klees et al., "Evaluating Fuzz Testing" (CCS 2018) is the standard reference and is worth reading before a large experiment.

Key ideas

  • A campaign is four commands: build, prepare, run, summary. --check verifies a plan, --resume continues one.
  • Give every tool the same seeds, so the comparison is about what the tool does after it starts.
  • Wall time is roughly targets × seconds × repetitions ÷ parallel campaigns. Each AFL++ instance uses one core.
  • Fuzzing is random. Repeat each configuration and report the spread, not a single run.

Check yourself

Answer all five to see your score

50 targets, 2 hours each, 5 repetitions, 10 campaigns in parallel. Roughly how long is the experiment?

  1. 10 hours
  2. 50 hours
  3. 500 hours

50 × 2 × 5 = 500 campaign-hours, divided by 10 in parallel: about 50 hours.

What does afl run --check do?

  1. Checks the plan, the image, and the seed hashes without starting the campaign
  2. Runs a one-minute campaign
  3. Checks the target for crashes

It is a dry run: everything a campaign verifies before starting, and nothing after.

You compare two fuzzers. What should the seeds be?

  1. Whatever each tool generates itself
  2. The control for one tool and a large corpus for the other
  3. The same seed set for both

Same seeds isolate the effect of the tool. Different seeds mix two variables.

Why run each configuration several times?

  1. To use the hardware fully
  2. Because two runs with the same setup can end with different coverage and crashes
  3. Because AFL++ requires it

Random mutation produces different paths each time. One run per configuration says little about the tool.

A campaign stopped halfway because the machine rebooted. How do you continue it?

  1. afl run with --resume on the same run directory
  2. Prepare a new run with the queue as seeds
  3. Start over; AFL++ cannot continue

--resume continues the existing afl-out/ directory with its queue and statistics.

Exercises

Exercise 7.1Warm-up

Use the bench to build a selection of four ZIP targets, one per language, and download its manifest. Run it with a five-minute budget per target:

$ parsarium run-selection sel-….json --mode afl --seconds 300

Collect execs_per_sec, corpus_count, and saved_crashes from each afl-summary.json.

Hint

Select ZIP, set Targets per format × language to 1, and choose four languages.

Solution

Each target's numbers are in instances[0].fuzzer_stats of its afl-summary.json. Compiled targets will usually show the highest execs_per_sec. Keep the table: exercise 7.3 builds on it.

Exercise 7.2Practice

Measure the effect of seeds. Generate a PNG corpus with Fandango, then run stb_image twice for ten minutes: once from the control only, and once with the generated seeds.

$ parsarium fandango setup
$ parsarium fandango generate png c stb runs/stb-generated --count 100 --seed 42
$ parsarium afl prepare png c stb runs/stb-control-only
$ parsarium afl prepare png c stb runs/stb-seeded --seed-dir runs/stb-generated/generated
$ parsarium afl run runs/stb-control-only --seconds 600
$ parsarium afl run runs/stb-seeded --seconds 600

Compare corpus_count and edges_found. Does the richer seed set help, and is one run enough to say so?

Hint

fandango setup installs the locked version. Run afl build png c stb first if you have not built the image.

Solution

A varied seed corpus usually starts with more coverage, because it already contains chunk types the control lacks. Whether it ends higher after ten minutes is less certain. One run of each is not enough to claim a difference; run each configuration at least five times and compare the medians, as chapter 9 describes.

Exercise 7.3Challenge

Write a collector that walks a directory of AFL++ runs and writes one CSV row per campaign: project, execs_done, execs_per_sec, corpus_count, saved_crashes, saved_hangs.

Hint

Each run directory has an afl-summary.json with project and a list of instances, each with a fuzzer_stats dictionary.

Solution
python3 - runs/batch <<'EOF'
import csv, json, pathlib, sys
fields = ["execs_done", "execs_per_sec", "corpus_count", "saved_crashes", "saved_hangs"]
out = csv.writer(sys.stdout)
out.writerow(["project", "run", *fields])
for path in sorted(pathlib.Path(sys.argv[1]).rglob("afl-summary.json")):
    summary = json.loads(path.read_text())
    for instance in summary["instances"]:
        stats = instance["fuzzer_stats"]
        out.writerow([summary["project"], path.parent.name, *(stats.get(f, "") for f in fields)])
EOF

Run it once per batch and keep the CSV next to the selection manifest.