parsarium

Chapter 01

What a parser does

Bytes come in, structure comes out. Why that step matters for testing and security, and how Parsarium organises it.

7 min read5 quiz questions3 exercises

A parser takes bytes and turns them into structure a program can use. A PNG decoder reads a signature and a sequence of chunks, then gives back a width, a height, and pixels. An HTTP parser reads a request line and headers, then gives back a method, a target, and a map of fields. The rest of the program works with that structure and trusts that it is right.

That makes the parser the first code to see external input. Files arrive from users, from email attachments, from uploads, from other services. Whatever the bytes say, the parser has to decide what they mean before anything else runs.

Why parsers are worth testing

Most formats describe their own layout. A PNG chunk starts with its length, a ZIP archive stores offsets to its entries, an MP4 box declares its size, a PLY header announces how many vertices follow. The parser reads those numbers and acts on them: it allocates memory, seeks to an offset, loops a number of times. When the numbers are wrong, a careful parser rejects the input. A less careful one may read past a buffer, allocate far too much memory, recurse without limit, or loop for a very long time.

A second kind of problem appears when two parsers read the same bytes differently. A proxy and a server that disagree about where an HTTP request ends can be tricked into seeing two requests where one was intended. An archive tool and a signature checker that pick different entries from the same ZIP can check one file and extract another. These bugs do not need a crash. They need two correct-looking programs that make different decisions.

Both kinds of bugs are found by running many inputs through real parsers and watching what happens. That is what Parsarium is built for.

Accepting and rejecting

Every parser draws a line between inputs it accepts and inputs it rejects. Strict parsers follow the specification closely. Lenient parsers try to recover from small mistakes, which is friendly to users and sometimes surprising to security reviewers. Neither choice is wrong on its own; what matters for testing is knowing where each parser draws its line and whether it handles everything on the accepting side safely.

How Parsarium organises parsers

Parsarium is organised by format first, then by language. One format in one language is a cell: PNG in C, ZIP in Go, HTTP/1.1 in PHP. There are 14 formats and eight languages, so 112 cells.

FormatStructure
PNGSignature, then length-typed chunks with checksums
ZIPLocal entries, with a central directory at the end
MP4Nested boxes, each with a size and a type
WAVRIFF chunks: a format description, then samples
PDFNumbered objects and a cross-reference table
SVGAn XML element tree
HTTP/1.1A text start line, headers, and an optional body
ASCII PLYA text header declaring elements, then records
ASCII PNMWhitespace-separated tokens: magic, size, samples
MP3 + ID3v2A tagged metadata prelude before audio frames

The formats were chosen to cover different kinds of structure: binary and text, flat and nested, sequential and offset-based, files and protocol messages. A tool that works well on one kind often struggles with another, so a mix gives a more honest picture.

Each entry in a cell is a package path: an open-source library, or a standard-library API, together with the exact function that parses the input. Each one has been checked with a valid input before it was added. Most have a Docker harness you can run.

Key ideas

  • A parser turns bytes into structure, and the rest of the program trusts that structure.
  • Formats describe their own layout with lengths, counts, and offsets. Parsers act on those numbers, which is where many bugs start.
  • There are two kinds of parser bugs worth looking for: unsafe handling of a single input, and two parsers that disagree about the same input.
  • Parsarium is organised by format, then language. One format in one language is a cell; 14 formats and eight languages make 112 cells.

Check yourself

Answer all five to see your score

Why is the parser often the first place to test in a program that handles files?

  1. It is the first code that interprets external input.
  2. Parsers are usually written in C.
  3. Parsers run with more privileges than the rest of the program.

Whatever the bytes are, the parser decides what they mean before anything else runs. That is true in every language, not only in C.

A ZIP archive keeps a central directory at the end that points back to each entry. What kind of structure is this?

  1. A sequence of chunks
  2. Offsets and a directory
  3. A tree of text

The reader finds the directory first and then jumps to the entries it lists, so the same bytes can be reached in more than one way.

A proxy and a server read the same HTTP request. The proxy thinks it ends after the headers; the server reads a body. Neither crashes. Is that a problem?

  1. No. Without a crash there is no bug.
  2. Yes. The disagreement can let one request hide inside another.
  3. Only if both programs are written in the same language.

Differential bugs need no crash. Two programs that each look correct make different decisions about the same bytes.

What is a cell in Parsarium?

  1. One release of one parser
  2. One control file
  3. One format in one language

PNG in C, ZIP in Go, and HTTP/1.1 in PHP are all cells. Each cell has its own search, ranking, and verified packages.

A lenient parser accepts a PNG with a wrong checksum. What does that tell you?

  1. It is buggy.
  2. It draws its accept line in a different place. Whether that matters depends on how it handles what it accepts.
  3. It is not really a PNG parser.

Leniency is a design choice. For testing, what matters is where each parser draws the line and whether it handles everything on the accepting side safely.

Exercises

Exercise 1.1Warm-up

List the catalog, open one cell, and inspect one target.

$ parsarium list
$ parsarium catalog png c
$ parsarium show png c stb

The catalog output ranks the C packages by GitHub stars. Look at the last column, the underlying family. How many packages read PNG through stb_image?

Hint

Search the family column for the text stb_image. Bundled copies count too.

Solution

Three: raylib and sdl3 bundle stb_image, and stb is stb_image itself. Three packages in the ranking, one decoder underneath. Chapter 4 comes back to this.

Exercise 1.2Practice

Compare how well each language is covered for one format.

$ parsarium list --format http11

Which languages have the most runnable HTTP/1.1 targets, and which the fewest? Check your answer against the HTTP/1.1 row of the atlas.

Hint

The runnable count is the number of targets with a Docker harness. It can be higher than the verified count because standard-library APIs are counted separately.

Solution

Read the RUNNABLE column against the verified columns. Where a language has fewer runnable targets than verified packages, harnesses are still being built. For HTTP/1.1 this shows most in C and C++: many of their HTTP parsers are embedded in larger servers and are harder to drive from a file.

Exercise 1.3Challenge

Find a published vulnerability in a parser for one of the 14 formats. The NVD search or a project's security advisories are good places to start. Answer three questions: which field of the format did the attacker control, what did the parser do with it, and was it a single-input bug or a disagreement between two programs?

Hint

Search for a library name together with words like overflow, out-of-bounds, or path traversal. Advisories that include a proof-of-concept file are the easiest to analyse.

Solution

One well-known example is Zip Slip (2018). The attacker controls the entry name in a ZIP archive and writes names like ../../etc/cron.d/job. Many extraction routines joined that name to the target directory without checking it, so files were written outside it. It is a single-input bug: one program trusted a field it should have validated. Write your own example down the same way: field, behaviour, kind.