The Problem with Ad-Hoc Parsing

Every C++ developer has written the same loop: skip whitespace, collect an identifier, track line numbers, and hope the input isn't malformed. Branimir Karadžić, creator of the bgfx rendering library, got tired of repeating this pattern. He also disliked the code generated by the Lemon parser generator, which he used for a shader front-end parser.

"It was small, it worked, but I never liked the code it produced," he writes. "Every time the grammar changed I had to re-learn the shape of the resulting code."

So he built a middle ground: a small set of reusable primitives that handle scanning without a generator or a dependency graph. The result is bx::Scanner, now part of the bx library. It's zero-copy, allocation-free, and readable in a debugger.

Design Constraints

bx::Scanner is built around four deliberate constraints:

  • Zero-copy and non-owning. The scanner never allocates and never copies text. Every result is a StringView pointing directly into the original input.
  • One cursor, and it never moves on its own. A single current position. accept / acceptWhile / acceptUntil move it forward. peek runs the same test without moving. No implicit backtracking.
  • Built-in line and column tracking. Any movement across a newline updates the line number, even a backwards seek. getLine and getColumn are always available, making error messages nearly free.
  • Character classes instead of a mini-language. A handful of classes (Space, NonSpace, Identifier, EndOfLine, NewLine) cover most scanning. Anything more specific is an ordinary bool(*)(char) predicate.

The entire public interface fits in one small header.

Basic Usage

The workflow is always the same: construct a Scanner over a StringView, then peek and accept until done.

bx::Scanner scanner(input);

// Skip leading whitespace.
scanner.accept(bx::Scanner::Class::Space);

// Read an identifier.
const bx::StringView ident = scanner.accept(bx::Scanner::Class::Identifier);

accept consumes matching text and returns it as a StringView. If the expected token isn't there, it returns an empty view and leaves the cursor alone, so you can try the next alternative. peek runs the same test without moving.

One subtlety: StringView has no operator bool, so a successful match reads as !scanner.accept('=').isEmpty(). Noisier than a bool, but the matched text comes back with the answer, saving a second call.

LineReader: The Simplest Helper

LineReader splits input into lines, handles \n and \r\n, and trims stray trailing \r characters.

for (bx::LineReader lr(fileContents); !lr.isDone(); )
{
    const bx::StringView line = lr.next();
    // `line` excludes the terminator, and lr.getLine() is its line number.
}

Note the loop condition. The obvious while (!lr.next().isEmpty()) is wrong: a file can contain a blank line, and that blank line is a valid result. Empty means "this line has no characters," not "there are no more lines." Use isDone for exhaustion.

INI Parsing with Sub-Scanners

One of the most compelling examples is INI parsing. A Scanner constructed from a StringView and acceptUntil returns a StringView, so a single line can become its own scanner. This bounds each line's parser, making "run past the end of a malformed line" unrepresentable.

bx::Scanner scanner(data);

while (!scanner.isDone())
{
    scanner.accept(bx::Scanner::Class::Space);

    // One line, as its own scanner.
    bx::Scanner line(scanner.acceptUntil(bx::Scanner::Class::EndOfLine));

    if (!line.accept(';').isEmpty())   // Line comment.
    {
        continue;
    }

    if (!line.accept('[').isEmpty())   // [section] header.
    {
        line.accept(bx::Scanner::Class::Space);
        const bx::StringView name = bx::strRTrimSpace(line.acceptUntil("]"));

        if (!line.accept(']').isEmpty() && !name.isEmpty())
        {
            section = addSection(name);
        }

        continue;
    }

    const bx::StringView name = bx::strRTrimSpace(line.acceptUntil("="));

    if (line.accept('=').isEmpty() || name.isEmpty())
    {
        continue;
    }

    line.accept(bx::Scanner::Class::Space);
    setProperty(section, name, bx::strRTrimSpace(line.acceptAll()));
}

This replaced a third-party INI library outright.

URL Parsing: The Meaning of Empty

Because a match comes back as a StringView and not a bool, "matched nothing" and "didn't match" arrive as the same value. acceptUntil returns empty both when the delimiter isn't in the input and when it's already at the cursor. This collapse is often what you want: an empty token is a legitimate token. http://example.com has no path, and most URLs have no userinfo.

bx::Scanner scanner(url);

const bx::StringView scheme = scanner.acceptUntil("://");

// Ask separately whether there was a scheme.
const bool hasScheme = !scanner.accept("://").isEmpty();

const bx::StringView authority = scanner.acceptWhile(isNotSlash);

const bool hasPath = !scanner.peek('/').isEmpty();

The rule of thumb: the return value is the token; peek or accept answers the structural question "was the delimiter present?" If you need to capture a span assembled from multiple accepts, use getCursor and getPosition.

Karadžić notes: "Total 75 lines of code to parse URL, not 23k lines of C++ header files…"

File Path Normalization

Path handling deals with drive letters, mixed / and \, and collapsing . and ... Separators are matched with a predicate rather than a string, because accept("/\\") would look for the literal two-character sequence /\.

bx::Scanner scanner(src);

if (2 <= src.getLength() && ':' == src.getPtr()[1])   // Windows drive letter.
{
    size += write(&writer, toUpper(src.getPtr()[0]), &err);
    size += write(&writer, ':', &err);
    scanner.seek(2);
}

const bool rooted = !scanner.accept(isPathSeparator).isEmpty();

while (!scanner.isDone() && err.isOk())
{
    if (!scanner.acceptWhile(isPathSeparator).isEmpty())
    {
        trailingSlash = scanner.isDone();
        continue;
    }

    const bx::StringView component = scanner.acceptWhile(isNotPathSeparator);

    // `.` is skipped, `..` rewinds the writer to the previous separator.
}

The .. handling rewinds the output, not the input, with a watermark preventing escape from the prefix. The scanner's job is only to hand over components one at a time.

Stack Trace Symbolication

Parsing atos or addr2line output becomes straightforward:

for (bx::LineReader lr({atosBuffer, bytes}); !lr.isDone(); )
{
    bx::Scanner scanner(lr.next());

    functionName = scanner.acceptUntil(" (");

    if (functionName.isEmpty())
    {
        break;
    }

    scanner.accept(" (");

    filePath = scanner.acceptUntil(":");

    if (filePath.isEmpty())
    {
        filePath = "";
        break;
    }

    scanner.accept(':');

    const bx::StringView lineStr = scanner.acceptUntil(")");

    if (!lineStr.isEmpty())
    {
        bx::fromString(&line, lineStr);
    }
}

Again, the collapse of "missing" and "empty" works in your favor: an empty function name isn't a valid result, so the plain return value is the only test needed.

Conclusion

Karadžić first needed this while writing yet another ad-hoc parser for addr2line output. That produced strConsumeTo, which later became the core of bx::Scanner. By using it, he deleted an INI library dependency and three hand-rolled copies of the same whitespace-and-identifier loop.

If you find yourself writing another pointer-chasing loop to skip whitespace and collect an identifier, consider reaching for something like this. Parsers really don't have to be complicated.