Efficient Grammar Fuzzing¶

In the chapter on grammars, we have seen how to use grammars for very effective and efficient testing. In this chapter, we refine the previous string-based algorithm into a tree-based algorithm, which is much faster and allows for much more control over the production of fuzz inputs.

The algorithm in this chapter serves as a foundation for several more techniques; this chapter thus is a "hub" in the book.

Prerequisites

• You should know how grammar-based fuzzing works, e.g. from the chapter on grammars.

An Insufficient Algorithm¶

In the previous chapter, we have introduced the simple_grammar_fuzzer() function which takes a grammar and automatically produces a syntactically valid string from it. However, simple_grammar_fuzzer() is just what its name suggests – simple. To illustrate the problem, let us get back to the expr_grammar we created from EXPR_GRAMMAR_BNF in the chapter on grammars:

expr_grammar has an interesting property. If we feed it into simple_grammar_fuzzer(), the function gets stuck:

Why is that so? Have a look at the grammar; remember what you know about simple_grammar_fuzzer(); and run simple_grammar_fuzzer() with log=true argument to see the expansions.

Indeed! The problem is in this rule:

Here, any choice except for (expr) increases the number of symbols, even if only temporary. Since we place a hard limit on the number of symbols to expand, the only choice left for expanding <factor> is (<expr>), which leads to an infinite addition of parentheses.

The problem of potentially infinite expansion is only one of the problems with simple_grammar_fuzzer(). More problems include:

1. It is inefficient. With each iteration, this fuzzer would go search the string produced so far for symbols to expand. This becomes inefficient as the production string grows.

2. It is hard to control. Even while limiting the number of symbols, it is still possible to obtain very long strings – and even infinitely long ones, as discussed above.

Let us illustrate both problems by plotting the time required for strings of different lengths.

We see that (1) the time needed to generate an output increases quadratically with the length of that output, and that (2) a large portion of the produced outputs are tens of thousands of characters long.

To address these problems, we need a smarter algorithm – one that is more efficient, that gets us better control over expansions, and that is able to foresee in expr_grammar that the (expr) alternative yields a potentially infinite expansion, in contrast to the other two.

Derivation Trees¶

To both obtain a more efficient algorithm and exercise better control over expansions, we will use a special representation for the strings that our grammar produces. The general idea is to use a tree structure that will be subsequently expanded – a so-called derivation tree. This representation allows us to always keep track of our expansion status – answering questions such as which elements have been expanded into which others, and which symbols still need to be expanded. Furthermore, adding new elements to a tree is far more efficient than replacing strings again and again.

Like other trees used in programming, a derivation tree (also known as parse tree or concrete syntax tree) consists of nodes which have other nodes (called child nodes) as their children. The tree starts with one node that has no parent; this is called the root node; a node without children is called a leaf.

The grammar expansion process with derivation trees is illustrated in the following steps, using the arithmetic grammar from the chapter on grammars. We start with a single node as root of the tree, representing the start symbol – in our case <start>.

To expand the tree, we traverse it, searching for a nonterminal symbol $S$ without children. $S$ thus is a symbol that still has to be expanded. We then chose an expansion for $S$ from the grammar. Then, we add the expansion as a new child of $S$. For our start symbol <start>, the only expansion is <expr>, so we add it as a child.

To construct the produced string from a derivation tree, we traverse the tree in order and collect the symbols at the leaves of the tree. In the case above, we obtain the string "<expr>".

To further expand the tree, we choose another symbol to expand, and add its expansion as new children. This would get us the <expr> symbol, which gets expanded into <expr> + <term>, adding three children.

We repeat the expansion until there are no symbols left to expand:

We now have a representation for the string 2 + 2. In contrast to the string alone, though, the derivation tree records the entire structure (and production history, or derivation history) of the produced string. It also allows for simple comparison and manipulation – say, replacing one subtree (substructure) against another.

Representing Derivation Trees¶

To represent a derivation tree in Python, we use the following format. A node is a pair

(SYMBOL_NAME, CHILDREN)

where SYMBOL_NAME is a string representing the node (i.e. "<start>" or "+") and CHILDREN is a list of children nodes.

CHILDREN can take some special values:

1. None as a placeholder for future expansion. This means that the node is a nonterminal symbol that should be expanded further.
2. [] (i.e., the empty list) to indicate no children. This means that the node is a terminal symbol that can no longer be expanded.

The type DerivationTree captures this very structure. (Any should actually read DerivationTree, but the Python static type checker cannot handle recursive types well.)

Let us take a very simple derivation tree, representing the intermediate step <expr> + <term>, above.

To better understand the structure of this tree, let us introduce a function display_tree() that visualizes this tree.

This is what our tree visualizes into:

You can check it out yourself:

Within this book, we also occasionally use a function display_annotated_tree() which allows adding annotations to individual nodes.

If we want to see all the leaf nodes in a tree as a string, the following all_terminals() function comes in handy:

The alternative tree_to_string() function also converts the tree to a string; however, it replaces nonterminal symbols by empty strings.

Expanding a Node¶

Let us now develop an algorithm that takes a tree with non-expanded symbols (say, derivation_tree, above), and expands all these symbols one after the other. As with earlier fuzzers, we create a special subclass of Fuzzer – in this case, GrammarFuzzer. A GrammarFuzzer gets a grammar and a start symbol; the other parameters will be used later to further control creation and to support debugging.

To add further methods to GrammarFuzzer, we use the hack already introduced for the MutationFuzzer class. The construct

class GrammarFuzzer(GrammarFuzzer):
    def new_method(self, args):
        pass

allows us to add a new method new_method() to the GrammarFuzzer class. (Actually, we get a new GrammarFuzzer class that extends the old one, but for all our purposes, this does not matter.)

Let us now define a helper method init_tree() that constructs a tree with just the start symbol:

This is the tree we want to expand.

Picking a Children Alternative to be Expanded¶

One of the central methods in GrammarFuzzer is choose_node_expansion(). This method gets a node (say, the <start> node) and a list of possible lists of children to be expanded (one for every possible expansion from the grammar), chooses one of them, and returns its index in the possible children list.

By overloading this method (notably in later chapters), we can implement different strategies – for now, it simply randomly picks one of the given lists of children (which in turn are lists of derivation trees).

Getting a List of Possible Expansions¶

To actually obtain the list of possible children, we will need a helper function expansion_to_children() that takes an expansion string and decomposes it into a list of derivation trees – one for each symbol (terminal or nonterminal) in the string.

The case of an epsilon expansion, i.e. expanding into an empty string as in <symbol> ::= needs special treatment:

Just like nonterminals() in the chapter on Grammars, we provide for future extensions, allowing the expansion to be a tuple with extra data (which will be ignored).

We realize this helper as a method in GrammarFuzzer such that it can be overloaded by subclasses:

Putting Things Together¶

With this, we can now take

1. some non-expanded node in the tree,
2. choose a random expansion, and
3. return the new tree.

This is what the method expand_node_randomly() does.

This is how expand_node_randomly() works:

We can surely put this to the test, right? Here we go:

We see that <digit> gets expanded again according to the grammar rules – namely, into a single digit.

Although we have changed one of the subtrees, the original expr_tree is unaffected:

That is because expand_node_randomly() returns a new (expanded) tree and does not change the tree passed as argument.

Expanding a Tree¶

Let us now apply our functions for expanding a single node to some node in the tree. To this end, we first need to search the tree for non-expanded nodes. possible_expansions() counts how many unexpanded symbols there are in a tree:

The method any_possible_expansions() returns True if the tree has any non-expanded nodes.

Here comes expand_tree_once(), the core method of our tree expansion algorithm. It first checks whether it is currently being applied on a nonterminal symbol without expansion; if so, it invokes expand_node() on it, as discussed above.

If the node is already expanded (i.e. has children), it checks the subset of children which still have non-expanded symbols, randomly selects one of them, and applies itself recursively on that child.

Let us illustrate how expand_tree_once() works. We start with our derivation tree from above...

... and now expand it twice:

We see that with each step, one more symbol is expanded. Now all it takes is to apply this again and again, expanding the tree further and further.

Closing the Expansion¶

With expand_tree_once(), we can keep on expanding the tree – but how do we actually stop? The key idea here, introduced by Luke in [Luke et al, 2000], is that after inflating the derivation tree to some maximum size, we only want to apply expansions that increase the size of the tree by a minimum. For <factor>, for instance, we would prefer an expansion into <integer>, as this will not introduce further recursion (and potential size inflation); for <integer>, likewise, an expansion into <digit> is preferred, as it will less increase tree size than <digit><integer>.

To identify the cost of expanding a symbol, we introduce two functions that mutually rely on each other:

• symbol_cost() returns the minimum cost of all expansions of a symbol, using expansion_cost() to compute the cost for each expansion.
• expansion_cost() returns the sum of all expansions in expansions. If a nonterminal is encountered again during traversal, the cost of the expansion is $\infty$, indicating (potentially infinite) recursion.

Here are two examples: The minimum cost of expanding a digit is 1, since we have to choose from one of its expansions.

The minimum cost of expanding <expr>, though, is five, as this is the minimum number of expansions required. (<expr> $\rightarrow$ <term> $\rightarrow$ <factor> $\rightarrow$ <integer> $\rightarrow$ <digit> $\rightarrow$ 1)

We define expand_node_by_cost(self, node, choose), a variant of expand_node() that takes the above cost into account. It determines the minimum cost cost across all children and then chooses a child from the list using the choose function, which by default is the minimum cost. If multiple children all have the same minimum cost, it chooses randomly between these.

The shortcut expand_node_min_cost() passes min() as the choose function, which makes it expand nodes at minimum cost.

We can now apply this function to close the expansion of our derivation tree, using expand_tree_once() with the above expand_node_min_cost() as expansion function.

We keep on expanding until all nonterminals are expanded.

Here is the final tree:

We see that in each step, expand_node_min_cost() chooses an expansion that does not increase the number of symbols, eventually closing all open expansions.

Node Inflation¶

Especially at the beginning of an expansion, we may be interested in getting as many nodes as possible – that is, we'd like to prefer expansions that give us more nonterminals to expand. This is actually the exact opposite of what expand_node_min_cost() gives us, and we can implement a method expand_node_max_cost() that will always choose among the nodes with the highest cost:

To illustrate expand_node_max_cost(), we can again redefine expand_node() to use it, and then use expand_tree_once() to show a few expansion steps:

We see that with each step, the number of nonterminals increases. Obviously, we have to put a limit on this number.

Three Expansion Phases¶

We can now put all three phases together in a single function expand_tree() which will work as follows:

1. Max cost expansion. Expand the tree using expansions with maximum cost until we have at least min_nonterminals nonterminals. This phase can be easily skipped by setting min_nonterminals to zero.
2. Random expansion. Keep on expanding the tree randomly until we reach max_nonterminals nonterminals.
3. Min cost expansion. Close the expansion with minimum cost.

We implement these three phases by having expand_node reference the expansion method to apply. This is controlled by setting expand_node (the method reference) to first expand_node_max_cost (i.e., calling expand_node() invokes expand_node_max_cost()), then expand_node_randomly, and finally expand_node_min_cost. In the first two phases, we also set a maximum limit of min_nonterminals and max_nonterminals, respectively.

Let us try this out on our example. We start with a half-expanded derivation tree:

We now apply our expansion strategy on this tree. We see that initially, nodes are expanded at maximum cost, then randomly, and then closing the expansion at minimum cost.

This is the final derivation tree:

And this is the resulting string:

Putting it all Together¶

Based on this, we can now define a function fuzz() that – like simple_grammar_fuzzer() – simply takes a grammar and produces a string from it. It thus no longer exposes the complexity of derivation trees.

We can now apply this on all our defined grammars (and visualize the derivation tree along)

After calling fuzz(), the produced derivation tree is accessible in the derivation_tree attribute:

Let us try out the grammar fuzzer (and its trees) on other grammar formats.

How do we stack up against simple_grammar_fuzzer()?