Efficient Grammar Fuzzing¶

In the chapter on grammars, we have seen how to use grammars for very effective and efficient testing. In this chapter, we refine the previous string-based algorithm into a tree-based algorithm, which is much faster and allows for much more control over the production of fuzz inputs.

An Insufficient Algorithm¶

In the previous chapter, we have introduced the simple_grammar_fuzzer() function which takes a grammar and automatically produces a syntactically valid string from it. However, simple_grammar_fuzzer() is just what its name suggests – simple. To illustrate the problem, let us get back to the expr_grammar we created from EXPR_GRAMMAR_BNF in the chapter on grammars:

In [7]:
expr_grammar = convert_ebnf_grammar(EXPR_EBNF_GRAMMAR)
expr_grammar
Out[7]:
{'<start>': ['<expr>'],
 '<expr>': ['<term> + <expr>', '<term> - <expr>', '<term>'],
 '<term>': ['<factor> * <term>', '<factor> / <term>', '<factor>'],
 '<factor>': ['<sign-1><factor>', '(<expr>)', '<integer><symbol-1>'],
 '<sign>': ['+', '-'],
 '<integer>': ['<digit-1>'],
 '<digit>': ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'],
 '<symbol>': ['.<integer>'],
 '<sign-1>': ['', '<sign>'],
 '<symbol-1>': ['', '<symbol>'],
 '<digit-1>': ['<digit>', '<digit><digit-1>']}

expr_grammar has an interesting property. If we feed it into simple_grammar_fuzzer(), the function gets stuck:

In [9]:
with ExpectTimeout(1):
    simple_grammar_fuzzer(grammar=expr_grammar, max_nonterminals=3)

Traceback (most recent call last):
  File "/var/folders/n2/xd9445p97rb3xh7m1dfx8_4h0006ts/T/ipykernel_81067/3259437052.py", line 2, in <cell line: 1>
    simple_grammar_fuzzer(grammar=expr_grammar, max_nonterminals=3)
  File "Grammars.ipynb", line 87, in simple_grammar_fuzzer
    symbol_to_expand = random.choice(nonterminals(term))
  File "Grammars.ipynb", line 61, in nonterminals
    return RE_NONTERMINAL.findall(expansion)
  File "Timeout.ipynb", line 43, in timeout_handler
    raise TimeoutError()
TimeoutError (expected)

Why is that so? Have a look at the grammar; remember what you know about simple_grammar_fuzzer(); and run simple_grammar_fuzzer() with log=true argument to see the expansions.

In [10]:
quiz("Why does `simple_grammar_fuzzer()` hang?",
     [
         "It produces an infinite number of additions",
         "It produces an infinite number of digits",
         "It produces an infinite number of parentheses",
         "It produces an infinite number of signs",
     ], '(3 * 3 * 3) ** (3 / (3 * 3))')
Out[10]:

Quiz

Why does simple_grammar_fuzzer() hang?





Indeed! The problem is in this rule:

In [11]:
expr_grammar['<factor>']
Out[11]:
['<sign-1><factor>', '(<expr>)', '<integer><symbol-1>']

Here, any choice except for (expr) increases the number of symbols, even if only temporary. Since we place a hard limit on the number of symbols to expand, the only choice left for expanding <factor> is (<expr>), which leads to an infinite addition of parentheses.

The problem of potentially infinite expansion is only one of the problems with simple_grammar_fuzzer(). More problems include:

  1. It is inefficient. With each iteration, this fuzzer would go search the string produced so far for symbols to expand. This becomes inefficient as the production string grows.

  2. It is hard to control. Even while limiting the number of symbols, it is still possible to obtain very long strings – and even infinitely long ones, as discussed above.

Let us illustrate both problems by plotting the time required for strings of different lengths.

In [16]:
trials = 50
xs = []
ys = []
for i in range(trials):
    with Timer() as t:
        s = simple_grammar_fuzzer(EXPR_GRAMMAR, max_nonterminals=15)
    xs.append(len(s))
    ys.append(t.elapsed_time())
    print(i, end=" ")
print()

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
In [17]:
average_time = sum(ys) / trials
print("Average time:", average_time)

Average time: 0.14829922830220313
In [18]:
%matplotlib inline

import matplotlib.pyplot as plt
plt.scatter(xs, ys)
plt.title('Time required for generating an output');

We see that (1) the time needed to generate an output increases quadratically with the length of that output, and that (2) a large portion of the produced outputs are tens of thousands of characters long.

To address these problems, we need a smarter algorithm – one that is more efficient, that gets us better control over expansions, and that is able to foresee in expr_grammar that the (expr) alternative yields a potentially infinite expansion, in contrast to the other two.

Derivation Trees¶

To both obtain a more efficient algorithm and exercise better control over expansions, we will use a special representation for the strings that our grammar produces. The general idea is to use a tree structure that will be subsequently expanded – a so-called derivation tree. This representation allows us to always keep track of our expansion status – answering questions such as which elements have been expanded into which others, and which symbols still need to be expanded. Furthermore, adding new elements to a tree is far more efficient than replacing strings again and again.

Like other trees used in programming, a derivation tree (also known as parse tree or concrete syntax tree) consists of nodes which have other nodes (called child nodes) as their children. The tree starts with one node that has no parent; this is called the root node; a node without children is called a leaf.

The grammar expansion process with derivation trees is illustrated in the following steps, using the arithmetic grammar from the chapter on grammars. We start with a single node as root of the tree, representing the start symbol – in our case <start>.

In [21]:
# ignore
tree
Out[21]:
root \<start\> <start>

To expand the tree, we traverse it, searching for a nonterminal symbol $S$ without children. $S$ thus is a symbol that still has to be expanded. We then chose an expansion for $S$ from the grammar. Then, we add the expansion as a new child of $S$. For our start symbol <start>, the only expansion is <expr>, so we add it as a child.

In [23]:
# ignore
tree
Out[23]:
root \<start\> <start> \<expr\> <expr> \<start\>->\<expr\>

To construct the produced string from a derivation tree, we traverse the tree in order and collect the symbols at the leaves of the tree. In the case above, we obtain the string "<expr>".

To further expand the tree, we choose another symbol to expand, and add its expansion as new children. This would get us the <expr> symbol, which gets expanded into <expr> + <term>, adding three children.

In [25]:
# ignore
tree
Out[25]:
root \<start\> <start> \<expr\> <expr> \<start\>->\<expr\> \<expr\> <expr> \<expr\>->\<expr\> + + \<expr\>->+ \<term\> <term> \<expr\>->\<term\>

We repeat the expansion until there are no symbols left to expand:

In [27]:
# ignore
tree
Out[27]:
root \<start\> <start> \<expr\> <expr> \<start\>->\<expr\> \<expr\> <expr> \<expr\>->\<expr\> + + \<expr\>->+ \<term\> <term> \<expr\>->\<term\> \<term\> <term> \<expr\> ->\<term\> \<factor\> <factor> \<term\>->\<factor\> \<factor\> <factor> \<term\> ->\<factor\> \<integer\> <integer> \<factor\> ->\<integer\> \<digit\> <digit> \<integer\> ->\<digit\> 2 2 \<digit\> ->2 \<integer\> <integer> \<factor\>->\<integer\> \<digit\> <digit> \<integer\>->\<digit\> 2 2 \<digit\>->2

We now have a representation for the string 2 + 2. In contrast to the string alone, though, the derivation tree records the entire structure (and production history, or derivation history) of the produced string. It also allows for simple comparison and manipulation – say, replacing one subtree (substructure) against another.

Representing Derivation Trees¶

To represent a derivation tree in Python, we use the following format. A node is a pair

(SYMBOL_NAME, CHILDREN)

where SYMBOL_NAME is a string representing the node (i.e. "<start>" or "+") and CHILDREN is a list of children nodes.

CHILDREN can take some special values:

  1. None as a placeholder for future expansion. This means that the node is a nonterminal symbol that should be expanded further.
  2. [] (i.e., the empty list) to indicate no children. This means that the node is a terminal symbol that can no longer be expanded.

The type DerivationTree captures this very structure. (Any should actually read DerivationTree, but the Python static type checker cannot handle recursive types well.)

In [28]:
DerivationTree = Tuple[str, Optional[List[Any]]]

Let us take a very simple derivation tree, representing the intermediate step <expr> + <term>, above.

In [29]:
derivation_tree: DerivationTree = ("<start>",
                   [("<expr>",
                     [("<expr>", None),
                      (" + ", []),
                         ("<term>", None)]
                     )])

To better understand the structure of this tree, let us introduce a function display_tree() that visualizes this tree.

This is what our tree visualizes into:

In [47]:
display_tree(derivation_tree)
Out[47]:
0 <start> 1 <expr> 0->1 2 <expr> 1->2 3 + 1->3 4 <term> 1->4
In [48]:
quiz("And which of these is the internal representation of `derivation_tree`?",
    [
        "`('<start>', [('<expr>', (['<expr> + <term>']))])`",
        "`('<start>', [('<expr>', (['<expr>', ' + ', <term>']))])`",
        "`" + repr(derivation_tree) + "`",
        "`(" + repr(derivation_tree) + ", None)`"
    ], len("eleven") - len("one"))
Out[48]:

Quiz

And which of these is the internal representation of derivation_tree?





You can check it out yourself:

In [49]:
derivation_tree
Out[49]:
('<start>', [('<expr>', [('<expr>', None), (' + ', []), ('<term>', None)])])

Within this book, we also occasionally use a function display_annotated_tree() which allows adding annotations to individual nodes.

If we want to see all the leaf nodes in a tree as a string, the following all_terminals() function comes in handy:

In [52]:
def all_terminals(tree: DerivationTree) -> str:
    (symbol, children) = tree
    if children is None:
        # This is a nonterminal symbol not expanded yet
        return symbol

    if len(children) == 0:
        # This is a terminal symbol
        return symbol

    # This is an expanded symbol:
    # Concatenate all terminal symbols from all children
    return ''.join([all_terminals(c) for c in children])
In [53]:
all_terminals(derivation_tree)
Out[53]:
'<expr> + <term>'

The alternative tree_to_string() function also converts the tree to a string; however, it replaces nonterminal symbols by empty strings.

In [54]:
def tree_to_string(tree: DerivationTree) -> str:
    symbol, children, *_ = tree
    if children:
        return ''.join(tree_to_string(c) for c in children)
    else:
        return '' if is_nonterminal(symbol) else symbol
In [55]:
tree_to_string(derivation_tree)
Out[55]:
' + '

Expanding a Node¶

Let us now develop an algorithm that takes a tree with non-expanded symbols (say, derivation_tree, above), and expands all these symbols one after the other. As with earlier fuzzers, we create a special subclass of Fuzzer – in this case, GrammarFuzzer. A GrammarFuzzer gets a grammar and a start symbol; the other parameters will be used later to further control creation and to support debugging.

In [57]:
class GrammarFuzzer(Fuzzer):
    """Produce strings from grammars efficiently, using derivation trees."""

    def __init__(self,
                 grammar: Grammar,
                 start_symbol: str = START_SYMBOL,
                 min_nonterminals: int = 0,
                 max_nonterminals: int = 10,
                 disp: bool = False,
                 log: Union[bool, int] = False) -> None:
        """Produce strings from `grammar`, starting with `start_symbol`.
        If `min_nonterminals` or `max_nonterminals` is given, use them as limits 
        for the number of nonterminals produced.  
        If `disp` is set, display the intermediate derivation trees.
        If `log` is set, show intermediate steps as text on standard output."""

        self.grammar = grammar
        self.start_symbol = start_symbol
        self.min_nonterminals = min_nonterminals
        self.max_nonterminals = max_nonterminals
        self.disp = disp
        self.log = log
        self.check_grammar()  # Invokes is_valid_grammar()

To add further methods to GrammarFuzzer, we use the hack already introduced for the MutationFuzzer class. The construct

class GrammarFuzzer(GrammarFuzzer):
    def new_method(self, args):
        pass

allows us to add a new method new_method() to the GrammarFuzzer class. (Actually, we get a new GrammarFuzzer class that extends the old one, but for all our purposes, this does not matter.)

Let us now define a helper method init_tree() that constructs a tree with just the start symbol:

In [59]:
class GrammarFuzzer(GrammarFuzzer):
    def init_tree(self) -> DerivationTree:
        return (self.start_symbol, None)
In [60]:
f = GrammarFuzzer(EXPR_GRAMMAR)
display_tree(f.init_tree())
Out[60]:
0 <start>

This is the tree we want to expand.

Picking a Children Alternative to be Expanded¶

One of the central methods in GrammarFuzzer is choose_node_expansion(). This method gets a node (say, the <start> node) and a list of possible lists of children to be expanded (one for every possible expansion from the grammar), chooses one of them, and returns its index in the possible children list.

By overloading this method (notably in later chapters), we can implement different strategies – for now, it simply randomly picks one of the given lists of children (which in turn are lists of derivation trees).

In [61]:
class GrammarFuzzer(GrammarFuzzer):
    def choose_node_expansion(self, node: DerivationTree,
                              children_alternatives: List[List[DerivationTree]]) -> int:
        """Return index of expansion in `children_alternatives` to be selected.
           'children_alternatives`: a list of possible children for `node`.
           Defaults to random. To be overloaded in subclasses."""
        return random.randrange(0, len(children_alternatives))

Getting a List of Possible Expansions¶

To actually obtain the list of possible children, we will need a helper function expansion_to_children() that takes an expansion string and decomposes it into a list of derivation trees – one for each symbol (terminal or nonterminal) in the string.

In [63]:
expansion_to_children("<term> + <expr>")
Out[63]:
[('<term>', None), (' + ', []), ('<expr>', None)]

The case of an epsilon expansion, i.e. expanding into an empty string as in <symbol> ::= needs special treatment:

In [64]:
expansion_to_children("")
Out[64]:
[('', [])]

Just like nonterminals() in the chapter on Grammars, we provide for future extensions, allowing the expansion to be a tuple with extra data (which will be ignored).

In [65]:
expansion_to_children(("+<term>", {"extra_data": 1234}))
Out[65]:
[('+', []), ('<term>', None)]

We realize this helper as a method in GrammarFuzzer such that it can be overloaded by subclasses:

In [66]:
class GrammarFuzzer(GrammarFuzzer):
    def expansion_to_children(self, expansion: Expansion) -> List[DerivationTree]:
        return expansion_to_children(expansion)

Putting Things Together¶

With this, we can now take

  1. some non-expanded node in the tree,
  2. choose a random expansion, and
  3. return the new tree.

This is what the method expand_node_randomly() does.

This is how expand_node_randomly() works:

In [71]:
f = GrammarFuzzer(EXPR_GRAMMAR, log=True)

print("Before expand_node_randomly():")
expr_tree = ("<integer>", None)
display_tree(expr_tree)

Before expand_node_randomly():
Out[71]:
0 <integer>
In [72]:
print("After expand_node_randomly():")
expr_tree = f.expand_node_randomly(expr_tree)
display_tree(expr_tree)

After expand_node_randomly():
Expanding <integer> randomly
Out[72]:
0 <integer> 1 <digit> 0->1
In [73]:
# docassert
assert expr_tree[1][0][0] == '<digit>'
In [74]:
quiz("What tree do we get if we expand the `<digit>` subtree?",
     [
         "We get another `<digit>` as new child of `<digit>`",
         "We get some digit as child of `<digit>`",
         "We get another `<digit>` as second child of `<integer>`",
         "The entire tree becomes a single node with a digit"
     ], 'len("2") + len("2")')
Out[74]:

Quiz

What tree do we get if we expand the <digit> subtree?





We can surely put this to the test, right? Here we go:

In [75]:
digit_subtree = expr_tree[1][0]  # type: ignore
display_tree(digit_subtree)
Out[75]:
0 <digit>
In [76]:
print("After expanding the <digit> subtree:")
digit_subtree = f.expand_node_randomly(digit_subtree)
display_tree(digit_subtree)

After expanding the <digit> subtree:
Expanding <digit> randomly
Out[76]:
0 <digit> 1 7 (55) 0->1

We see that <digit> gets expanded again according to the grammar rules – namely, into a single digit.

In [77]:
quiz("Is the original `expr_tree` affected by this change?",
     [
         "No, it is unchanged",
         "Yes, it has also gained a new child"
     ], "1 ** (1 - 1)")
Out[77]:

Quiz

Is the original expr_tree affected by this change?



Although we have changed one of the subtrees, the original expr_tree is unaffected:

In [78]:
display_tree(expr_tree)
Out[78]:
0 <integer> 1 <digit> 0->1

That is because expand_node_randomly() returns a new (expanded) tree and does not change the tree passed as argument.

Expanding a Tree¶

Let us now apply our functions for expanding a single node to some node in the tree. To this end, we first need to search the tree for non-expanded nodes. possible_expansions() counts how many unexpanded symbols there are in a tree:

In [79]:
class GrammarFuzzer(GrammarFuzzer):
    def possible_expansions(self, node: DerivationTree) -> int:
        (symbol, children) = node
        if children is None:
            return 1

        return sum(self.possible_expansions(c) for c in children)
In [80]:
f = GrammarFuzzer(EXPR_GRAMMAR)
print(f.possible_expansions(derivation_tree))

2

The method any_possible_expansions() returns True if the tree has any non-expanded nodes.

In [81]:
class GrammarFuzzer(GrammarFuzzer):
    def any_possible_expansions(self, node: DerivationTree) -> bool:
        (symbol, children) = node
        if children is None:
            return True

        return any(self.any_possible_expansions(c) for c in children)
In [82]:
f = GrammarFuzzer(EXPR_GRAMMAR)
f.any_possible_expansions(derivation_tree)
Out[82]:
True

Here comes expand_tree_once(), the core method of our tree expansion algorithm. It first checks whether it is currently being applied on a nonterminal symbol without expansion; if so, it invokes expand_node() on it, as discussed above.

If the node is already expanded (i.e. has children), it checks the subset of children which still have non-expanded symbols, randomly selects one of them, and applies itself recursively on that child.

Let us illustrate how expand_tree_once() works. We start with our derivation tree from above...

In [84]:
derivation_tree = ("<start>",
                   [("<expr>",
                     [("<expr>", None),
                      (" + ", []),
                         ("<term>", None)]
                     )])
display_tree(derivation_tree)
Out[84]:
0 <start> 1 <expr> 0->1 2 <expr> 1->2 3 + 1->3 4 <term> 1->4

... and now expand it twice:

In [85]:
f = GrammarFuzzer(EXPR_GRAMMAR, log=True)
derivation_tree = f.expand_tree_once(derivation_tree)
display_tree(derivation_tree)

Expanding <expr> randomly
Out[85]:
0 <start> 1 <expr> 0->1 2 <expr> 1->2 6 + 1->6 7 <term> 1->7 3 <term> 2->3 4 - 2->4 5 <expr> 2->5
In [86]:
derivation_tree = f.expand_tree_once(derivation_tree)
display_tree(derivation_tree)

Expanding <term> randomly
Out[86]:
0 <start> 1 <expr> 0->1 2 <expr> 1->2 6 + 1->6 7 <term> 1->7 3 <term> 2->3 4 - 2->4 5 <expr> 2->5 8 <factor> 7->8

We see that with each step, one more symbol is expanded. Now all it takes is to apply this again and again, expanding the tree further and further.

Closing the Expansion¶

With expand_tree_once(), we can keep on expanding the tree – but how do we actually stop? The key idea here, introduced by Luke in \cite{Luke2000}, is that after inflating the derivation tree to some maximum size, we only want to apply expansions that increase the size of the tree by a minimum. For <factor>, for instance, we would prefer an expansion into <integer>, as this will not introduce further recursion (and potential size inflation); for <integer>, likewise, an expansion into <digit> is preferred, as it will less increase tree size than <digit><integer>.

To identify the cost of expanding a symbol, we introduce two functions that mutually rely on each other:

  • symbol_cost() returns the minimum cost of all expansions of a symbol, using expansion_cost() to compute the cost for each expansion.
  • expansion_cost() returns the sum of all expansions in expansions. If a nonterminal is encountered again during traversal, the cost of the expansion is $\infty$, indicating (potentially infinite) recursion.

Here are two examples: The minimum cost of expanding a digit is 1, since we have to choose from one of its expansions.

In [88]:
f = GrammarFuzzer(EXPR_GRAMMAR)
assert f.symbol_cost("<digit>") == 1

The minimum cost of expanding <expr>, though, is five, as this is the minimum number of expansions required. (<expr> $\rightarrow$ <term> $\rightarrow$ <factor> $\rightarrow$ <integer> $\rightarrow$ <digit> $\rightarrow$ 1)

In [89]:
assert f.symbol_cost("<expr>") == 5

We define expand_node_by_cost(self, node, choose), a variant of expand_node() that takes the above cost into account. It determines the minimum cost cost across all children and then chooses a child from the list using the choose function, which by default is the minimum cost. If multiple children all have the same minimum cost, it chooses randomly between these.

The shortcut expand_node_min_cost() passes min() as the choose function, which makes it expand nodes at minimum cost.

In [91]:
class GrammarFuzzer(GrammarFuzzer):
    def expand_node_min_cost(self, node: DerivationTree) -> DerivationTree:
        if self.log:
            print("Expanding", all_terminals(node), "at minimum cost")

        return self.expand_node_by_cost(node, min)

We can now apply this function to close the expansion of our derivation tree, using expand_tree_once() with the above expand_node_min_cost() as expansion function.

In [92]:
class GrammarFuzzer(GrammarFuzzer):
    def expand_node(self, node: DerivationTree) -> DerivationTree:
        return self.expand_node_min_cost(node)
In [93]:
f = GrammarFuzzer(EXPR_GRAMMAR, log=True)
display_tree(derivation_tree)
Out[93]:
0 <start> 1 <expr> 0->1 2 <expr> 1->2 6 + 1->6 7 <term> 1->7 3 <term> 2->3 4 - 2->4 5 <expr> 2->5 8 <factor> 7->8
In [94]:
# docassert
assert f.any_possible_expansions(derivation_tree)
In [95]:
if f.any_possible_expansions(derivation_tree):
    derivation_tree = f.expand_tree_once(derivation_tree)
display_tree(derivation_tree)

Expanding <expr> at minimum cost
Out[95]:
0 <start> 1 <expr> 0->1 2 <expr> 1->2 7 + 1->7 8 <term> 1->8 3 <term> 2->3 4 - 2->4 5 <expr> 2->5 6 <term> 5->6 9 <factor> 8->9
In [96]:
# docassert
assert f.any_possible_expansions(derivation_tree)
In [97]:
if f.any_possible_expansions(derivation_tree):
    derivation_tree = f.expand_tree_once(derivation_tree)
display_tree(derivation_tree)

Expanding <factor> at minimum cost
Out[97]:
0 <start> 1 <expr> 0->1 2 <expr> 1->2 7 + 1->7 8 <term> 1->8 3 <term> 2->3 4 - 2->4 5 <expr> 2->5 6 <term> 5->6 9 <factor> 8->9 10 <integer> 9->10
In [98]:
# docassert
assert f.any_possible_expansions(derivation_tree)
In [99]:
if f.any_possible_expansions(derivation_tree):
    derivation_tree = f.expand_tree_once(derivation_tree)
display_tree(derivation_tree)

Expanding <term> at minimum cost
Out[99]:
0 <start> 1 <expr> 0->1 2 <expr> 1->2 8 + 1->8 9 <term> 1->9 3 <term> 2->3 5 - 2->5 6 <expr> 2->6 4 <factor> 3->4 7 <term> 6->7 10 <factor> 9->10 11 <integer> 10->11

We keep on expanding until all nonterminals are expanded.

In [100]:
while f.any_possible_expansions(derivation_tree):
    derivation_tree = f.expand_tree_once(derivation_tree)    

Expanding <factor> at minimum cost
Expanding <integer> at minimum cost
Expanding <integer> at minimum cost
Expanding <term> at minimum cost
Expanding <digit> at minimum cost
Expanding <digit> at minimum cost
Expanding <factor> at minimum cost
Expanding <integer> at minimum cost
Expanding <digit> at minimum cost

Here is the final tree:

In [101]:
display_tree(derivation_tree)
Out[101]:
0 <start> 1 <expr> 0->1 2 <expr> 1->2 15 + 1->15 16 <term> 1->16 3 <term> 2->3 8 - 2->8 9 <expr> 2->9 4 <factor> 3->4 5 <integer> 4->5 6 <digit> 5->6 7 4 (52) 6->7 10 <term> 9->10 11 <factor> 10->11 12 <integer> 11->12 13 <digit> 12->13 14 4 (52) 13->14 17 <factor> 16->17 18 <integer> 17->18 19 <digit> 18->19 20 7 (55) 19->20

We see that in each step, expand_node_min_cost() chooses an expansion that does not increase the number of symbols, eventually closing all open expansions.

Node Inflation¶

Especially at the beginning of an expansion, we may be interested in getting as many nodes as possible – that is, we'd like to prefer expansions that give us more nonterminals to expand. This is actually the exact opposite of what expand_node_min_cost() gives us, and we can implement a method expand_node_max_cost() that will always choose among the nodes with the highest cost:

In [102]:
class GrammarFuzzer(GrammarFuzzer):
    def expand_node_max_cost(self, node: DerivationTree) -> DerivationTree:
        if self.log:
            print("Expanding", all_terminals(node), "at maximum cost")

        return self.expand_node_by_cost(node, max)

To illustrate expand_node_max_cost(), we can again redefine expand_node() to use it, and then use expand_tree_once() to show a few expansion steps:

In [103]:
class GrammarFuzzer(GrammarFuzzer):
    def expand_node(self, node: DerivationTree) -> DerivationTree:
        return self.expand_node_max_cost(node)
In [104]:
derivation_tree = ("<start>",
                   [("<expr>",
                     [("<expr>", None),
                      (" + ", []),
                         ("<term>", None)]
                     )])
In [105]:
f = GrammarFuzzer(EXPR_GRAMMAR, log=True)
display_tree(derivation_tree)
Out[105]:
0 <start> 1 <expr> 0->1 2 <expr> 1->2 3 + 1->3 4 <term> 1->4
In [106]:
# docassert
assert f.any_possible_expansions(derivation_tree)
In [107]:
if f.any_possible_expansions(derivation_tree):
    derivation_tree = f.expand_tree_once(derivation_tree)
display_tree(derivation_tree)

Expanding <expr> at maximum cost
Out[107]:
0 <start> 1 <expr> 0->1 2 <expr> 1->2 6 + 1->6 7 <term> 1->7 3 <term> 2->3 4 - 2->4 5 <expr> 2->5
In [108]:
# docassert
assert f.any_possible_expansions(derivation_tree)
In [109]:
if f.any_possible_expansions(derivation_tree):
    derivation_tree = f.expand_tree_once(derivation_tree)
display_tree(derivation_tree)

Expanding <expr> at maximum cost
Out[109]:
0 <start> 1 <expr> 0->1 2 <expr> 1->2 9 + 1->9 10 <term> 1->10 3 <term> 2->3 4 - 2->4 5 <expr> 2->5 6 <term> 5->6 7 - 5->7 8 <expr> 5->8
In [110]:
# docassert
assert f.any_possible_expansions(derivation_tree)
In [111]:
if f.any_possible_expansions(derivation_tree):
    derivation_tree = f.expand_tree_once(derivation_tree)
display_tree(derivation_tree)

Expanding <term> at maximum cost
Out[111]:
0 <start> 1 <expr> 0->1 2 <expr> 1->2 9 + 1->9 10 <term> 1->10 3 <term> 2->3 4 - 2->4 5 <expr> 2->5 6 <term> 5->6 7 - 5->7 8 <expr> 5->8 11 <factor> 10->11 12 * 10->12 13 <term> 10->13

We see that with each step, the number of nonterminals increases. Obviously, we have to put a limit on this number.

Three Expansion Phases¶

We can now put all three phases together in a single function expand_tree() which will work as follows:

  1. Max cost expansion. Expand the tree using expansions with maximum cost until we have at least min_nonterminals nonterminals. This phase can be easily skipped by setting min_nonterminals to zero.
  2. Random expansion. Keep on expanding the tree randomly until we reach max_nonterminals nonterminals.
  3. Min cost expansion. Close the expansion with minimum cost.

We implement these three phases by having expand_node reference the expansion method to apply. This is controlled by setting expand_node (the method reference) to first expand_node_max_cost (i.e., calling expand_node() invokes expand_node_max_cost()), then expand_node_randomly, and finally expand_node_min_cost. In the first two phases, we also set a maximum limit of min_nonterminals and max_nonterminals, respectively.

Let us try this out on our example. We start with a half-expanded derivation tree:

In [113]:
initial_derivation_tree: DerivationTree = ("<start>",
                   [("<expr>",
                     [("<expr>", None),
                      (" + ", []),
                         ("<term>", None)]
                     )])
In [114]:
display_tree(initial_derivation_tree)
Out[114]:
0 <start> 1 <expr> 0->1 2 <expr> 1->2 3 + 1->3 4 <term> 1->4

We now apply our expansion strategy on this tree. We see that initially, nodes are expanded at maximum cost, then randomly, and then closing the expansion at minimum cost.

In [115]:
f = GrammarFuzzer(
    EXPR_GRAMMAR,
    min_nonterminals=3,
    max_nonterminals=5,
    log=True)
derivation_tree = f.expand_tree(initial_derivation_tree)

Tree: <expr> + <term>
Expanding <term> at maximum cost
Tree: <expr> + <factor> / <term>
Expanding <factor> randomly
Tree: <expr> + <integer> / <term>
Expanding <integer> randomly
Tree: <expr> + <digit><integer> / <term>
Expanding <expr> randomly
Tree: <term> + <digit><integer> / <term>
Expanding <term> randomly
Tree: <factor> + <digit><integer> / <term>
Expanding <term> randomly
Tree: <factor> + <digit><integer> / <factor> * <term>
Expanding <integer> at minimum cost
Tree: <factor> + <digit><digit> / <factor> * <term>
Expanding <factor> at minimum cost
Tree: <integer> + <digit><digit> / <factor> * <term>
Expanding <integer> at minimum cost
Tree: <digit> + <digit><digit> / <factor> * <term>
Expanding <digit> at minimum cost
Tree: 1 + <digit><digit> / <factor> * <term>
Expanding <digit> at minimum cost
Tree: 1 + 4<digit> / <factor> * <term>
Expanding <factor> at minimum cost
Tree: 1 + 4<digit> / <integer> * <term>
Expanding <digit> at minimum cost
Tree: 1 + 43 / <integer> * <term>
Expanding <term> at minimum cost
Tree: 1 + 43 / <integer> * <factor>
Expanding <factor> at minimum cost
Tree: 1 + 43 / <integer> * <integer>
Expanding <integer> at minimum cost
Tree: 1 + 43 / <integer> * <digit>
Expanding <digit> at minimum cost
Tree: 1 + 43 / <integer> * 2
Expanding <integer> at minimum cost
Tree: 1 + 43 / <digit> * 2
Expanding <digit> at minimum cost
Tree: 1 + 43 / 3 * 2

This is the final derivation tree:

In [116]:
display_tree(derivation_tree)
Out[116]:
0 <start> 1 <expr> 0->1 2 <expr> 1->2 8 + 1->8 9 <term> 1->9 3 <term> 2->3 4 <factor> 3->4 5 <integer> 4->5 6 <digit> 5->6 7 1 (49) 6->7 10 <factor> 9->10 17 / 9->17 18 <term> 9->18 11 <integer> 10->11 12 <digit> 11->12 14 <integer> 11->14 13 4 (52) 12->13 15 <digit> 14->15 16 3 (51) 15->16 19 <factor> 18->19 23 * 18->23 24 <term> 18->24 20 <integer> 19->20 21 <digit> 20->21 22 3 (51) 21->22 25 <factor> 24->25 26 <integer> 25->26 27 <digit> 26->27 28 2 (50) 27->28

And this is the resulting string:

In [117]:
all_terminals(derivation_tree)
Out[117]:
'1 + 43 / 3 * 2'

Putting it all Together¶

Based on this, we can now define a function fuzz() that – like simple_grammar_fuzzer() – simply takes a grammar and produces a string from it. It thus no longer exposes the complexity of derivation trees.

In [118]:
class GrammarFuzzer(GrammarFuzzer):
    def fuzz_tree(self) -> DerivationTree:
        """Produce a derivation tree from the grammar."""
        tree = self.init_tree()
        # print(tree)

        # Expand all nonterminals
        tree = self.expand_tree(tree)
        if self.log:
            print(repr(all_terminals(tree)))
        if self.disp:
            display(display_tree(tree))
        return tree

    def fuzz(self) -> str:
        """Produce a string from the grammar."""
        self.derivation_tree = self.fuzz_tree()
        return all_terminals(self.derivation_tree)

We can now apply this on all our defined grammars (and visualize the derivation tree along)

In [119]:
f = GrammarFuzzer(EXPR_GRAMMAR)
f.fuzz()
Out[119]:
'10.3 * 9 * (9 * 9 + 8 + 7 - 6) * 55.3 * 3 / 9'

After calling fuzz(), the produced derivation tree is accessible in the derivation_tree attribute:

In [120]:
display_tree(f.derivation_tree)
Out[120]:
0 <start> 1 <expr> 0->1 2 <term> 1->2 3 <factor> 2->3 14 * 2->14 15 <term> 2->15 4 <integer> 3->4 10 . (46) 3->10 11 <integer> 3->11 5 <digit> 4->5 7 <integer> 4->7 6 1 (49) 5->6 8 <digit> 7->8 9 0 (48) 8->9 12 <digit> 11->12 13 3 (51) 12->13 16 <factor> 15->16 20 * 15->20 21 <term> 15->21 17 <integer> 16->17 18 <digit> 17->18 19 9 (57) 18->19 22 <factor> 21->22 58 * 21->58 59 <term> 21->59 23 ( (40) 22->23 24 <expr> 22->24 57 ) (41) 22->57 25 <term> 24->25 36 + 24->36 37 <expr> 24->37 26 <factor> 25->26 30 * 25->30 31 <term> 25->31 27 <integer> 26->27 28 <digit> 27->28 29 9 (57) 28->29 32 <factor> 31->32 33 <integer> 32->33 34 <digit> 33->34 35 9 (57) 34->35 38 <term> 37->38 43 + 37->43 44 <expr> 37->44 39 <factor> 38->39 40 <integer> 39->40 41 <digit> 40->41 42 8 (56) 41->42 45 <term> 44->45 50 - 44->50 51 <expr> 44->51 46 <factor> 45->46 47 <integer> 46->47 48 <digit> 47->48 49 7 (55) 48->49 52 <term> 51->52 53 <factor> 52->53 54 <integer> 53->54 55 <digit> 54->55 56 6 (54) 55->56 60 <factor> 59->60 71 * 59->71 72 <term> 59->72 61 <integer> 60->61 67 . (46) 60->67 68 <integer> 60->68 62 <digit> 61->62 64 <integer> 61->64 63 5 (53) 62->63 65 <digit> 64->65 66 5 (53) 65->66 69 <digit> 68->69 70 3 (51) 69->70 73 <factor> 72->73 77 / 72->77 78 <term> 72->78 74 <integer> 73->74 75 <digit> 74->75 76 3 (51) 75->76 79 <factor> 78->79 80 <integer> 79->80 81 <digit> 80->81 82 9 (57) 81->82

Let us try out the grammar fuzzer (and its trees) on other grammar formats.

In [121]:
f = GrammarFuzzer(URL_GRAMMAR)
f.fuzz()
Out[121]:
'ftp://user:password@fuzzingbook.com/abc?x58=def&def=3&x97=x40&abc=abc'
In [122]:
display_tree(f.derivation_tree)
Out[122]:
0 <start> 1 <url> 0->1 2 <scheme> 1->2 4 :// 1->4 5 <authority> 1->5 11 <path> 1->11 15 <query> 1->15 3 ftp 2->3 6 <userinfo> 5->6 8 @ (64) 5->8 9 <host> 5->9 7 user:password 6->7 10 fuzzingbook.com 9->10 12 / (47) 11->12 13 <id> 11->13 14 abc 13->14 16 ? (63) 15->16 17 <params> 15->17 18 <param> 17->18 28 & (38) 17->28 29 <params> 17->29 19 <id> 18->19 25 = (61) 18->25 26 <id> 18->26 20 x (120) 19->20 21 <digit> 19->21 23 <digit> 19->23 22 5 (53) 21->22 24 8 (56) 23->24 27 def 26->27 30 <param> 29->30 37 & (38) 29->37 38 <params> 29->38 31 <id> 30->31 33 = (61) 30->33 34 <nat> 30->34 32 def 31->32 35 <digit> 34->35 36 3 (51) 35->36 39 <param> 38->39 53 & (38) 38->53 54 <params> 38->54 40 <id> 39->40 46 = (61) 39->46 47 <id> 39->47 41 x (120) 40->41 42 <digit> 40->42 44 <digit> 40->44 43 9 (57) 42->43 45 7 (55) 44->45 48 x (120) 47->48 49 <digit> 47->49 51 <digit> 47->51 50 4 (52) 49->50 52 0 (48) 51->52 55 <param> 54->55 56 <id> 55->56 58 = (61) 55->58 59 <id> 55->59 57 abc 56->57 60 abc 59->60
In [123]:
f = GrammarFuzzer(CGI_GRAMMAR, min_nonterminals=3, max_nonterminals=5)
f.fuzz()
Out[123]:
'%d213'
In [124]:
display_tree(f.derivation_tree)
Out[124]:
0 <start> 1 <string> 0->1 2 <letter> 1->2 9 <string> 1->9 3 <percent> 2->3 4 % (37) 3->4 5 <hexdigit> 3->5 7 <hexdigit> 3->7 6 d (100) 5->6 8 2 (50) 7->8 10 <letter> 9->10 13 <string> 9->13 11 <other> 10->11 12 1 (49) 11->12 14 <letter> 13->14 15 <other> 14->15 16 3 (51) 15->16

How do we stack up against simple_grammar_fuzzer()?

In [125]:
trials = 50
xs = []
ys = []
f = GrammarFuzzer(EXPR_GRAMMAR, max_nonterminals=20)
for i in range(trials):
    with Timer() as t:
        s = f.fuzz()
    xs.append(len(s))
    ys.append(t.elapsed_time())
    print(i, end=" ")
print()

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
In [126]:
average_time = sum(ys) / trials
print("Average time:", average_time)

Average time: 0.014464827459305525
In [127]:
%matplotlib inline

import matplotlib.pyplot as plt
plt.scatter(xs, ys)
plt.title('Time required for generating an output');

Our test generation is much faster, but also our inputs are much smaller. We see that with derivation trees, we can get much better control over grammar production.

Finally, how does GrammarFuzzer work with expr_grammar, where simple_grammar_fuzzer() failed? It works without any issue:

In [128]:
f = GrammarFuzzer(expr_grammar, max_nonterminals=10)
f.fuzz()
Out[128]:
'--((3) / 7 * 9 / 0) * (9 + (7) * 4 * 3)'

With GrammarFuzzer, we now have a solid foundation on which to build further fuzzers and illustrate more exciting concepts from the world of generating software tests. Many of these do not even require writing a grammar – instead, they infer a grammar from the domain at hand, and thus allow using grammar-based fuzzing even without writing a grammar. Stay tuned!

Synopsis¶

Efficient Grammar Fuzzing¶

This chapter introduces GrammarFuzzer, an efficient grammar fuzzer that takes a grammar to produce syntactically valid input strings. Here's a typical usage:

In [130]:
phone_fuzzer = GrammarFuzzer(US_PHONE_GRAMMAR)
phone_fuzzer.fuzz()
Out[130]:
'(236)844-1154'

The GrammarFuzzer constructor takes a number of keyword arguments to control its behavior. start_symbol, for instance, allows setting the symbol that expansion starts with (instead of <start>):

In [131]:
area_fuzzer = GrammarFuzzer(US_PHONE_GRAMMAR, start_symbol='<area>')
area_fuzzer.fuzz()
Out[131]:
'996'

Here's how to parameterize the GrammarFuzzer constructor:

In [132]:
# ignore
import inspect
In [133]:
# ignore
print(inspect.getdoc(GrammarFuzzer.__init__))

Produce strings from `grammar`, starting with `start_symbol`.
If `min_nonterminals` or `max_nonterminals` is given, use them as limits 
for the number of nonterminals produced.  
If `disp` is set, display the intermediate derivation trees.
If `log` is set, show intermediate steps as text on standard output.
In [134]:
# ignore
from ClassDiagram import display_class_hierarchy
In [135]:
# ignore
display_class_hierarchy([GrammarFuzzer],
                        public_methods=[
                            Fuzzer.__init__,
                            Fuzzer.fuzz,
                            Fuzzer.run,
                            Fuzzer.runs,
                            GrammarFuzzer.__init__,
                            GrammarFuzzer.fuzz,
                            GrammarFuzzer.fuzz_tree,
                        ],
                        types={
                            'DerivationTree': DerivationTree,
                            'Expansion': Expansion,
                            'Grammar': Grammar
                        },
                        project='fuzzingbook')
Out[135]:
GrammarFuzzer GrammarFuzzer __init__() fuzz() fuzz_tree() any_possible_expansions() check_grammar() choose_node_expansion() choose_tree_expansion() expand_node() expand_node_by_cost() expand_node_max_cost() expand_node_min_cost() expand_node_randomly() expand_tree() expand_tree_once() expand_tree_with_strategy() expansion_cost() expansion_to_children() init_tree() log_tree() possible_expansions() process_chosen_children() supported_opts() symbol_cost() Fuzzer Fuzzer __init__() fuzz() run() runs() GrammarFuzzer->Fuzzer Legend Legend •  public_method() •  private_method() •  overloaded_method() Hover over names to see doc

Derivation Trees¶

Internally, GrammarFuzzer makes use of derivation trees, which it expands step by step. After producing a string, the tree produced can be accessed in the derivation_tree attribute.

In [136]:
display_tree(phone_fuzzer.derivation_tree)
Out[136]:
0 <start> 1 <phone-number> 0->1 2 ( (40) 1->2 3 <area> 1->3 10 ) (41) 1->10 11 <exchange> 1->11 18 - (45) 1->18 19 <line> 1->19 4 <lead-digit> 3->4 6 <digit> 3->6 8 <digit> 3->8 5 2 (50) 4->5 7 3 (51) 6->7 9 6 (54) 8->9 12 <lead-digit> 11->12 14 <digit> 11->14 16 <digit> 11->16 13 8 (56) 12->13 15 4 (52) 14->15 17 4 (52) 16->17 20 <digit> 19->20 22 <digit> 19->22 24 <digit> 19->24 26 <digit> 19->26 21 1 (49) 20->21 23 1 (49) 22->23 25 5 (53) 24->25 27 4 (52) 26->27

In the internal representation of a derivation tree, a node is a pair (symbol, children). For nonterminals, symbol is the symbol that is being expanded, and children is a list of further nodes. For terminals, symbol is the terminal string, and children is empty.

In [137]:
phone_fuzzer.derivation_tree
Out[137]:
('<start>',
 [('<phone-number>',
   [('(', []),
    ('<area>',
     [('<lead-digit>', [('2', [])]),
      ('<digit>', [('3', [])]),
      ('<digit>', [('6', [])])]),
    (')', []),
    ('<exchange>',
     [('<lead-digit>', [('8', [])]),
      ('<digit>', [('4', [])]),
      ('<digit>', [('4', [])])]),
    ('-', []),
    ('<line>',
     [('<digit>', [('1', [])]),
      ('<digit>', [('1', [])]),
      ('<digit>', [('5', [])]),
      ('<digit>', [('4', [])])])])])

The chapter contains various helpers to work with derivation trees, including visualization tools – notably, display_tree(), above.

Lessons Learned¶

  • Derivation trees are important for expressing input structure
  • Grammar fuzzing based on derivation trees
    1. is much more efficient than string-based grammar fuzzing,
    2. gives much better control over input generation, and
    3. effectively avoids running into infinite expansions.

Background¶

Derivation trees (then frequently called parse trees) are a standard data structure into which parsers decompose inputs. The Dragon Book (also known as Compilers: Principles, Techniques, and Tools) \cite{Aho2006} discusses parsing into derivation trees as part of compiling programs. We also use derivation trees when parsing and recombining inputs.

The key idea in this chapter, namely expanding until a limit of symbols is reached, and then always choosing the shortest path, stems from Luke \cite{Luke2000}.

Exercises¶

Exercise 1: Caching Method Results¶

Tracking GrammarFuzzer reveals that some methods are called again and again, always with the same values.

Set up a class FasterGrammarFuzzer with a cache that checks whether the method has been called before, and if so, return the previously computed "memoized" value. Do this for expansion_to_children(). Compare the number of invocations before and after the optimization.

Important: For expansion_to_children(), make sure that each list returned is an individual copy. If you return the same (cached) list, this will interfere with the in-place modification of GrammarFuzzer. Use the Python copy.deepcopy() function for this purpose.

Exercise 2: Grammar Pre-Compilation¶

Some methods such as symbol_cost() or expansion_cost() return a value that is dependent on the grammar only. Set up a class EvenFasterGrammarFuzzer() that pre-computes these values once upon initialization, such that later invocations of symbol_cost() or expansion_cost() need only look up these values.

Exercise 3: Maintaining Trees to be Expanded¶

In expand_tree_once(), the algorithm traverses the tree again and again to find nonterminals that still can be extended. Speed up the process by keeping a list of nonterminal symbols in the tree that still can be expanded.

Exercise 4: Alternate Random Expansions¶

We could define expand_node_randomly() such that it simply invokes expand_node_by_cost(node, random.choice):

In [149]:
class ExerciseGrammarFuzzer(GrammarFuzzer):
    def expand_node_randomly(self, node: DerivationTree) -> DerivationTree:
        if self.log:
            print("Expanding", all_terminals(node), "randomly by cost")

        return self.expand_node_by_cost(node, random.choice)

What is the difference between the original implementation and this alternative?