Testing Compilers¶

In this chapter, we will make use of grammars and grammar-based testing to systematically generate program code – for instance, to test a compiler or an interpreter. Not very surprisingly, we use Python and the Python interpreter as our domain.

We chose Python not only because the rest of the book is also based on Python. Most importantly, Python brings lots of built-in infrastructure we can leverage, especially

  • parsers that convert Python code into an abstract syntax tree (AST) representation and
  • unparsers that take an AST and convert it back into Python code.

This allows us to leverage grammars that operate on ASTs rather than concrete syntax, greatly reducing complexity.

Prerequisites

  • You must read the chapter on Fuzzing with Grammars to understand how grammars and grammar-based testing work.
In [2]:
# ignore
import sys
In [3]:
# ignore
if sys.version_info < (3, 10):
    print("This code requires Python 3.10 or later")
    sys.exit(0)

A Grammar for Concrete Code¶

To produce code, it is fairly easy to write a grammar with concrete syntax. If we want to produce, say, arithmetic expressions, we can easily create a concrete grammar which does precisely that.

We use the Fuzzingbook format for grammars, in which grammars are represented as dictionaries from symbols to lists of expansion alternatives.

In [7]:
EXPR_GRAMMAR: Grammar = {
    "<start>":
        ["<expr>"],

    "<expr>":
        ["<term> + <expr>", "<term> - <expr>", "<term>"],

    "<term>":
        ["<factor> * <term>", "<factor> / <term>", "<factor>"],

    "<factor>":
        ["+<factor>",
         "-<factor>",
         "(<expr>)",
         "<integer>.<integer>",
         "<integer>"],

    "<integer>":
        ["<digit><integer>", "<digit>"],

    "<digit>":
        ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"]
}
In [8]:
assert is_valid_grammar(EXPR_GRAMMAR)

We can use this grammar to produce syntactically valid arithmetic expressions. We use the ISLa solver as our generator, as it is the most powerful; but we could also use any other of our grammar fuzzers such as GrammarFuzzer at this point.

Here are some concrete inputs produced from the grammar:

In [10]:
expr_solver = ISLaSolver(EXPR_GRAMMAR)
for _ in range(10):
    print(expr_solver.solve())

4.3 + 512 / -(7 / 6 - 0 / 9 * 1 * 1) * +8.3 / 7 * 4 / 6
(4 / 7 + 1) / (4) / 9 / 8 + 4 / (3 + 6 - 7)
+--(--(-9) * (4 * 7 + (4) + 4) + --(+(3)) - 6 + 0 / 7 + 7)
(2 * 6 + 0 - 5) * 4 - +1 * (2 - 2) / 8 / 6
(+-(0 - (1) * 7 / 3)) / ((1 * 3 + 8) + 9 - +1 / --0) - 5 * (-+939.491)
+2.9 * 0 / 501.19814 / --+--(6.05002)
+-8.8 / (1) * -+1 + -8 + 9 - 3 / 8 * 6 + 4 * 3 * 5
(+(8 / 9 - 1 - 7)) + ---06.30 / +4.39
8786.82 - +01.170 / 9.2 - +(7) + 1 * 9 - 0
+-6 * 0 / 5 * (-(1.7 * +(-1 / +4.9 * 5 * 1 * 2) + -4.2 + (6 + -5) / (4 * 3 + 4)))

We could extend the grammar further to also produce assignments and other statements, and piece by piece cover the entire syntax of the programming language. However, this would be a not-so-great idea. Why?

The problem is that when testing compilers, you not only want to be able to produce code, but also to parse code, such that you can mutate and manipulate it at will. And this is where our "concrete" syntax will give us problems. While we can easily parse code (or expressions) that exactly adheres to the syntax...

In [11]:
expr_solver.check('2 + 2')
Out[11]:
True

... a single space will already suffice to make it fail...

In [12]:
expr_solver.check('2 +  2')

Error parsing "2 +  2" starting with "<start>"
Out[12]:
False

... as does the absence of spaces:

In [13]:
expr_solver.check('2+2')

Error parsing "2+2" starting with "<start>"
Out[13]:
False

Indeed, spaces are optional in most programming languages. We could update our grammar such that it can handle optional spaces at all times (introducing a <space> nonterminal). But then, there are other features like comments...

In [14]:
expr_solver.check('2 + 2    # should be 4')

Error parsing "2 + 2    # should be 4" starting with "<start>"
Out[14]:
False

... or continuation lines ...

In [15]:
expr_solver.check('2 + \\\n2')  # An expression split over two lines

Error parsing "2 + \
2" starting with "<start>"
Out[15]:
False

that our grammar would have to cover.

On top, there are language features that cannot be even represented properly in a context-free grammar:

  • In the C programming language, for instance, the parser needs to know whether an identifier has been defined as a type
  • In Python, indentation levels cannot be represented by a context-free grammar.

For this reason, it is often a good idea to make use of a dedicated parser (or preprocessor) to turn input into a more abstract representation - typically a tree structure. In programming languages, such a tree is called an abstract syntax tree (AST); it is the data structure that compilers operate on.

Abstract Syntax Trees¶

Abstract Syntax Trees (ASTs) that represent program code are among the most complex data structures in the world (if not the most complex data structures) - notably because they reflect all the complexity of the programming language and its features. The good news is that in Python, working with ASTs is particularly easy - one can work with them using standard language features.

Let us illustrate ASTs using an example. Here is a piece of code that we'd like to work with:

In [16]:
def main():
    print("Hello, world!")  # A simple example
In [17]:
main()

Hello, world!

Let us obtain the source code of this function:

In [19]:
main_source = inspect.getsource(main)
print(main_source)

def main():
    print("Hello, world!")  # A simple example

We make use of the Python AST module to convert this code string to an AST and back.

With ast.parse(), we can parse the main() source into an AST:

In [21]:
main_tree = ast.parse(main_source)

This is what this tree looks like:

In [23]:
show_ast(main_tree)
Out[23]:
0 Module 1 FunctionDef 0--1 2 "main" 1--2 3 arguments 1--3 4 Expr 1--4 5 Call 4--5 6 Name 5--6 9 Constant 5--9 7 "print" 6--7 8 Load 6--8 10 "Hello, world!" 9--10

We see how the function definition has become a FunctionDef node, whose third child is an Expr node, which in turn becomes a Call – of the "print" function with an argument of "Hello, world!".

Each of these AST nodes comes as a constructor – that is, we can invoke FunctionDef() to obtain a function definition node, or Call() to obtain a call node. These constructors take the AST children as arguments, but also lots of optional arguments (which we did not use so far). The dump of the AST into a string reveals all the arguments for each constructor:

In [24]:
print(ast.dump(main_tree, indent=4))

Module(
    body=[
        FunctionDef(
            name='main',
            args=arguments(
                posonlyargs=[],
                args=[],
                kwonlyargs=[],
                kw_defaults=[],
                defaults=[]),
            body=[
                Expr(
                    value=Call(
                        func=Name(id='print', ctx=Load()),
                        args=[
                            Constant(value='Hello, world!')],
                        keywords=[]))],
            decorator_list=[])],
    type_ignores=[])

The Python ast documentation lists all these constructors, which make up the abstract syntax. There are more than 100 individual constructors! (We said that ASTs are complex, right?)

The nice thing about the above string representation is that we can take it as is and turn it into a tree again:

In [26]:
my_main_tree = Module(
    body=[
        FunctionDef(
            name='main',
            args=arguments(
                posonlyargs=[],
                args=[],
                kwonlyargs=[],
                kw_defaults=[],
                defaults=[]),
            body=[
                Expr(
                    value=Call(
                        func=Name(id='print', ctx=Load()),
                        args=[
                            Constant(value='Hello, world!')],
                        keywords=[]))],
            decorator_list=[])],
    type_ignores=[])

We can take this tree and compile it into executable code:

In [27]:
my_main_tree = fix_missing_locations(my_main_tree)  # required for trees built from constructors
my_main_code = compile(my_main_tree, filename='<unknown>', mode='exec')
In [28]:
del main  # This deletes the definition of main()
In [29]:
exec(my_main_code)  # This defines main() again from `code`
In [30]:
main()

Hello, world!

We can also unparse the tree (= turn it into source code again). (Note how the comment got lost during parsing.)

In [31]:
print(ast.unparse(my_main_tree))

def main():
    print('Hello, world!')

Hence, we can

  1. Parse concrete code into ASTs (with ast.parse())
  2. Generate new ASTs and mutate existing ones
  3. Unparse ASTs to obtain concrete code again (with ast.unparse())

To generate and mutate ASTs (step #2, above), we need means to produce correct ASTs, invoking all constructors with the correct arguments. The plan is thus to have a grammar for ASTs, which produces (and parses) ASTs as we like.

A Grammar for ASTs¶

Programming language grammars are among the most complicated formal grammars around, and ASTs reflect much of this complexity. We will use the abstract AST grammar as specified in the Python documentation as base, and build a formal context-free grammar step by step.

Constants¶

We will start with simple constants – strings and integers. Again, we use the fuzzingbook syntax for grammars, as it allows for easier extension.

In [33]:
ANYTHING_BUT_DOUBLE_QUOTES_AND_BACKSLASH = (string.digits + string.ascii_letters + string.punctuation + ' ').replace('"', '').replace('\\', '')
ANYTHING_BUT_SINGLE_QUOTES_AND_BACKSLASH = (string.digits + string.ascii_letters + string.punctuation + ' ').replace("'", '').replace('\\', '')
In [34]:
ANYTHING_BUT_DOUBLE_QUOTES_AND_BACKSLASH
Out[34]:
"0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!#$%&'()*+,-./:;<=>?@[]^_`{|}~ "
In [35]:
ANYTHING_BUT_SINGLE_QUOTES_AND_BACKSLASH
Out[35]:
'0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&()*+,-./:;<=>?@[]^_`{|}~ '
In [36]:
PYTHON_AST_CONSTANTS_GRAMMAR: Grammar = {
    '<start>': [ '<expr>' ],

    # Expressions
    '<expr>': [ '<Constant>', '<Expr>' ],
    '<Expr>': [ 'Expr(value=<expr>)' ],

    # Constants
    '<Constant>': [ 'Constant(value=<literal>)' ],
    '<literal>': [ '<string>', '<integer>', '<float>', '<bool>', '<none>' ],

    # Strings
    '<string>': [ '"<not_double_quotes>*"', "'<not_single_quotes>*'" ],
    '<not_double_quotes>': list(ANYTHING_BUT_DOUBLE_QUOTES_AND_BACKSLASH),
    '<not_single_quotes>': list(ANYTHING_BUT_SINGLE_QUOTES_AND_BACKSLASH),
    # FIXME: The actual rules for Python strings are also more complex:
    # https://docs.python.org/3/reference/lexical_analysis.html#numeric-literals

    # Numbers
    '<integer>': [ '<digit>', '<nonzerodigit><digits>' ],
    '<float>': [ '<integer>.<integer>' ],
    '<nonzerodigit>': ['1', '2', '3', '4', '5', '6', '7', '8', '9'],
    '<digits>': [ '<digit><digits>', '<digit>' ],
    '<digit>': ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'],
    # FIXME: There are _many_ more ways to express numbers in Python; see
    # https://docs.python.org/3/reference/lexical_analysis.html#numeric-literals

    # More
    '<bool>': [ 'True', 'False' ],
    '<none>': [ 'None' ],

    # FIXME: Not supported: bytes, format strings, regex strings...
}

Note that we use extended Backus-Naur form in our grammars (here: <string>):

  • <elem>+ stands for one or more instances of <elem>;
  • <elem>* stands for zero or more instances of <elem>;
  • <elem>? stands for one or zero instances of <elem>.

A call to is_valid_grammar() ensures our grammar is free of common mistakes. Don't write grammars without it!

In [37]:
assert is_valid_grammar(PYTHON_AST_CONSTANTS_GRAMMAR)
In [38]:
constants_grammar = convert_ebnf_grammar(PYTHON_AST_CONSTANTS_GRAMMAR)
constants_solver = ISLaSolver(constants_grammar)
constants_tree_str = str(constants_solver.solve())
print(constants_tree_str)

Expr(value=Constant(value=None))

We can create an AST from this expression and turn it into Python code (well, a literal):

In [39]:
constants_tree = eval(constants_tree_str)
ast.unparse(constants_tree)
Out[39]:
'None'

Let's do this a number of times:

In [40]:
def test_samples(grammar: Grammar, iterations: int = 10, start_symbol = None, log: bool = True):
    g = convert_ebnf_grammar(grammar)
    solver = ISLaSolver(g, start_symbol=start_symbol, max_number_free_instantiations=iterations)
    for i in range(iterations):
        tree_str = str(solver.solve())
        tree = eval(tree_str)
        ast.fix_missing_locations(tree)
        if log:
            code = ast.unparse(tree)
            print(f'{code:40} # {tree_str}')
In [41]:
test_samples(PYTHON_AST_CONSTANTS_GRAMMAR)

False                                    # Expr(value=Constant(value=False))
2                                        # Constant(value=2)
None                                     # Constant(value=None)
'#'                                      # Constant(value="#")
550.81                                   # Constant(value=550.81)
True                                     # Constant(value=True)
'.'                                      # Constant(value='.')
467                                      # Constant(value=467)
7894                                     # Constant(value=7894)
263                                      # Constant(value=263)

Our grammar can also parse ASTs obtained from concrete code.

In [42]:
sample_constant_code = "4711"
sample_constant_ast = ast.parse(sample_constant_code).body[0]  # get the `Expr` node
sample_constant_ast_str = ast.dump(sample_constant_ast)
print(sample_constant_ast_str)

Expr(value=Constant(value=4711))
In [43]:
constant_solver = ISLaSolver(constants_grammar)
constant_solver.check(sample_constant_ast_str)
Out[43]:
True

Let us now come up with a quiz question: Does our grammar support negative numbers? For this, let's first find out if the Constant() constructor also take a negative number as an argument? It turns out it can:

In [44]:
ast.unparse(Constant(value=-1))
Out[44]:
'-1'

But what happens if we parse a negative number, say -1? One might assume that this simply results in a Constant(-1), right? Try it out yourself!

In [46]:
quiz("If we parse a negative number, do we obtain ",
    [
        "a `Constant()` with a negative value, or",
        "a unary `-` operator applied to a positive value?"
    ], 1 ** 0 + 1 ** 1)
Out[46]:

Quiz

If we parse a negative number, do we obtain



The answer is that parsing -1 yields a unary minus USub() applied to a positive value:

In [47]:
print(ast.dump(ast.parse('-1')))

Module(body=[Expr(value=UnaryOp(op=USub(), operand=Constant(value=1)))], type_ignores=[])

As unary operators are not part of our grammar (yet), it cannot handle negative numbers:

In [48]:
sample_constant_code = "-1"
sample_constant_ast = ast.parse(sample_constant_code).body[0]  # get the `Expr` node
sample_constant_ast_str = ast.dump(sample_constant_ast)
constant_solver = ISLaSolver(constants_grammar)
constant_solver.check(sample_constant_ast_str)

Error parsing "Expr(value=UnaryOp(op=USub(), operand=Constant(value=1)))" starting with "<start>"
Out[48]:
False

In the next sections, we will gradually expand our grammar with more and more Python features, eventually covering (almost) the entire language.

At this point, we have covered (almost) all AST elements of Python. There would be a few more Python elements to consider (marked as FIXME, above), but we'll leave these to the reader. Let us define PYTHON_AST_GRAMMAR as the official grammar coming out of this chapter.

In [95]:
PYTHON_AST_GRAMMAR = PYTHON_AST_MODULE_GRAMMAR
python_ast_grammar = convert_ebnf_grammar(PYTHON_AST_GRAMMAR)

Here are a few (very weird) examples of Python functions we can produce. All of these are valid, but only syntactically – very few of the code samples produced this way will actually result in something meaningful.

In [96]:
for elt in [ '<FunctionDef>' ]:
    print(elt)
    test_samples(PYTHON_AST_GRAMMAR, start_symbol=elt)
    print()

<FunctionDef>
def w():
    pass                        # FunctionDef(name='w', args=arguments(posonlyargs=[], args=[], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[Pass()], decorator_list=[])
def a():
    break                       # FunctionDef(name='a', args=arguments(posonlyargs=[], args=[], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[Break()], decorator_list=[])
def o():
    return                      # FunctionDef(name='o', args=arguments(posonlyargs=[], args=[], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[Return()], decorator_list=[])
def v(): # type: 
    continue           # FunctionDef(name='v', args=arguments(posonlyargs=[], args=[], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[Continue()], decorator_list=[], type_comment='')
def j(): # type: 
    return             # FunctionDef(name='j', args=arguments(posonlyargs=[], args=[], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[Return()], decorator_list=[], type_comment="")
def k():
    return
    return           # FunctionDef(name='k', args=arguments(posonlyargs=[], args=[], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[Return(), Return()], decorator_list=[])
def Q() -> set(): # type: 
    return    # FunctionDef(name='Q', args=arguments(posonlyargs=[], args=[], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[Return()], decorator_list=[], returns=Call(func=Name(id="set", ctx=Load()), args=[], keywords=[]), type_comment='')
def d() -> None:
    return
    assert set(), set()
    return # FunctionDef(name='d', args=arguments(posonlyargs=[], args=[], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[Return(), Assert(test=Call(func=Name(id="set", ctx=Load()), args=[], keywords=[]), msg=Call(func=Name(id="set", ctx=Load()), args=[], keywords=[])), Return()], decorator_list=[], returns=Constant(value=None))
def K() -> set():
    return             # FunctionDef(name='K', args=arguments(posonlyargs=[], args=[], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[Return()], decorator_list=[], returns=Call(func=Name(id="set", ctx=Load()), args=[], keywords=[]))
def y(): # type: 
    return             # FunctionDef(name='y', args=arguments(posonlyargs=[], args=[], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[Return()], decorator_list=[], type_comment='')

A Class for Fuzzing Python¶

For convenience, let us introduce a class PythonFuzzer that makes use of the above grammar in order to produce Python code. This will be fairly easy to use.

In [97]:
class PythonFuzzer(ISLaSolver):
    """Produce Python code."""

    def __init__(self,
                 start_symbol: Optional[str] = None, *,
                 grammar: Optional[Grammar] = None,
                 constraint: Optional[str] =None,
                 **kw_params) -> None:
        """Produce Python code. Parameters are:

        * `start_symbol`: The grammatical entity to be generated (default: `<FunctionDef>`)
        * `grammar`: The EBNF grammar to be used (default: `PYTHON__AST_GRAMMAR`); and
        * `constraint` an ISLa constraint (if any).

        Additional keyword parameters are passed to the `ISLaSolver` superclass.
        """
        if start_symbol is None:
            start_symbol = '<FunctionDef>'
        if grammar is None:
            grammar = PYTHON_AST_GRAMMAR
        assert start_symbol in grammar

        g = convert_ebnf_grammar(grammar)
        if constraint is None:
            super().__init__(g, start_symbol=start_symbol, **kw_params)
        else:
            super().__init__(g, constraint, start_symbol=start_symbol, **kw_params)

    def fuzz(self) -> str:
        """Produce a Python code string."""
        abstract_syntax_tree = eval(str(self.solve()))
        ast.fix_missing_locations(abstract_syntax_tree)
        return ast.unparse(abstract_syntax_tree)

By default, the PythonFuzzer will produce a function definition - that is, a function header and body.

In [98]:
fuzzer = PythonFuzzer()
print(fuzzer.fuzz())

def L():
    continue

By passing a start symbol as parameter, you can have PythonFuzzer produce arbitrary Python elements:

In [99]:
fuzzer = PythonFuzzer('<While>')
print(fuzzer.fuzz())

while (set()[set():set()], *(set())):
    if {}:
        while set():
            continue
        break
    else:
        del 
        return

Here is a list of all possible start symbols:

In [100]:
sorted(list(PYTHON_AST_GRAMMAR.keys()))
Out[100]:
['<Assert>',
 '<Assign>',
 '<Attribute>',
 '<AugAssign>',
 '<BinOp>',
 '<BoolOp>',
 '<Break>',
 '<Call>',
 '<Compare>',
 '<Constant>',
 '<Continue>',
 '<Delete>',
 '<Dict>',
 '<EmptySet>',
 '<Expr>',
 '<For>',
 '<FunctionDef>',
 '<If>',
 '<List>',
 '<Module>',
 '<Name>',
 '<Pass>',
 '<Return>',
 '<Set>',
 '<Slice>',
 '<Starred>',
 '<Subscript>',
 '<Tuple>',
 '<UnaryOp>',
 '<While>',
 '<With>',
 '<arg>',
 '<arg_list>',
 '<args>',
 '<args_param>',
 '<arguments>',
 '<bool>',
 '<boolop>',
 '<cmpop>',
 '<cmpop_list>',
 '<cmpops>',
 '<decorator_list_param>',
 '<defaults_param>',
 '<digit>',
 '<digits>',
 '<expr>',
 '<expr_list>',
 '<exprs>',
 '<float>',
 '<func>',
 '<id>',
 '<id_continue>',
 '<id_start>',
 '<identifier>',
 '<integer>',
 '<keyword>',
 '<keyword_list>',
 '<keywords>',
 '<keywords_param>',
 '<kw_defaults_param>',
 '<kwarg>',
 '<kwonlyargs_param>',
 '<lhs_Attribute>',
 '<lhs_List>',
 '<lhs_Name>',
 '<lhs_Starred>',
 '<lhs_Subscript>',
 '<lhs_Tuple>',
 '<lhs_expr>',
 '<lhs_exprs>',
 '<literal>',
 '<mod>',
 '<none>',
 '<nonempty_expr_list>',
 '<nonempty_lhs_expr_list>',
 '<nonempty_stmt_list>',
 '<nonzerodigit>',
 '<not_double_quotes>',
 '<not_single_quotes>',
 '<operator>',
 '<orelse_param>',
 '<posonlyargs_param>',
 '<returns>',
 '<start>',
 '<stmt>',
 '<stmt_list>',
 '<stmts>',
 '<string>',
 '<type_comment>',
 '<type_ignore>',
 '<type_ignore_list>',
 '<type_ignore_param>',
 '<type_ignores>',
 '<unaryop>',
 '<vararg>',
 '<withitem>',
 '<withitem_list>',
 '<withitems>']

Customizing the Python Fuzzer¶

When fuzzing, you may be interested in specific properties of the produced output. How can we influence the code that PythonFuzzer produces? We explore two ways:

  • By adjusting the grammar to our needs
  • By adding constraints that customize the output for us.

Adjusting the Grammar¶

A simple way to adjust output generation is to adapt the grammar.

Let us assume you'd like to have function definitions without decorators. To achieve this, you can alter the rule that produces function definitions:

In [101]:
PYTHON_AST_GRAMMAR['<FunctionDef>']
Out[101]:
['FunctionDef(name=<identifier>, args=<arguments>, body=<nonempty_stmt_list><decorator_list_param><returns>?<type_comment>?)']

As any AST rule, it comes in abstract syntax, so we first have to identify the element we'd like to adjust. In our case, this is decorator_list.

Since decorator_list is a list, we can alter the rule to produce empty lists only. To create a new adapted grammar, we do not alter the existing PYTHON_AST_GRAMMAR. Instead, we use the extend_grammar() function to create a new grammar with a new, adapted rule for <FunctionDef>:

In [102]:
python_ast_grammar_without_decorators: Grammar = extend_grammar(PYTHON_AST_GRAMMAR,
{
    '<FunctionDef>' :
        ['FunctionDef(name=<identifier>, args=<arguments>, body=<nonempty_stmt_list>, decorator_list=[])']
})

However, we're not done yet. We also need to ensure that our grammar is valid, as any misspelled nonterminal identifier will result in problems during production. For this, we use the is_valid_grammar() function:

In [104]:
with ExpectError():
    assert is_valid_grammar(python_ast_grammar_without_decorators)

'<decorator_list_param>': defined, but not used. Consider applying trim_grammar() on the grammar
'<returns>': defined, but not used. Consider applying trim_grammar() on the grammar
'<decorator_list_param>': unreachable from <start>. Consider applying trim_grammar() on the grammar
'<returns>': unreachable from <start>. Consider applying trim_grammar() on the grammar
Traceback (most recent call last):
  File "/var/folders/n2/xd9445p97rb3xh7m1dfx8_4h0006ts/T/ipykernel_32402/3611578183.py", line 2, in <cell line: 1>
    assert is_valid_grammar(python_ast_grammar_without_decorators)
AssertionError (expected)

We see that with our change, our grammar has an orphaned rule: The <returns> rule is no longer used. This is because <returns> is part of the <type_annotation> we just have deleted. (<type_annotation> is still used when defining types for variables.)

To fix this, we need to delete the <returns> rule from our grammar. Fortunately, we have a function trim_grammar(), which deletes all orphaned rules:

In [105]:
python_ast_grammar_without_decorators = trim_grammar(python_ast_grammar_without_decorators)

With this, our grammar becomes valid...

In [106]:
assert is_valid_grammar(python_ast_grammar_without_decorators)

... and we can use it for fuzzing - now without decorators:

In [107]:
fuzzer = PythonFuzzer(grammar=python_ast_grammar_without_decorators)
print(fuzzer.fuzz())

def X():
    break

Adjusting the grammar is straightforward once you understood the grammar structure, but the AST grammar is complex; also, your changes and extensions tie you closely to the grammar structure. Carefully study how the individual rules are defined, above.

Using Constraints for Customizing¶

A more elegant alternative to altering the grammar is to make use of constraints that tune the grammar to your needs. Since PythonFuzzer is derived from ISLaSolver, we can pass a constraint argument constraining the grammar, as discussed in the chapter on fuzzing with constraints.

If we want to have a function definition with 10 characters in each identifier, we make use of an ISLa constraint:

In [108]:
fuzzer = PythonFuzzer(constraint='str.len(<id>) = 10')
print(fuzzer.fuzz())

def yWOOLwypwp(): # type: 
    return

We can also constrain individual children – say, the actual identifier of the function.

In [109]:
# Also works (the <identifier> has quotes)
fuzzer = PythonFuzzer(constraint='<FunctionDef>.<identifier> = "\'my_favorite_function\'"')
print(fuzzer.fuzz())

@[set(), set()]
@set() | {}
@(-*set())[set():():
set()[:]()]
def my_favorite_function(dlFf=Qr, l1M=set(), *) -> 942.5:
    return

Assume we want to test how the compiler handles large numbers. Let us define a constraint such that the function body (<nonempty_stmt_list>) contains at least one integer (<integer>) with a value of at least 1000:

In [110]:
fuzzer = PythonFuzzer(constraint=
"""
    exists <integer> x:
        (inside(x, <nonempty_stmt_list>) and str.to.int(x) > 1000)
""")
print(fuzzer.fuzz())

@[set(), +set(), 
set()]
@{set(): set(), set(): set()}
@(set(), *set() & set())
def l(r, a, /, *uXLV, _=set()[:], **Z) -> sdTYWE9b or {set(), set().R}.Vy != z1vw([]):
    del 1008

Assume we'd like to test compilers with non-trivial functions. Here's how to define a constraint such that the function body has exactly three statements (<stmt>). Note that this can take more than a minute to resolve, but the result definitely is a nontrivial function.

In [111]:
# This will not work with ISLa 2
fuzzer = PythonFuzzer(constraint="""
    forall <FunctionDef> def: count(def, "<stmt>", "3")
""")
print(fuzzer.fuzz())

@3.91
def V8(w, /, *, t=set(), C5D=set(), **foT6):
    if *{}.S[:] - ((set()) not in set() in set()):
        break
    else:
        return

And finally, if we want the decorator list to be empty, as in our grammar-altering example, we can constrain the decorator list to be empty:

In [112]:
# ignore
# with ExpectError(mute=True):
#     # Triggers an ISLa error (AssertionError)
#     fuzzer = PythonFuzzer(constraint='''
#         str.contains(<FunctionDef>, "decorator_list=[]")
#     ''')
#     print(fuzzer.fuzz())
In [113]:
# ignore
# with ExpectError(mute=True):
#     # Triggers an ISLa error (AssertionError)
#     fuzzer = PythonFuzzer(constraint='<FunctionDef>.<expr_list> = "[]"')
#     print(fuzzer.fuzz())
In [114]:
fuzzer = PythonFuzzer(constraint='<FunctionDef>..<expr_list> = "[]"')
print(fuzzer.fuzz())

def l(Jws4IzSPx_O2ajk687obQB3mflULCTJWnAv9GHg0YRtVNycueKFDMihZ5rXd1pqEo, /, *, **g):
    return

Mutating Code¶

When producing code for compilers (or actually, producing inputs in general), it is often a good idea to not just create everything from scratch, but rather to mutate existing inputs. This way, one can achieve a better balance between common inputs (the ones to mutate) and uncommon inputs (the new parts added via mutation).

Parsing Inputs¶

To mutate inputs, we first need to be able to parse them. This is where a grammar is really put to test - can it really parse all possible code? This is why relying on an existing parser that is tried and proven (in our case the Python parser) and operating on an abstraction (in our case the AST) is really handy.

We already have seen how to parse code into an AST, using ast.parse():

In [115]:
def sum(a, b):    # A simple example
    the_sum = a + b
    return the_sum
In [116]:
sum_source = inspect.getsource(sum)
sum_tree = ast.parse(sum_source)
print(ast.unparse(sum_tree))

def sum(a, b):
    the_sum = a + b
    return the_sum
In [117]:
sum_str = ast.dump(sum_tree)
sum_str
Out[117]:
"Module(body=[FunctionDef(name='sum', args=arguments(posonlyargs=[], args=[arg(arg='a'), arg(arg='b')], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[Assign(targets=[Name(id='the_sum', ctx=Store())], value=BinOp(left=Name(id='a', ctx=Load()), op=Add(), right=Name(id='b', ctx=Load()))), Return(value=Name(id='the_sum', ctx=Load()))], decorator_list=[])], type_ignores=[])"

Our grammar is able to parse this (non_trivial) string:

In [118]:
solver = ISLaSolver(python_ast_grammar)
assert solver.check(sum_str)

To mutate the input, we first have to parse it into a derivation tree structure. This is (again) a tree representation of the code, but this time, using the elements of our grammar.

In [119]:
sum_tree = solver.parse(sum_str)

Let us inspect what a derivation tree looks like. Alas, the string representation is very long and not that useful:

In [120]:
len(repr(sum_tree))
Out[120]:
8737
In [121]:
repr(sum_tree)[:200]
Out[121]:
"DerivationTree('<start>', (DerivationTree('<mod>', (DerivationTree('<Module>', (DerivationTree('Module(body=', (), id=495073), DerivationTree('<nonempty_stmt_list>', (DerivationTree('[', (), id=495071"

However, we can visualize the derivation tree:

In [123]:
display_tree(sum_tree)
Out[123]:
0 <start> 1 <mod> 0->1 2 <Module> 1->2 3 Module(body= 2->3 4 <nonempty_stmt_list> 2->4 200 <type_ignore_param> 2->200 206 ) (41) 2->206 5 [ (91) 4->5 6 <stmts> 4->6 199 ] (93) 4->199 7 <stmt> 6->7 8 <FunctionDef> 7->8 9 FunctionDef(name= 8->9 10 <identifier> 8->10 23 , args= 8->23 24 <arguments> 8->24 81 , body= 8->81 82 <nonempty_stmt_list> 8->82 190 <decorator_list_param> 8->190 196 <returns-1> 8->196 197 <type_comment-3> 8->197 198 ) (41) 8->198 11 ' (39) 10->11 12 <id> 10->12 22 ' (39) 10->22 13 <id_start> 12->13 15 <id_continue-1> 12->15 14 s (115) 13->14 16 <id_continue> 15->16 18 <id_continue-1> 15->18 17 u (117) 16->17 19 <id_continue> 18->19 21 <id_continue-1> 18->21 20 m (109) 19->20 25 arguments( 24->25 26 <posonlyargs_param> 24->26 33 args= 24->33 34 <arg_list> 24->34 60 <vararg-1> 24->60 61 <kwonlyargs_param> 24->61 67 <kw_defaults_param> 24->67 73 <kwarg-1> 24->73 74 <defaults_param> 24->74 80 ) (41) 24->80 27 posonlyargs= 26->27 28 <arg_list> 26->28 32 , 26->32 29 [ (91) 28->29 30 <args-1> 28->30 31 ] (93) 28->31 35 [ (91) 34->35 36 <args-1> 34->36 59 ] (93) 34->59 37 <args> 36->37 38 <arg> 37->38 48 , 37->48 49 <arg> 37->49 39 arg(arg= 38->39 40 <identifier> 38->40 47 ) (41) 38->47 41 ' (39) 40->41 42 <id> 40->42 46 ' (39) 40->46 43 <id_start> 42->43 45 <id_continue-1> 42->45 44 a (97) 43->44 50 arg(arg= 49->50 51 <identifier> 49->51 58 ) (41) 49->58 52 ' (39) 51->52 53 <id> 51->53 57 ' (39) 51->57 54 <id_start> 53->54 56 <id_continue-1> 53->56 55 b (98) 54->55 62 , kwonlyargs= 61->62 63 <arg_list> 61->63 64 [ (91) 63->64 65 <args-1> 63->65 66 ] (93) 63->66 68 , kw_defaults= 67->68 69 <expr_list> 67->69 70 [ (91) 69->70 71 <exprs-1> 69->71 72 ] (93) 69->72 75 , defaults= 74->75 76 <expr_list> 74->76 77 [ (91) 76->77 78 <exprs-1> 76->78 79 ] (93) 76->79 83 [ (91) 82->83 84 <stmts> 82->84 189 ] (93) 82->189 85 <stmt> 84->85 154 , 84->154 155 <stmts> 84->155 86 <Assign> 85->86 87 Assign(targets= 86->87 88 <nonempty_lhs_expr_list> 86->88 121 , value= 86->121 122 <expr> 86->122 152 <type_comment-1> 86->152 153 ) (41) 86->153 89 [ (91) 88->89 90 <lhs_exprs> 88->90 120 ] (93) 88->120 91 <lhs_expr> 90->91 92 <lhs_Name> 91->92 93 Name(id= 92->93 94 <identifier> 92->94 119 , ctx=Store()) 92->119 95 ' (39) 94->95 96 <id> 94->96 118 ' (39) 94->118 97 <id_start> 96->97 99 <id_continue-1> 96->99 98 t (116) 97->98 100 <id_continue> 99->100 102 <id_continue-1> 99->102 101 h (104) 100->101 103 <id_continue> 102->103 105 <id_continue-1> 102->105 104 e (101) 103->104 106 <id_continue> 105->106 108 <id_continue-1> 105->108 107 _ (95) 106->107 109 <id_continue> 108->109 111 <id_continue-1> 108->111 110 s (115) 109->110 112 <id_continue> 111->112 114 <id_continue-1> 111->114 113 u (117) 112->113 115 <id_continue> 114->115 117 <id_continue-1> 114->117 116 m (109) 115->116 123 <BinOp> 122->123 124 BinOp(left= 123->124 125 <expr> 123->125 136 , op= 123->136 137 <operator> 123->137 139 , right= 123->139 140 <expr> 123->140 151 ) (41) 123->151 126 <Name> 125->126 127 Name(id= 126->127 128 <identifier> 126->128 135 , ctx=Load()) 126->135 129 ' (39) 128->129 130 <id> 128->130 134 ' (39) 128->134 131 <id_start> 130->131 133 <id_continue-1> 130->133 132 a (97) 131->132 138 Add() 137->138 141 <Name> 140->141 142 Name(id= 141->142 143 <identifier> 141->143 150 , ctx=Load()) 141->150 144 ' (39) 143->144 145 <id> 143->145 149 ' (39) 143->149 146 <id_start> 145->146 148 <id_continue-1> 145->148 147 b (98) 146->147 156 <stmt> 155->156 157 <Return> 156->157 158 Return(value= 157->158 159 <expr> 157->159 188 ) (41) 157->188 160 <Name> 159->160 161 Name(id= 160->161 162 <identifier> 160->162 187 , ctx=Load()) 160->187 163 ' (39) 162->163 164 <id> 162->164 186 ' (39) 162->186 165 <id_start> 164->165 167 <id_continue-1> 164->167 166 t (116) 165->166 168 <id_continue> 167->168 170 <id_continue-1> 167->170 169 h (104) 168->169 171 <id_continue> 170->171 173 <id_continue-1> 170->173 172 e (101) 171->172 174 <id_continue> 173->174 176 <id_continue-1> 173->176 175 _ (95) 174->175 177 <id_continue> 176->177 179 <id_continue-1> 176->179 178 s (115) 177->178 180 <id_continue> 179->180 182 <id_continue-1> 179->182 181 u (117) 180->181 183 <id_continue> 182->183 185 <id_continue-1> 182->185 184 m (109) 183->184 191 , decorator_list= 190->191 192 <expr_list> 190->192 193 [ (91) 192->193 194 <exprs-1> 192->194 195 ] (93) 192->195 201 , type_ignores= 200->201 202 <type_ignore_list> 200->202 203 [ (91) 202->203 204 <type_ignores-1> 202->204 205 ] (93) 202->205

We see that a derivation tree consists of nonterminal nodes whose children make up an expansion from the grammar. For instance, at the very top, we see that a <start> nonterminal expands into a <mod> nonterminal, which again expands into a <Module> nonterminal. This comes right from the grammar rules

In [124]:
python_ast_grammar['<start>']
Out[124]:
['<mod>']

and

In [125]:
python_ast_grammar['<mod>']
Out[125]:
['<Module>']

The child of <mod> is a <Module>, which expands into the nodes

  • (body=
  • <nonempty_stmt_list>
  • , type_ignores=
  • <type_ignore_list>
  • )

Here, nodes like (body= or , type_ignores= are called terminal nodes (because they have no more elements to expand). The nonterminals like <nonempty_stmt_list> get expanded further below – notably, <nonempty_stmt_list> expands into a <FunctionDef> node that represents the sum() definition.

Again, the structure exactly follows the <Module> definition in our grammar:

In [126]:
python_ast_grammar['<Module>']
Out[126]:
['Module(body=<nonempty_stmt_list><type_ignore_param>)']

If we traverse the tree depth-first, left to right, and only collect the terminal symbols, we obtain the original string we parsed. Applying the str() function to the derivation tree gets us exactly that string:

In [127]:
str(sum_tree)
Out[127]:
"Module(body=[FunctionDef(name='sum', args=arguments(posonlyargs=[], args=[arg(arg='a'), arg(arg='b')], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[Assign(targets=[Name(id='the_sum', ctx=Store())], value=BinOp(left=Name(id='a', ctx=Load()), op=Add(), right=Name(id='b', ctx=Load()))), Return(value=Name(id='the_sum', ctx=Load()))], decorator_list=[])], type_ignores=[])"

And again, we can convert this string into an AST and thus obtain our original function:

In [128]:
sum_ast = ast.fix_missing_locations(eval(str(sum_tree)))
print(ast.unparse(sum_ast))

def sum(a, b):
    the_sum = a + b
    return the_sum

Mutating Inputs¶

With derivation trees, we can have a structured representation of our input. In our case, we already have that with ASTs, so why bother introducing a new one? The answer is simple: Derivation trees also allow us to synthesize new inputs, because we have a grammar that describes their structure.

Most notably, we can mutate inputs as follows:

  1. Parse the input into a derivation tree, as shown above.
  2. Randomly choose some node <symbol> in the derivation tree to be mutated.
  3. Use the grammar to produce a new expansion for <symbol>.
  4. Replace the children of <symbol> by the expansion just generated.
  5. Repeat the process as often as needed.

This is a decent programming task, and if you'd like a blueprint, have a look at the FragmentMutator in this tutorial on greybox fuzzing with grammars.

Fortunately, ISLa already provides us with functionality that does exactly this. The ISLaSolver.mutate() method takes an input and mutates it according to the rules in the grammar. The input to mutate can be given as a derivation tree, or as a string; its output is a derivation tree (which can again be converted into a string).

Let us apply mutate() on our sum() function. The min_mutations and max_mutations parameters define how many mutation steps should be performed; we set both to 1 in order to have exactly one mutation.

In [129]:
sum_mutated_tree = solver.mutate(sum_str, min_mutations=1, max_mutations=1)
In [130]:
sum_mutated_ast = ast.fix_missing_locations(eval(str(sum_mutated_tree)))
print(ast.unparse(sum_mutated_ast))

def sum(a, b):
    the_sum = a + b
    return the_sum

Toy with the above to see the effect of a mutation. Note if one of the top-level nodes (like <FunctionDef> or <Module>) is selected for mutation, then sum() will be replaced by something entirely different. Otherwise, though, the code will still be pretty similar to the original sum() code.

Of course, the more we increase the number of mutations, the more different the code will look like:

In [131]:
sum_mutated_tree = solver.mutate(sum_str, min_mutations=10, max_mutations=20)
In [132]:
sum_mutated_ast = ast.fix_missing_locations(eval(str(sum_mutated_tree)))
print(ast.unparse(sum_mutated_ast))

def sum(a, b):
    the_9GuWCvL4cpgyi37K5I_ = a + b
    return the_jXHPe1oqMG

By toying with the mutate() parameters, we can control how common and how uncommon our input should be.

How Effective is Mutation?¶

Does mutating existing code help us in finding bugs? Let us assume we have a buggy compiler that generates bad code for an expression of the form <elem> * (<elem> + <elem>). The code in has_distributive_law() checks an AST for the presence of this bug:

In [133]:
def has_distributive_law(tree) -> bool:
    for node in walk(tree):  # iterate over all nodes in `tree`
        # print(node)
        if isinstance(node, ast.BinOp):
            if isinstance(node.op, ast.Mult):
                if isinstance(node.right, ast.BinOp):
                    if isinstance(node.right.op, ast.Add):
                        return True

                if isinstance(node.left, ast.BinOp):
                    if isinstance(node.left.op, ast.Add):
                        return True

    return False

To understand how this works, a visualization of the AST comes in handy:

In [134]:
show_ast(ast.parse("1 + (2 * 3)"))
Out[134]:
0 Module 1 Expr 0--1 2 BinOp 1--2 3 Constant 2--3 5 Add 2--5 6 BinOp 2--6 4 1 3--4 7 Constant 6--7 9 Mult 6--9 10 Constant 6--10 8 2 7--8 11 3 10--11
In [135]:
has_distributive_law(ast.parse("1 * (2 + 3)"))
Out[135]:
True
In [136]:
has_distributive_law(ast.parse("(1 + 2) * 3"))
Out[136]:
True
In [137]:
has_distributive_law(ast.parse("1 + (2 * 3)"))
Out[137]:
False
In [138]:
has_distributive_law(ast.parse("def f(a, b):\n    return a * (b + 10)"))
Out[138]:
True

How many attempts does it take for each until we find a mutation that triggers the bug in has_distributive_law()? Let us write a function that computes this number.

In [139]:
def how_many_mutations(code: str) -> int:
    solver = ISLaSolver(python_ast_grammar)

    code_ast = ast.parse(code)
    code_ast = ast.fix_missing_locations(code_ast)
    code_ast_str = ast.dump(code_ast)
    code_derivation_tree = solver.parse(code_ast_str)
    mutations = 0
    mutated_code_ast = code_ast

    while not has_distributive_law(mutated_code_ast):
        mutations += 1
        if mutations % 100 == 0:
            print(f'{mutations}...', end='')

        mutated_code_str = str(solver.mutate(code_derivation_tree))
        mutated_code_ast = eval(mutated_code_str)
        # mutated_code_ast = ast.fix_missing_locations(mutated_code_ast)
        # print(ast.dump(mutated_code_ast))
        # print(ast.unparse(mutated_code_ast))

    return mutations

If we pass an input that already exhibits the bug, we do not need any mutation:

In [140]:
assert how_many_mutations('1 * (2 + 3)') == 0

However, the further we are away from the bug, the more mutations (and the more time) it takes to find it. Notably, mutating 2 + 2 until we have a distributive law still is much faster than mutating 2.

In [141]:
how_many_mutations('2 + 2')    # <-- Note: this can take a minute
Out[141]:
54
In [142]:
how_many_mutations('2')  # <-- Note: this can take several minutes

100...200...300...400...500...600...700...800...900...1000...1100...1200...1300...1400...1500...1600...1700...1800...1900...2000...2100...2200...2300...2400...2500...
Out[142]:
2500

We conclude that mutating existing code can indeed be helpful, especially if it is syntactically close to inputs that trigger bugs. If you want to have a good chance in finding bugs, focus on inputs that have triggered bugs before – sometimes a simple mutation of these already helps finding a new bug.

Evolutionary Fuzzing¶

One interesting application of mutating inputs is to use mutations for evolutionary fuzzing. The idea is to have a population of inputs, to apply mutations on them, and to check whether they improve on a particular goal (mostly code coverage). Those inputs that do improve are being retained ("survival of the fittest") as the next generation, and evolved further. By repeating this process often enough, we may obtain inputs that cover large parts of code and thus improve chances to uncover bugs.

Let us assume we have a buggy compiler that generates bad code for an expression of the form <elem> * (<elem> + <elem>). The function has_distributive_law(), above, checks an AST for the presence of this bug.

Our aim is to detect this bug via fuzzing. But if we simply generate random inputs from scratch, it may take a long time until we generate the exact copmbination of operators that triggers the bug.

Getting Coverage¶

To have our fuzzers guided by coverage, we first need to measure code coverage. We make use of the Coverage module from the Fuzzing Book, which is particularly easy to use. It simply uses a with clause to obtain coverage from the code in the with body. Here is how to obtain coverage for our has_distributive_law() code, above:

In [144]:
mult_ast = ast.parse("1 * 2")
with Coverage() as cov:
    has_distributive_law(mult_ast)

The coverage() method tells us which lines in the code actually have been reached. This includes lines from has_distributive_law(), but also lines from other functions called.

In [145]:
cov.coverage()
Out[145]:
{('_handle_fromlist', 1063),
 ('_handle_fromlist', 1064),
 ('_handle_fromlist', 1071),
 ('_handle_fromlist', 1075),
 ('_handle_fromlist', 1087),
 ('has_distributive_law', 2),
 ('has_distributive_law', 4),
 ('has_distributive_law', 5),
 ('has_distributive_law', 6),
 ('has_distributive_law', 10),
 ('has_distributive_law', 14),
 ('iter_child_nodes', 264),
 ('iter_child_nodes', 265),
 ('iter_child_nodes', 266),
 ('iter_child_nodes', 267),
 ('iter_child_nodes', 268),
 ('iter_child_nodes', 269),
 ('iter_child_nodes', 270),
 ('iter_fields', 252),
 ('iter_fields', 253),
 ('iter_fields', 254),
 ('walk', 378),
 ('walk', 379),
 ('walk', 380),
 ('walk', 381),
 ('walk', 382),
 ('walk', 383)}

Which are the lines executed? With a bit of code inspection, we can easily visualize the covered lines:

In [146]:
def show_coverage(cov, fun):
    fun_lines, fun_start = inspect.getsourcelines(fun)
    fun_name = fun.__name__
    coverage = cov.coverage()
    for line in range(len(fun_lines)):
        if (fun_name, line + fun_start) in coverage:
            print('# ', end='')  # covered lines
        else:
            print('  ', end='')  # uncovered lines
        print(line + fun_start, fun_lines[line], end='')
In [147]:
show_coverage(cov, has_distributive_law)

  1 def has_distributive_law(tree) -> bool:
# 2     for node in walk(tree):  # iterate over all nodes in `tree`
  3         # print(node)
# 4         if isinstance(node, ast.BinOp):
# 5             if isinstance(node.op, ast.Mult):
# 6                 if isinstance(node.right, ast.BinOp):
  7                     if isinstance(node.right.op, ast.Add):
  8                         return True
  9 
# 10                 if isinstance(node.left, ast.BinOp):
  11                     if isinstance(node.left.op, ast.Add):
  12                         return True
  13 
# 14     return False

In this listing, a # indicates that the code has been executed (covered). We see that our input "1 * 2" satisfies the conditions in Lines 4 and 5, but does not satisfy the conditions in later lines.

Fitness¶

Let us now use coverage as a fitness function to guide evolution. The higher the fitness (the coverage), the higher the chances of an input to be retained for further evolution. Our ast_fitness() function simply counts the number of lines covered in has_distributive_law().

In [148]:
def ast_fitness(code_ast) -> int:
    with Coverage() as cov:
        has_distributive_law(code_ast)
    lines = set()
    for (name, line) in cov.coverage():
        if name == has_distributive_law.__name__:
            lines.add(line)
    return len(lines)

Here is the fitness of a number of given inputs:

In [149]:
ast_fitness(ast.parse("1"))
Out[149]:
3
In [150]:
ast_fitness(ast.parse("1 + 1"))
Out[150]:
4
In [151]:
ast_fitness(ast.parse("1 * 2"))
Out[151]:
6
In [152]:
ast_fitness(ast.parse("1 * (2 + 3)"))
Out[152]:
6

Now, let's set up a fitness function that takes derivation trees. Essentially, our tree_fitness() function is based on the ast_fitness() function, above; however, we also add a small component 1 / len(code_str) to give extra fitness to shorter inputs. Otherwise, our inputs may grow and keep on growing, making mutations inefficient.

In [153]:
def tree_fitness(tree) -> float:
    code_str = str(tree)
    code_ast = ast.fix_missing_locations(eval(code_str))
    fitness = ast_fitness(code_ast)
    # print(ast.unparse(code_ast), f"\n=> Fitness = {fitness}\n")
    return fitness + 1 / len(code_str)
In [154]:
tree_fitness(sum_tree)
Out[154]:
4.002666666666666

Evolving Inputs¶

Let us now make use of our fitness function to implement a simple evolutionary fuzzing algorithm. We start with evolution – that is, taking a population and adding offspring via mutations. Our initial population consists of a single candidate – in our case, sum_tree reflecting the sum() function, above.

In [155]:
def initial_population(tree):
    return [ (tree, tree_fitness(tree)) ]
In [156]:
sum_population = initial_population(sum_tree)
In [157]:
len(sum_population)
Out[157]:
1

Our evolve() function adds two new children to each population member.

In [158]:
OFFSPRING = 2
In [159]:
def evolve(population, min_fitness=-1):
    solver = ISLaSolver(python_ast_grammar)

    for (candidate, _) in list(population):
        for i in range(OFFSPRING):
            child = solver.mutate(candidate, min_mutations=1, max_mutations=1)
            child_fitness = tree_fitness(child)
            if child_fitness > min_fitness:
                population.append((child, child_fitness))
    return population
In [160]:
sum_population = evolve(sum_population)
len(sum_population)
Out[160]:
3

As we can evolve all these, too, we get an exponential growth.

In [161]:
sum_population = evolve(sum_population)
len(sum_population)
Out[161]:
9
In [162]:
sum_population = evolve(sum_population)
len(sum_population)
Out[162]:
27
In [163]:
sum_population = evolve(sum_population)
len(sum_population)
Out[163]:
81
In [164]:
sum_population = evolve(sum_population)
len(sum_population)
Out[164]:
243

Survival of the Fittest¶

No population can expand forever and still survive. Let us thus limit the population to a certain size.

In [165]:
POPULATION_SIZE = 100

The select() function implements survival of the fittest: It limits the population to at most POPULATION_SIZE elements, sorting them by their fitness (highest to lowest). Members with low fitness beyond POPULATION_SIZE do not survive.

In [166]:
def get_fitness(elem):
    (candidate, fitness) = elem
    return fitness

def select(population):
    population = sorted(population, key=get_fitness, reverse=True)
    population = population[:POPULATION_SIZE]
    return population

We can use the following call to trim our sum_population to the fittest members:

In [167]:
sum_population = select(sum_population)
len(sum_population)
Out[167]:
100

Evolution¶

We now have everything in place:

  • We have a population (say, sum_population)
  • We can evolve the population (using evolve())
  • We can have only the fittest survive (using select())

Let us repeat this process over several generations. We track whenever we have found a new "best" candidate and log them. If we find a candidate that triggers the bug, we stop. Note that this may take a long time, and not necessarily yield a perfect result.

As common in search-based approaches, we stop and restart the search if we have not found a sufficient solution after a number of generations (here: GENERATIONS). Other than that, we keep searching until we have a solution.

In [168]:
GENERATIONS = 100  # Upper bound
In [169]:
trial = 1
found = False

while not found:
    sum_population = initial_population(sum_tree)
    prev_best_fitness = -1

    for generation in range(GENERATIONS):
        sum_population = evolve(sum_population, min_fitness=prev_best_fitness)
        sum_population = select(sum_population)
        best_candidate, best_fitness = sum_population[0]
        if best_fitness > prev_best_fitness:
            print(f"Generation {generation}: found new best candidate (fitness={best_fitness}):")
            best_ast = ast.fix_missing_locations(eval(str(best_candidate)))
            print(ast.unparse(best_ast))
            prev_best_fitness = best_fitness

            if has_distributive_law(best_ast):
                print("Done!")
                found = True
                break

    trial = trial + 1
    print(f"\n\nRestarting; trial #{trial}")

Generation 0: found new best candidate (fitness=4.002666666666666):
def sum(a, b):
    the_sum = a + b
    return the_sum
Generation 1: found new best candidate (fitness=4.0027027027027025):
def sum(a, b):
    the_sum = a + b
    return FE
Generation 4: found new best candidate (fitness=4.002865329512894):
def sum():
    the_sum = a + b
    return the_sum
Generation 5: found new best candidate (fitness=6.00094696969697):
if set()[:] * *set():

    def sum(a, b):
        mc = a + b
        return FE
else:
    M = set()
continue

set().f[set():set()]()
Generation 7: found new best candidate (fitness=7.002364066193853):
def sum(a, b):
    mc = (a + b) * ()
    return FE
Done!


Restarting; trial #2

Success! We found a piece of code that triggers the bug. Check for occurrences of the distributive law.

In [170]:
print(ast.unparse(best_ast))

def sum(a, b):
    mc = (a + b) * ()
    return FE
In [171]:
assert has_distributive_law(best_ast)

You may note that not all of the code is required to trigger the bug. We could run our evolutionary fuzzer a bit longer to see whether it can be further reduced, or use a dedicated input reduction technique such as Delta Debugging.

Chances of Evolutionary Fuzzing¶

Could the bug in distributive_law() have been found without evolutionary guidance - i.e., simply by applying one mutation to sum()?

When producing an expression (<expr>), we calculate how big the chances are to

  • Produce a binary operator, and
  • Produce a *, and
  • Produce another binary operator as one child, and
  • Produce a +

Let's do a few queries on our grammar to compute the chances.

In [172]:
assert '<BinOp>' in python_ast_grammar['<expr>']
In [173]:
len(python_ast_grammar['<expr>'])
Out[173]:
15
In [174]:
assert 'Add()' in python_ast_grammar['<operator>']
assert 'Mult()' in python_ast_grammar['<operator>']
In [175]:
len(python_ast_grammar['<operator>'])
Out[175]:
13
In [176]:
(len(python_ast_grammar['<expr>'])       # chances of choosing a `BinOp`
* len(python_ast_grammar['<operator>'])  # chances of choosing a `*`
* len(python_ast_grammar['<expr>'])      # chances of choosing a `BinOp` as a child
* len(python_ast_grammar['<operator>'])  # chances of choosing a `+`
/ 2)   # two chances - one for the left child, one for the right
Out[176]:
19012.5

On average, it would take about 19000 (non-evolutionary) runs until we have an expression that triggers the distributive law. So it is definitely better to make use of additional information (say, coverage) in order to guide mutations towards a goal.

Synopsis¶

This chapter provides a PythonFuzzer class that allows producing arbitrary Python code elements:

In [177]:
fuzzer = PythonFuzzer()
print(fuzzer.fuzz())

def R():
    break

By default, PythonFuzzer produces a function definition – that is, a list of statements as above. You can pass a start_symbol argument to state which Python element you'd like to have:

In [178]:
fuzzer = PythonFuzzer('<While>')
print(fuzzer.fuzz())

while {set()[set():set():set()]}:
    C = set()
    D @= set()
    break
else:
    return

Here is a list of all possible start symbols. Their names reflect the nonterminals from the Python ast module documentation.

In [179]:
sorted(list(PYTHON_AST_GRAMMAR.keys()))
Out[179]:
['<Assert>',
 '<Assign>',
 '<Attribute>',
 '<AugAssign>',
 '<BinOp>',
 '<BoolOp>',
 '<Break>',
 '<Call>',
 '<Compare>',
 '<Constant>',
 '<Continue>',
 '<Delete>',
 '<Dict>',
 '<EmptySet>',
 '<Expr>',
 '<For>',
 '<FunctionDef>',
 '<If>',
 '<List>',
 '<Module>',
 '<Name>',
 '<Pass>',
 '<Return>',
 '<Set>',
 '<Slice>',
 '<Starred>',
 '<Subscript>',
 '<Tuple>',
 '<UnaryOp>',
 '<While>',
 '<With>',
 '<arg>',
 '<arg_list>',
 '<args>',
 '<args_param>',
 '<arguments>',
 '<bool>',
 '<boolop>',
 '<cmpop>',
 '<cmpop_list>',
 '<cmpops>',
 '<decorator_list_param>',
 '<defaults_param>',
 '<digit>',
 '<digits>',
 '<expr>',
 '<expr_list>',
 '<exprs>',
 '<float>',
 '<func>',
 '<id>',
 '<id_continue>',
 '<id_start>',
 '<identifier>',
 '<integer>',
 '<keyword>',
 '<keyword_list>',
 '<keywords>',
 '<keywords_param>',
 '<kw_defaults_param>',
 '<kwarg>',
 '<kwonlyargs_param>',
 '<lhs_Attribute>',
 '<lhs_List>',
 '<lhs_Name>',
 '<lhs_Starred>',
 '<lhs_Subscript>',
 '<lhs_Tuple>',
 '<lhs_expr>',
 '<lhs_exprs>',
 '<literal>',
 '<mod>',
 '<none>',
 '<nonempty_expr_list>',
 '<nonempty_lhs_expr_list>',
 '<nonempty_stmt_list>',
 '<nonzerodigit>',
 '<not_double_quotes>',
 '<not_single_quotes>',
 '<operator>',
 '<orelse_param>',
 '<posonlyargs_param>',
 '<returns>',
 '<start>',
 '<stmt>',
 '<stmt_list>',
 '<stmts>',
 '<string>',
 '<type_comment>',
 '<type_ignore>',
 '<type_ignore_list>',
 '<type_ignore_param>',
 '<type_ignores>',
 '<unaryop>',
 '<vararg>',
 '<withitem>',
 '<withitem_list>',
 '<withitems>']

If you'd like more control over Python code generation, here is what is happening behind the scenes. The EBNF grammar PYTHON_AST_GRAMMAR can parse and produce abstract syntax trees for Python. To produce a Python module without PythonFuzzer, you would take these steps:

Step 1: Create a non-EBNF grammar suitable for ISLaSolver (or any other grammar fuzzer):

In [180]:
python_ast_grammar = convert_ebnf_grammar(PYTHON_AST_GRAMMAR)

Step 2: Feed the resulting grammar into a grammar fuzzer such as ISLa:

In [181]:
solver = ISLaSolver(python_ast_grammar, start_symbol='<FunctionDef>')

Step 3: Have the grammar fuzzer produce a string. This string represents an AST.

In [182]:
ast_string = str(solver.solve())
ast_string
Out[182]:
'FunctionDef(name=\'y\', args=arguments(posonlyargs=[], args=[], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[Return()], decorator_list=[Call(func=Name(id="set", ctx=Load()), args=[], keywords=[])])'

Step 4: Convert the AST into an actual Python AST data structure.

In [184]:
abstract_syntax_tree = eval(ast_string)

Step 5: Finally, convert the AST structure back into readable Python code:

In [185]:
ast.fix_missing_locations(abstract_syntax_tree)
print(ast.unparse(abstract_syntax_tree))

@set()
def y():
    return

The chapter has many more applications, including parsing and mutating Python code, evolutionary fuzzing, and more.

Here are the details on the PythonFuzzer constructor:

In [186]:
# ignore
import inspect
import markdown
from bookutils import HTML
In [187]:
# ignore
sig = inspect.signature(PythonFuzzer.__init__)
sig_str = str(sig) if sig else ""
doc = inspect.getdoc(PythonFuzzer.__init__) or ""
HTML(markdown.markdown('`PythonFuzzer' + sig_str + '`\n\n' + doc))
Out[187]:

PythonFuzzer(self, start_symbol: Optional[str] = None, *, grammar: Optional[Dict[str, List[Union[str, Tuple[str, Dict[str, Any]]]]]] = None, constraint: Optional[str] = None, **kw_params) -> None

Produce Python code. Parameters are:

  • start_symbol: The grammatical entity to be generated (default: <FunctionDef>)
  • grammar: The EBNF grammar to be used (default: PYTHON__AST_GRAMMAR); and
  • constraint an ISLa constraint (if any).

Additional keyword parameters are passed to the ISLaSolver superclass.

In [188]:
# ignore
from ClassDiagram import display_class_hierarchy
In [189]:
# ignore
display_class_hierarchy([PythonFuzzer],
                        public_methods=[
                            PythonFuzzer.__init__,
                            PythonFuzzer.fuzz,
                            ISLaSolver.__init__
                        ],
                        project='fuzzingbook')
Out[189]:
PythonFuzzer PythonFuzzer __init__() fuzz() ISLaSolver ISLaSolver __init__() PythonFuzzer->ISLaSolver Legend Legend •  public_method() •  private_method() •  overloaded_method() Hover over names to see doc

Lessons Learned¶

  • When creating and processing complex inputs such as program code,
    • try to rely on existing infrastructure to parse inputs into some abstract syntax, and then
    • have your grammars process that abstract syntax rather than the concrete syntax.
  • Specifically, program code is normally converted into abstract syntax trees before being compiled or interpreted, and you can (and should) make use of such conversions.
  • Once program code is turned into an AST, it is fairly easy to generate, mutate, and evolve despite its complexity.

Background¶

The seminal work on compiler testing is Csmith \cite{Yang2011}, a generator of C programs. Csmith has been used to thoroughly test compilers such as Clang or GCC; beyond producing code that is syntactically correct, it also aims at semantic correctness as well as avoiding undefined and unspecified behaviors. This is a must read for anyone in the field in compiler testing.