Fuzz Testing - Overview

Recently, I was interested in learning more about Fuzz Testing, such things as what it is, how it is used, what tools exist and how they differ between each other and, in general, if fuzz testing is really a useful thing :D

After going through different resources I decided to document my personal findings in a form of this post.

Contents:

What is Fuzz Testing

Fuzz testing (fuzzing) is an automated software testing technique, so it is usually performed using a tool. This tool tries to find faults in a program by sending different inputs and observing the behavior. Inputs can be random as well as intentionally invalid and malformed.

This testing technique is not new, it emerged in 1980’s, but term “fuzzing” appeared only in 1988.

Real-World Applications

To make learning about fuzzing more interesting, I wanted to find some real-world applications of it to understand if it really works. Turns out that yes, it has many applications and extensively used in the industry. Here are some interesting cases:

This list can go on, I’ve picked just a few examples that resonated with me.

Also, fuzzing can be used in finding regressions in REST APIs, authors call it differential regression testing. [11]

Another interesting point is that fuzzing is often used by attackers to find vulnerabilities which can be exploited.

What Can Be Tested

In my understanding, all interfaces that accept input could be fuzz tested. Moreover, one application can undergo fuzzing on different levels.

For example, if we have a web api we could test endpoints with different parameter values, how server handles malformed HTTP requests, JSON parsing, code logic itself, database, etc.

Here are some broad and quite vague categories of what can be tested that I found (this list is by no means exhaustive):

One point to mention, fuzzing native code, for example, applications written in C/C++ helps reveal many bugs caused by the need for developers to handle memory management. This type of problems is less relevant to managed languages, e.g. C# and Java, since they take care of memory management.

What Faults & Problems It Looks For

In this section let us understand what a fuzzing tool looks for during execution.

Crashes and errors are the most obvious and simple to detect indicators that something went wrong in a program. For example, HTTP code 500 “Internal Server Error” clearly tells the tool that there is a significant issue.

However, many bugs do not cause system to crash immediately but are nonetheless important. To make fuzzer capable of detecting different types of issues, sanitizers can be used.

Please find below some examples of problems that can be found with the help of fuzzing. It should give a rough understanding.

As we can see, fuzz testing is capable of finding many different types of bugs and vulnerabilities, and some of them are very significant.

How Fuzzers Work

From the first glance it might appear that fuzzers just throw random inputs at the program in a hope to see it crash. Although it was a common approach in the early stages of fuzz testing, today we have more smart techniques of generating inputs that can reveal bugs in our programs.

Generations of Fuzzers

  1. First generation - random fuzzers. They generate random input and treat a program as a black box. These techniques still work today but not as well as other alternatives. We have already discussed some of the downsides of this approach. On the other hand, dumb fuzzers could be reused in more cases since they don’t have specific knowledge about system under test.

  2. Second generation - protocol/grammar based fuzzers. Here a tool takes some template which tells how to generate inputs. By providing a template, we constrain the set of possible things to explore.

  3. Third generation - instrumentation-guided fuzzing. Fuzzers of this type incorporate feedback and learn by watching how program executes with a given input. It helps the tool to come up with new test cases based on what it has seen.

Ways To Categorize Fuzzers

There are several ways to categorize fuzzers by its properties, any tool can belong to multiple categories mentioned below. You can read more about it on Wikipedia.

Advantages and Disadvantages

Advantages

Disadvantages

Fuzz Testing vs Random Testing vs Monkey Testing

The short answer is that it is the same thing. However, there are some differences which may be important for you.

At first, I was a bit confused when encountered all these terms. But in general for myself as a developer I concluded that it is roughly the same. At the same time I understand that not everyone would agree with me.

Here are some points and links that might be helpful in understanding differences:

Just a side note, I found Infinite monkey theorem pretty fun, some people believe that it has to do with the name of monkey testing.

Static Analysis and Fuzz Testing

Static analysis and fuzz testing are not the same:

Both techniques are useful and don’t replace each other. Moreover, fuzz testing can be used to verify problems reported by static analysis.

Tools

There are a number of fuzzing tools, commercial and open source. Here is a small list of what I’ve encountered so far: Defensics, Peach Fuzzer, Fuzzit, libFuzz, OWASP WSFuzzer, ZAProxy, OneFuzz, RESTler, etc.

Also, GitLab acquired Peach Tech and Fuzzit to provide fuzz testing as a part of its CI/CD environment.

Next, we’ll just talk about two tools: battle-tested AFL and quite new RESTler.

AFL - American Fuzzy Lop

This tool is well-known among fuzzing community. You might want to take a look at the list of products where AFL managed to find bugs. For instance, Firefox, Safari, nginx, OpenSSL, MySQL are among them. More examples here.

By the way, it is named after a cute rabbit breed.

For detailed information please visit AFL GitHub.

RESTler

RESTler is a stateful REST API fuzzing tool. This is quite new project developed by Microsoft Research.

Despite being relatively new, it has already found more than a hundred issues in Azure’s and GitLabs REST APIs.

More information about RESTler could be found in papers mentioned in RESTler GitHub.

Summary

Useful Resources

  1. https://en.wikipedia.org/wiki/Fuzzing
  2. https://browser-security.x41-dsec.de/X41-Browser-Security-White-Paper.pdf
  3. https://owasp.org/www-community/Fuzzing
  4. https://www.nytimes.com/2014/09/26/technology/security-experts-expect-shellshock-software-bug-to-be-significant.html
  5. https://www.blackhat.com/us-18/briefings/schedule/#god-mode-unlocked—hardware-backdoors-in-x86-cpus-10194
  6. https://www.techrepublic.com/article/fuzzing-fuzz-testing-tutorial-what-it-is-and-how-can-it-improve-application-security/
  7. https://builtin.com/software-engineering-perspectives/fuzz-testing
  8. https://owasp.org/www-project-web-security-testing-guide/v41/6-Appendix/C-Fuzz_Vectors
  9. https://github.com/google/AFL
  10. https://github.com/microsoft/restler-fuzzer
  11. https://patricegodefroid.github.io/public_psfiles/issta2020.pdf
  12. https://en.wikipedia.org/wiki/Differential_testing