Sign up for the KDAB Newsletter
Stay on top of the latest news, publications, events and more.
Go to Sign-up
Many KDAB engineers are part of the Qt Security Team. The purpose of this team is to get notified of security-related issues, and then decide the best course of action for the Qt project.
Most of the time, this implies identifying the problem, creating and submitting a patch through the usual Qt contribution process, waiting for it to be merged in all the relevant branches, and then releasing a notice to the users about the extent of the security issue. We also work together with downstreams, such as our customers, Linux distributions and so on, in order to minimize the risks for Qt users of exposing the security vulnerability.
However, that's only part of the story. As part of the security team, we can't simply wait for reports to fall in our laps; we also need to have a proactive approach and constantly review our code base and poke it in order to find problems. For that, we use a variety of tools: the excellent Coverity Scan service; the sanitizers available in GCC and Clang; clazy, maintained by KDAB's engineer Sérgio Martins; and so on.
Note that all these tools help catch any sorts of bugs, not only the security-related ones. For instance, take a look at the issues found and fixed by looking at the Undefined Behavior Sanitizer's reports, and the issues fixed by looking at Coverity Scan's reports.
Today I want to tell you a little more about one of the tools used to test Qt's code: the American Fuzzy Lop, or AFL to friends.
What is AFL? It's a fuzzer: a program that keeps changing the input to a test in order to make it crash (or, in general, misbehave). This "mutation" of the input goes on forever -- AFL never ends, just keeps finding more stuff, and optimizes its own searching process.
AFL gained a lot of popularity because:
The results speaks for themselves: AFL has found security issues in all major libraries out there. Therefore, I decided to give it a try on Qt.
Setting up AFL is straightforward: just download it from its website and run make
. That's it -- this will produce a series of executables that will act as a proxy for your compiler, instrumenting the generated binaries with information that AFL will need. So, after this step, we will end up with afl-gcc
, afl-g++
and so on.
You can go ahead and build an instrumented Qt. If you've never built Qt from source, here's the relevant documentation. On Unix systems it's really a matter of running configure
with some options, followed by make
and optionally make install
. The problem at this step is making Qt use AFL's compilers, not the system ones. This turns out to be very simple, however: just export a few environment variables, pointing them to AFL's binaries:
export CC=/path/to/afl-gcc
export CXX=/path/to/afl-g++
./configure ...
make
And that's it, this will build an instrumented Qt. (A more thorough solution would involve creating a custom mkspec for qmake; this would have the advantage of making the final testscase application also use AFL automatically. For this task, however, I felt it was not worth it.)
What you need here is to create a very simple application that takes an input file from the command line (or stdin) and uses it to stress the code paths you want to test.
Now, when looking at a big library like Qt, there are many places where Qt reads untrusted input from the user and tries to parse it: image loading, QML parsing, (binary) JSON parsing, and so on. I decided to give a shot at binary JSON parsing, feeding it with AFL's mutated input. The testcase I built was straightforward:
#include <QtCore>
int main(int argc, char **argv)
{
QCoreApplication app(argc, argv);
QFile file(app.arguments().at(1));
if (!file.open(QIODevice::ReadOnly))
return 1;
QJsonDocument jd = QJsonDocument::fromBinaryData(file.readAll());
return 0;
}
Together with the testcase, you will also need a few test files to bootstrap AFL's finding process. These files should be extremely small (ideally, 1-2KB at maximum) to let the fuzzer do its magic. For this, just dump a few interesting files somewhere next to your testcase. I've taken random JSON documents, converted them to binary JSON and put the results in a directory.
Once the testcase is ready, you can run it into the fuzzer like this:
afl-fuzz -m memorylimit \
-t timeoutlimit \
[master/slave options] \
-i testcases/ \
-o findings/ \
-- ./test @@
A few explanatory remarks:
testcases
directory contains your reference input files, while the findings of the fuzzers will be written in findings
.memorylimit
megabytes of virtual memory and it is allowed to run for at most timeoutlimit
milliseconds. You will typically want to raise the memory limit from its default (50MB) to something bigger, depending on your system and on the test.-i
and -o
arguments. You should also give each instance a unique name and, if you want, elect one instance to do a deterministic search rather than a random one. This is all expressed through the master/slave options: pass to one instance the -M fuzzername
option, and to all the others pass the -S fuzzername
option. (All the fuzzernames must be unique).@@
gets replaced by the name of a file generated by AFL, containing the mutated input.For reference, I've launched my master like this:
afl-fuzz -m 512 -t 20 -i testcases -o findings-json -M fuzzer00 -- ./afl-qjson @@
The output is a nice colored summary of what's going on, updated in real time:
AFL running over a testcase.
Now: go do something else. This is supposed to run for days! So remember to launch it in a screen session, and maybe launch it via nice
so that it runs with a lower priority.
After running for a while, the first findings started to appear: inputs that crashed the test program or made it run for too long. Once AFL sees such inputs, it will save them for later inspection; you will find them under the findings/fuzzername
subdirectories:
findings-json/fuzzer00/crashes/id:000000,sig:06,src:000445,op:arith8,pos:168,val:+6
findings-json/fuzzer00/crashes/id:000001,sig:11,src:000445,op:arith8,pos:168,val:+7
findings-json/fuzzer00/crashes/id:000002,sig:11,src:000449,op:arith8,pos:196,val:+6
findings-json/fuzzer00/crashes/id:000003,sig:11,src:000489,op:flip1,pos:435
findings-json/fuzzer01/crashes/id:000000,sig:06,src:000526,op:havoc,rep:2
findings-json/fuzzer01/crashes/id:000001,sig:11,src:000532,op:havoc,rep:2
findings-json/fuzzer01/crashes/id:000002,sig:06,src:000533,op:havoc,rep:4
If you're lucky (well, I guess it depends how you look at it...), you will end up with inputs that indeed crash your testcase. Time to fix something!
You may also get false positives, in the form of crashes because the testcase runs out of memory. Remember that AFL imposes a strict memory limit on your executable, so if your testcase allocates too much memory and does not know how to recover from OOM it will crash. If you see many inputs crashing into AFL but not crashing when running normally, maybe your testcase is behaving properly, but just running out of memory, and increasing the memory limit passed to AFL will fix this.
The sig
part in the name of each saved input should give you a hint, telling you which Unix signal caused the crash. In the listing above, signal number 11 is a SIGSEGV, which is indeed a problem. The signal 06 is SIGABRT (that is, an abort), which was generated due to running out of memory.
To reproduce this last case, just manually run the test over that input, and check that it doesn't misbehave; then rerun it, but this time limiting its available memory via ulimit -v memory_available_in_kilobytes
. If the testcase works normally but crashes under a stricter ulimit
, it's likely that you're in an out-of-memory scenario. This may or may not require a fix in your code; it really depends whether it makes sense for your application/library to recover from an OOM.
After reporting the findings to the Security Team, it was a matter of a few days before a fix was produced, tested and merged into Qt. You can find the patches here and here.
If you want to play with AFL, I would recommend you to do a couple of things:
$ mkdir afl
# mount -t tmpfs -o size=1024M tmpfs afl/
$ cd afl/
$ afl-fuzz -i inputs -o findings ...
Fuzzing is an excellent technique for testing code that needs to accept untrusted inputs. It is straightforward to set up and run, requires no modifications to the tested code, and it can find issues in a relatively short timespan. If your application feature parsers (especially of binary data), consider to keep AFL running over it for a while, as it may discover some serious problems. Happy fuzzing!
About KDAB
The KDAB Group is a globally recognized provider for software consulting, development and training, specializing in embedded devices and complex cross-platform desktop applications. In addition to being leading experts in Qt, C++ and 3D technologies for over two decades, KDAB provides deep expertise across the stack, including Linux, Rust and modern UI frameworks. With 100+ employees from 20 countries and offices in Sweden, Germany, USA, France and UK, we serve clients around the world.
Stay on top of the latest news, publications, events and more.
Go to Sign-up
Upgrade your applications from Qt 5 to Qt 6 with KDAB’s migration services. Get a free migration assessment and join a hands-on workshop to prepare your team for a successful transition!
Learn more
Learn Modern C++
Our hands-on Modern C++ training courses are designed to quickly familiarize newcomers with the language. They also update professional C++ developers on the latest changes in the language and standard library introduced in recent C++ editions.
Learn more
Need help with performance issues?
Let the KDAB experts solve concrete performance problems, improve your software architecture or teach your team how to apply debugging and profiling tools to your developement process in the best way.
Get in touch
3 Comments
30 - Nov - 2016
Litb
Nice article, thanks. Have you considered to apply AFL to Qt GUI components. Like, perhaps fuzzing sequences of events onto a dialog that consists of a bunch of widgets?
5 - Dec - 2016
Giuseppe D'Angelo
Hi, that seems in interesting idea, but what would it accomplish? If it's for "monkey testing", that's already available in Squish.
More in general, I'd also like to explore libFuzzer and how to stress test things such as the CSS parser in QStyleSheetStyle, the HTML parser in QTextDocument and other similar textual-based inputs.
13 - Dec - 2016
Leslie Zhai
And libFuzzer http://llvm.org/docs/LibFuzzer.html for QTBUG-57553 https://bugreports.qt.io/browse/QTBUG-57553 AFL can also test for it ;-)