Mozilla has been fuzzing Firefox and its underlying components for a while. It has proven itself to be one of the most efficient ways to identify quality and security issues. In general, we apply fuzzing on different levels: there is fuzzing the browser as a whole but a significant amount of time is also spent on fuzzing isolated code (e.g. with libFuzzer) or even whole components such as the JS engine using separate shells with various fuzzers. For the purpose of this blog post, we will talk specifically about browser fuzzing only, and go into detail on the pipeline we’ve developed. This single pipeline is the result of several years of work that the fuzzing team has put into aggregating all of our browser fuzzing efforts with the objective of providing consistently actionable issues to developers, and to ease integration of internal and external fuzzing tools as they become available.
To be as effective as possible we make use of multiple different methods of detecting errors. These include sanitizers such as AddressSanitizer (with LeakSanitizer), ThreadSanitizer and UndefinedBehaviorSanitizer, as well as using debug builds that enable assertions and additional runtime checks. We also make use of debuggers such as rr and Valgrind. Each of these tools provide a different lens to help uncover specific bug types. Many of these tools are incompatible and require their own custom build in order to function or provide optimal results.
In addition to debugging and error detection, tools such as code coverage and libFuzzer also require instrumentation. Each operating system and architecture combination requires a unique build and may only support a subset of these tools.
Last, each variation has multiple active branches including Release, Beta, Nightly and Extended Support Release (ESR). Each of these are built periodically and are available via the Firefox CI Taskcluster instance.
Taskcluster makes it easy to find and download the latest build to test. However, we discussed above the number of variants created by different instrumentation types, and we need to fuzz all of them in automation. Because of the large number of combinations of builds, artifacts, architectures, operating systems and unpacking each, downloading is a non-trivial task.
In order to help reduce the complexity of build management, we developed a tool called fuzzfetch. Fuzzfetch makes it easy to specify the required build parameters and it will download and unpack the build. It also supports downloading specified revisions to make it useful with bisection tools.
As the goal of this blog post is to explain the whole pipeline, we won’t spend much time explaining fuzzers. If you are interested, please read https://hacks.mozilla.org/2020/04/fuzzing-with-webidl/ and the in-tree documentation. We use a combination of publicly available and custom built fuzzers to generate test cases.
For fuzzers that target the browser, Grizzly is used to manage and run test cases and monitor for results. Creating an adapter allows us to run existing fuzzers easily via Grizzly.
Multiple instances of Grizzly can be run in parallel to make full use of the available resources.
Containers are created for each type of fuzzer to encapsulate the configuration required to run it. These exist in the Orion monorepo. Each fuzzer has a configuration with deployment specifics and resource allocation depending on the priority of the fuzzer. This configuration is continuously deployed using Taskcluster to distribute work and manage fuzzing nodes.
When issues are detected they are automatically packaged and reported to a FuzzManager server. The detection of issues such as hangs, crashes and other defects is handled by a Grizzly Target which is an interface between the Grizzly and the browser. The FuzzManager server provides automation and a UI for triaging the results.
Other more targeted fuzzers use JS shell and libFuzzer based targets use the Fuzzing interface. Also many third party libraries are fuzzed in OSS-Fuzz. These deserve to be mentioned but are outside of the scope of this post.
Running multiple fuzzers against various targets at scale generates a large amount of data. Multiple variations and duplicate results are reported in a state that may not be easily actionable by developers and these are not suitable for direct entry into a bug tracking system such as Bugzilla. There are multiple components in place to manage this data.
When results are discovered by fuzzing nodes they are reported to a FuzzManager server. The FuzzManager web interface allows for the creation of signatures that help group reports together in buckets. The buckets are automatically scanned and reduction jobs are scheduled. The contents of each bucket are processed until a successful reduction has been completed.
It is common for fuzzers to generate test cases that are hundreds or even thousands of lines long. The reduction tasks use Grizzly Reduce and Lithium to apply different reduction strategies, often removing the majority of the unnecessary data. Once the automatic reduction is complete the minimized test case can be cleaned up and attached to a bug report. The tests are often later used as crash tests.
Code coverage of the fuzzer is also measured. These tasks are run periodically. The code coverage data is also sent to FuzzManager and stored for the creation of coverage reports.
Our goal is to create actionable bug reports to get issues fixed as soon as possible while minimizing overhead for developers.
We do this by providing:
Grizzly Replay is used by Bugmon, it is used to collect rr traces to submit to Pernosco and it is also the core of Grizzly Reduce. It makes re-running browser test cases easy both for automation and manual use. Situations with suborn test cases are simplified. Test cases that trigger multiple results can also be properly handled.
The combination of Grizzly and Pernosco have also had the added benefit of making infrequent, hard to reproduce, issues actionable. A test case for a very inconsistent issue can be run hundreds or thousands of times until a result is reproduced under rr. The trace is automatically collected and ready to be submitted to Pernosco and fixed by a developer, instead of being passed over because it was not actionable.
To initially request new features get proper assessment the fuzzing team can be reached at [email protected] or on Matrix. This is also a great way to get in touch for any reason and will be happy to help you with any fuzzing related questions. We will also reach out when we receive information about new initiatives and features that we think will require attention.
PrefPicker is used to manage prefs for fuzzing. When adding features behind a pref it must also be added to the PrefPicker fuzzing template to get it enabled during fuzzing.
Once fuzzing begins, most communication is done via Bugzilla. As mentioned previously, we strive to open actionable issues or enhance existing issues logged by others. When an issue is fixed, Bugmon will verify that the attached test case can no longer reproduce the issue.
Some issues found during fuzzing will prevent us from effectively fuzzing a feature or build variant. These are known as fuzz-blockers, and they come in a few different forms. These issues may seem benign from a product perspective, but they can block fuzzers from targeting important code paths or even prevent fuzzing a target altogether. Prioritizing these issues appropriately and getting them fixed quickly is very helpful and much appreciated by the fuzzing team. Periodic audits of the PrefPicker fuzzing template can help ensure areas are not missed and resources are used as effectively as possible.
We have also been making use of Pernosco. Pernosco is a tool that provides a web interface for rr traces and makes them available to developers without the need for direct access to the traces. It is an amazing tool developed by a company of the same name which significantly helps debugging massively parallel applications. It is also very helpful when test cases are too unreliable to reduce or attach to bug reports. Creating an rr trace and uploading it can make stalled bug reports actionable.
Bugmon is used to automatically bisect regression ranges. This helps notify the appropriate people as quickly as possible. It is also used to verify bugs once they are marked as FIXED. When bugs are closed they are also automatically removed from FuzzManager so they can be identified again if they or similar bugs find their way into the code base.
As in other fields, measurement is a key part of evaluating success. We leverage the meta bug feature of Bugzilla to help us keep track of the issues that are identified by fuzzers. We strive to have a meta bug per fuzzer and for each new component fuzzed.
For example, the meta bug for Domino lists all the issues (more than 1100!) identified by this tool. Using this Bugzilla data, we are able to show the impact over the years of our various fuzzers.
These dashboards are particularly useful to help evaluate the return on investment of a fuzzer.
There are many components in the fuzzing pipeline. The components are constantly evolving to facilitate changes in debugging tools, execution environments and browser internals. Developers are always adding, removing and updating browser features. Bugs are being detected, triaged and logged. Keeping everything running smoothly and targeting as much code as possible requires constant and ongoing efforts.
If you work on the browser you can help by keeping us informed of new features and initiatives that may affect or require fuzzing, by prioritizing fuzz-blockers, and by curating the prefs fuzzing template in PrefPicker. If you are interested in fuzzing and wish to take part in the bug bounty program our tools are available publicly and bug hunting is encouraged.
The post How we fuzz the browser at Mozilla appeared first on Mozilla Hacks - the Web developer blog.
Источник: moz://a HACKS
Наш сайт является информационным посредником. Сообщить о нарушении авторских прав.
Uncategorized automation fuzzing Testing
