In web security, fuzz testing is becoming more popular by the day. RESTler expands the reach of fuzzing REST APIs by providing an intelligent, automated solution to do fuzzing given a Swagger/OpenAPI specification. In this article, Marina Polishchuk explains how RESTler works, how it is used and which bugs developers will find with RESTler.
Web APIs are everywhere right now. Over the past several years, there has been vast growth in public web APIs and microservice architectures. These services, which are continuously evolving, imply an increased attack surface and thus increasing demand for testing to keep up.
RESTler is the first stateful REST API fuzzer, developed at Microsoft Research. Our research was initially motivated by Microsoft-internal feedback about the need for deeper security testing for REST APIs. Previously, there were two main options: either using tools that required generating traffic that is then replayed and fuzzed, or loading up an API specification and fuzzing each request in isolation.
There are many ways to do traffic capture, but of course, the fuzzing coverage will depend on having representative traffic that can also be automatically updated when the service behavior changes. Fuzzing each request independently typically yields low coverage without a human in the loop to guide the fuzzer to improve coverage, e.g. by pre-creating the necessary resources. These solutions are simply too expensive to maintain with good coverage for large, frequently updated REST APIs.
RESTler first takes an OpenAPI specification and compiles it to produce a fuzzing grammar. The fuzzing grammar contains the information on how to fuzz each request type, and which resources must be passed between requests.
Then, the test engine executes requests against the service and analyzes test results. The engine has feedback loops to learn from past service responses and avoid redundant test cases. If bugs are found, they are bucketized and checked for reproducibility.
RESTler is very effective at finding input validation issues. The test oracle for these is getting 500 server errors instead of an expected 400.
With RESTler, we found several types of bugs in authentication. RESTler offers specific checkers to test whether you can access another user's resources without authorization. However, you can also just run RESTler with an invalid token across a large API. If it runs, and you see some 200s in the logs, this can be seen as an indicator that something might be going wrong.
RESTler offers specific checkers to test certain resource leak scenarios. However, service resource exhaustion can happen due to RESTler fuzzing, and can’t be detected by RESTler because it has no knowledge of the internal state of the back end service. If RESTler fuzzing causes resource exhaustion, this can be caught by service alerts and service incidents.
Although RESTler does not yet have an automation to find data leaks, some leaks have been found by examining the logs. Once such a leak is identified, a checker can be written to look for such leaks in responses across the entire API.
If you can run REST API fuzzing during development, a big problem is the time and the complexity of having a reliable test environment in CI/CD that will not break your builds. One strategy to mitigate this is doing RESTler testing during deployment validation, where a test environment is already available for other integration testing.
One thing we are driving toward is better coverage out-of-the-box. Today, if your service is relatively simple, the OpenAPI spec usually has enough detail to get near 100% coverage. If it is more complex, some initial configuration is required.
We have also created a self hosted REST API Fuzzing-As-A-Service platform, which can help you bootstrap REST API testing at scale. We are working on enhancing this solution to support more CI/CD scenarios, as well as regression fuzzing.
We recently received some great community feedback about specific security checkers that we did not initially cover. If you are interested or have already implemented a new checker, and you would like to integrate it or share your results, feel free to contact us on Github.
If you want to find out more about Stateful REST API Fuzzing, have a look at the recording of Marina Polishchuk’s talk at FuzzCon Europe - WebSecurity Edition.
Marina Polishchuk is a Research Software Engineer at Microsoft Research. She is passionate about systematic testing complex software as well as teaching engineering best practices. Over the last 15 years, she has written testing tools for the C++ compiler and the SQL server database engine. In her current work, she is focused on developing tools for finding security and reliability bugs in cloud services.