A few months ago, I set myself the goal of automating our vulnerability scan, and run it as part of our nightly builds. At that time I just started checking the different scanners that are out there, so I wasn’t attached to a particular scanner yet. I ended up with OWASP ZAP. Why? Because it’s free, it has an easy to use API and in general it’s just a great scanner. Maybe it’s not as complete as some of the expensive ones out there, but a very good start nonetheless. And because it is open source, there’s plenty of help available online.
I set a few objectives for this project. Obviously it had to be an automated process, so no user intervention to start the scan or interpret the results. The final result should be a ‘yes’ or a ‘no’, meaning: ‘yes’, everything is secure, there were no vulnerabilities detected, or ‘no’, there are potential vulnerabilities that need to be fixed or marked as false positive. Furthermore, the scan should have more or less repeatable results, so we can check whether previous findings are fixed the next day. The whole scan should run in more or less constant time, or at least not differ too much per run, and it should be done in a few hours. This last one proved to be the most difficult objective, because running the standard scan with all the options and without any constraints would run for several weeks easily, which is just not very practical for a ‘nightly’ scan.
There were also a few constraints. E.g. it should work with our build server TFS and the details about the vulnerabilities shouldn’t be sent across the network for everyone to see. A limited number of people have access to the scan results. It is fine however to send a summary to the build server.
These objectives and constraints led me to the following setup. I’ll first give a high level overview, and go into the details later.
ZAP is running on a Windows Server 2012 VM. Only a few people have access to this machine, so the final report can stay safely on this machine. I created a simple PHP WebService that uses the ZAP API and can be called by TFS. The WebService returns either a simple ‘OK’, or a summary of the findings.
By the way, I can’t advise the use of Windows here. I bumped into so many problems with installing stuff, memory issues, missing dll’s, etc. Partly this may however be caused by my lack of experience with this particular Windows Server version. The main reason for me to select this one, is that requesting a Linux VM would have taken years in our company, instead of ‘just 4 weeks’. If you want to automate a vulnerability scan in a similar way, take the OS that you feel most comfortable with. ZAP is very flexible and can adapt easily to your OS of choice.
The rest of this blog post will be written in a way that should allow others to follow the same steps and automate their vulnerability scan too.
Before you can automate anything, it’s best to first have a working manual scan that satisfies the objectives. For this we need to do a few things:
- Setup a ZAP Context
- Create an initializing ZEST script
- Create a scan policy
- Tune parameters
The ZAP context is the basis of the configuration. It will tell ZAP which URL’s are in scope and which aren’t. It will tell ZAP how authentication works, and how it can decide whether it’s logged in or out. Also, you can specify which technology you would like to target. By doing a manual scan first, you can build the ZAP context.
Initial Zest script
The ZAP spider needs to have an initial request in the site tree before it can do anything. Now, unless I’m missing something, the spider does not seem to be able to use the login URL to initiate this. It seems a bit unduly, but the only way I could find to automate creating that first request, is by using a ZEST script that does this.
Create a scan policy
The Scan Policy tells ZAP how strong the scan should be. If you want to scan mainly for XSS, then you set the strength of those tests higher, and other tests lower. Because we are doing a nightly scan that should finish within a few hours, I set strength a bit lower overall.
Finally, you should tune the parameters, so that the whole scan, including initialization, spidering and active scanning takes less than a few hours. Some ways to minimize the length of the scan are:
- Decrease the max depth for the spider
- Decrease the max children for the spider
- Use lower strength for the scan policy
- Take uninteresting parts of the site out of scope
Okay, on to the interesting part. You can use the ZAP API with any programming language, but I used PHP, because that’s what I do on a daily basis. For PHP a library is available for easy use of the API, but before I found that out, I already wrote my own. It’s just simple HTTP GET requests after all.
As said before, the biggest challenge was having the whole scan run automatically in more or less constant or at least predictable time. In the end it appeared that that means starting from scratch every time.
First I started with running ZAP in GUI mode under my own user account, so I could see what the API calls were doing. The API is available for webservice calls from outside anyway. That caused problems however with the session size. Although I started a new session every time (and used the overwrite option), session files kept growing. My .session.data file became more than 40GB and the server crashed. I think this is a bug in ZAP. Manually deleting the files doesn’t work, because they are locked. So, I had to restart ZAP on every run, which by the way is also an advantage, because that way it will automatically work when the server is rebooted. Now, because ZAP will be restarted from the webservice script, it cannot use GUI mode anymore. Fortunately, you can start ZAP in headless (daemon) mode.
Debugging the script in headless mode is a little more difficult, but if you want to debug, it shouldn’t be a problem to temporarily disable the restarting and use the GUI version to see what the API calls are doing.
Another challenge was loading the context, because a bug caused the structural parameters not to be loaded when importing the context file. Since our web application uses a structural parameter for differentiating between pages, that was absolutely required. Fortunately, the ZAP team was right on time with version 2.4.1 in which that issue was fixed. Kudos to the ZAP team!
Finally, it was difficult to get the spidering automated. As I told before, the spidering needs an initial request in the site tree, but by using a simple standalone ZEST script, that first request is easily generated.
My API calls can be broken down into four parts:
- Active Scanning
- Reporting findings
So, first you want to restart ZAP. If it’s still running, shut it down: (API call: ‘/JSON/core/action/shutdown/). If it wasn’t running, you’ll get a timeout, but the result is the same. If you need to delete old session files that have grown too big, now is the time to do this. Starting ZAP asynchronously in the background, in a way that the PHP script isn’t waiting for it to finish, appeared quite a challenge on Windows, but this is how I managed to get it to work:
popen('start zap.exe -daemon -config api.key=123456789abcdefg", 'r');
Now you start a new session: (/JSON/core/action/newSession/). Set parameters to overwrite the previous session, because you don’t need the previous one anymore. Results from every scan will be archived at the end. Optionally check if the context is already loaded by requesting a list of currently loaded contexts (/JSON/context/view/contextList/). If it hasn’t been loaded, then I load it (/JSON/context/action/importContext/).
Now run the Zest script, so the spidering can start. First you load it (/JSON/script/action/load/), then you run it (/JSON/script/action/runStandAloneScript/).
For later calls, you need to know both the contextId and the userId, so retrieve those also (/JSON/context/view/context/, and /JSON/users/view/usersList/).
Before starting the actual spidering, set the maximum spider depth (/JSON/spider/action/setOptionMaxDepth/), because spidering too deep requires a lot of extra time. I set it to 3 for a nightly scan. That will result in about 95-98% of all interesting URL’s. Further scanning will just result in more of the same pages with different parameters.
Now you can start spidering (/JSON/spider/action/scanAsUser/). For later calls you will need the scan id, which can be found by requesting a list of current scans (/JSON/spider/view/scans/).
Because you don’t know exactly how long the spidering takes, you can check on the status every few seconds and take action if needed (/JSON/spider/view/scans/). When I run the scan on the command line, I also write the progress percentage to the screen. Furthermore, if for some reason the spider scan is running (much) longer than expected, you can just cancel it (/JSON/spider/action/stop/) and continue with the results you have until then.
API calls for the active scan are similar to the spidering. First you add the required scan policy file (/JSON/ascan/action/addScanPolicy/). Then you start the scan (/JSON/ascan/action/scanAsUser/), and you wait until it is finished by checking the status every few seconds (/JSON/ascan/view/scans/), or until it has taken too long, after which you actively stop it (/JSON/ascan/action/stop/).
After the scan has finished, you check how many issues there are (/JSON/core/view/numberOfAlerts/). Hopefully there is nothing, and you can just return the highly anticipated and hoped for ‘OK’ to the build server.
However, if there are issues, you need to retrieve them (/JSON/core/view/alerts/) and report on them. Currently I use a list of regular expressions that analyze the resulting ‘parameter’, ‘evidence’, ‘url’ and ‘alert’ and filter out the false positives. I know you can mark alerts as ‘false positive’ in ZAP, but as far as I know you can mark only 1 item at a time, which is annoying if ZAP tells you you need a specific security header that you left out for a reason, and it tells you for all 1000+ different urls. Also, the false positives don’t seem to be part of the context but of the ZAP installation, which makes it difficult to transfer them between ZAP installations. So, for now I prefer the regex option.
For more details, I export a standard HTML report locally (/OTHER/core/other/htmlreport). This can later be used to reproduce the issues, and fix them.
After all the filtering is done, I return the summary to the build server. Because our team is using Slack as a collaboration and discussion tool, I decided to also send the summary there, so everybody in the team knows work needs to be done. Results ‘may’ look like this:
Now that I have this working, I can start thinking about the future. I have a few wishes to further improve on this:
- Better handling of false positives. The regular expressions are still not ideal, because the false positives are still exported to the HTML report. I guess the best way would be to have a ZAP add-on that allows you to graphically filter out multiple false positives at a time, while perhaps even adding remarks of why it’s a false positive. Obviously the configuration for this should be easily exportable to different installations of ZAP, and allow for easy backup.
- Optimize the scan policy, so the scan becomes more efficient.
- Add (Zest) scripts for more specific testing.
- Make the whole webservice and configuration more generic, so we can use this for other Products as well. Actually, I already started working on this one.
- Check out the ZAP code and start contributing…