What's the Best Java Scanning Tool?

Java is one of the most popular languages in software development—which means hackers are always trying to exploit what vulnerabilities they can in Java libraries. That’s why it’s important to find tools that allow you to scan your code for known vulnerabilities.

Our software composition analysis (SCA) tool is especially proficient in finding known vulnerabilities in your Java code—and for generating a very low amount of false positives.

But we’re not the only Java scanner out there.

In this article:

Java Scanning Tools Compared

We wanted to see how other tools on the market compare in terms of their abilities to catch vulnerabilities in Java code, so we compared a few tools on the market by pointing their free trial solutions at the very popular Spring Framework open-source project. This is a fairly complex project—and one that has several known java vulnerabilities.

NOTE: We wanted to see each tool’s immediate scanning capabilities (without setting up CI/CD pipelines) so we focused on tools that allow us to use command lines to scan our target library.

The high-level results:

Here’s an in-depth look at what each tool provided us:

Dependency-Check 

First up is Dependency-Check (OWASP), a free but excitable tool. Dependency-Check is easy to install and can be used completely locally—and it’s a common starting point for teams who want to level up their application security.

We ran the Spring Framework through Dependency Check, and here’s what it returned: 

That’s a pretty busy screenshot, but the main takeaway is that Dependency-Check flagged 226 CVEs, many of which were simple duplicates. When we de-duped the results, we were left with only 45 unique vulnerabilities.

SonaType Nexus

Sonatype’s Nexus was created to help developers manage components, binaries, and build artifacts across their entire software supply chain. Because we didn’t want to set up a pipeline product (SonaType Lift), we uploaded a .war file of the Spring Framework project and uploaded it to the nexus page.

Within a few minutes, we got an email from Sonatype with the following report:

Strangely, the scan turned up zero components, and therefore zero vulnerabilities—this was obviously a bum scan for Sonatype.

SNYK

Snyk is a popular tool in the premium software composition analysis space: they support a host of integrations, they maintain a thorough documentation bank, and their command line tool is very easy to install.

However, the process of getting CVE information from Snyk for this experiment was less than intuitive. We attempted to scan for CVE’s in the Spring Framework project using three different approaches (namely, a command to output a JSON file, a command to make the output available in our account’s user interface, and their IntelliJ IDE plugin).

While each method provided component names along with developer guidance for remediating vulnerabilities, it wasn’t easy to get the actual CVE names. This was important, as we wanted to cross-reference Snyk’s findings with the findings from other tools and more easily identify false positives.

The JSON output did include CVE names, however, the file was 153,552 lines long and listed 3305 CVEs—but only 26 of them were unique. And although the UI dashboard displays each component name, you need to click into each individual component to display CVE info.

Fine granularity is helpful, but a general snapshot of scan results would have been helpful, too. This is a lot to wade through just to get a comprehensive understanding of the scan we just performed.

Mend (formerly WhiteSource)

But we had no such qualms with Mend. We could easily install the program and conduct a scan from the command line. Furthermore, Mend generated a concise summary table within the command line once the scan was complete, giving us a quick outline of their findings followed by a long list with further information on each vulnerability—a helpful presentation of both high-level and granular information!

Mend returned every unique vulnerability that Snyk with one exception (CVE-2022-40155)—more on that later.

Finite State

Finally, we scanned the Spring Framework project using Finite State, resulting in a concise overview right in the command line:

And of course, the Finite State GUI gives a more detailed report:

Finite State finds the fewest number of vulnerabilities out of all of the tools—let’s see why.

Which Java scanner did the best job?

Now let’s pull all this together to see which tool did the job best. We’ll look at how the tools performed in terms of their ability to catch vulnerabilities, their ease of deployment, and their ability to generate useful, easy-to-understand reports.

Catching Vulnerabilities

Dependency-Check reports far more vulnerabilities than any other tool we checked—many of which are false positives, including CVE-2010-0538, which affects MacOs Apple Java. Unfortunately, this is in keeping with Dependency-Check’s reputation for generating false positives. Developers often start with Dependency-Check as a Java scanner, but given false positives’ negative effects on both security operations and team morale, it’s understandable why growing teams usually move on to a premium solution.

The disagreements between Snyk, Mend, and Finite State are mostly rooted in where each tool locates the vulnerability. For example, Finite State reports the undertow component as being vulnerable, whereas Mend and Snyk do not. But both Mend and Snyk report xnio as being vulnerable, whereas Finite State does not. This is because xnio is actually a dependency of undertow, so what’s going on is a difference in where in the dependency tree the vulnerability is being caught.

So why does Finite State report only 15 vulnerabilities while the other premium tools report 25 and 26? This is because Finite State doesn’t report dev dependencies such as Jython (which “provides implementations of Python in Java”)—instead it flags dependencies that actually end up in your production build.

Ease of deployment

SonaType was the only one we had any trouble getting working, and it was more intrusive than the other tools in terms of being able to scan quickly and privately.

Dependency-Check worked just fine, but only after using the IntelliJ Gradle plugin. Deploying Finite State, Mend, and Snyk was a quick and easy process all around.

Quality of reports

The ideal Java scanner will provide an immediate, high-level summary of the scan right in the CLI as well as thorough and well-organized reports in the GUI. While none of these tools generated reports that were difficult to understand, some reports were more immediately intuitive than others.

Dependency-Check was the clear loser in this respect: the results only really make sense if you see the report in the HTML page the tool generates when it completes a scan—their command line output is a wall of word salad.

Snyk was better but still leaves much to be desired in terms of intuitive reports. For example, Snyk found 72 medium violations—but does this represent 72 different CVEs or 72 different components are vulnerable? (We did some research: it’s the latter.) Furthermore, Snyk has the issue we mentioned previously: a massive granular report with no intuitive summary.

Mend does a bit better: they provide a summary in the CLI once it finishes a scan. But when needing JSON output, separates out the component information into one file and the vulnerability information in another. This means that in order to parse any real meaningful results, you need to stitch the two together by their matching SHAs. That’s a lot of extra work to understand the scan you just performed.

In some ways, Finite State has an unfair advantage here: while Snyk and Mend are broader cybersecurity solutions, Finite State’s software composition analysis tool was purpose-built to overcome these kinds of inconveniences developers experience when using other solutions. To that end, we generate a concise summary in the command line when a scan completes, as well as a thorough report in our GUI.

Run accurate Java code analysis with Finite State

Selecting the right Java code analysis tool is critical to protecting your product, your organization, and your users. We built Finite State to help you rapidly find true vulnerabilities (not false positives), remediate them, and keep your applications secure.

If you’d like to see how Finite State can keep your Java applications secure, start a free trial today and experience its advanced capabilities first hand.