Scanning .NET and NuGet projects for known vulnerabilities

We completed version 1 of our .NET scanner. Its goal is scanning .NET and NuGet projects for libraries with known vulnerabilities in any .NET project. For this blog post, we thought we’d take our scanner out for a spin and see how it compares against the competition.

TL;DR:

Here are the results of scanning .NET and Nuget projects for known vulnerabilities:

- Finite State – 18 vulnerabilities, 0 false positives.
- Snyk – 7 vulns and 5 false, or 4 vulns and 0 false (depends on scanner setup).
- WhiteSource – 12 vulns, 0 false.
- OWASP Dependency Check – 12 vulns, 17 false.
- Dotnet Retire – 2 vulns, 0 false.
- Sonatype – 0 vulns, 0 false.
- Dependabot – 0 vulns, 0 false.

Methodology

We chose the .NET Orleans project as to scan for .NET and NuGet vulnerabilities. It’s active, and complex, and it builds successfully (August 6th, 2020, master = 2e10856f7b7ed9443c). We also liked how this project contained a mix of Nuget styles (e.g., older “packages.config” style as well as the newer <“PackageReference”> style).

We type “dotnet build” before scanning. This way, .NET scanners can use the generated “obj/project.assets.json” files to supplement their scan data if they want to, and “dotnet build” is such a critical step for building any .NET project that we think it’s safe for an SCA tool to assume this command has completed successfully.

As for comparing results, we count CVE’s. If the scan outputs 1 or 300 or 9,000,000 hits against CVE-2018-8292, we count that as a single CVE. We then do a quick “desk check” to categorize the result as either a true-hit, a false-negative, or an ambiguous result (where it’s hard to say one way or the other). The “desk check” is very much based on my own decades of experience as a software engineer – I encourage others to rerun these scans and see if they agree or disagree.

Because this is a .NET scan for NuGet project, we ignore any results the scanners find from other file-types lying on the file system (e.g., “VotingWeb/wwwroot/lib/jquery/jquery.min.js”). We do, however, count results found from nuget references into other language artifacts (e.g., “GPSTracker.Web/packages.config” contains a nuget reference to “<package id=”bootstrap” version=”3.0.0″ targetFramework=”net45”/> in its packages.config file – we’ll count this.)

Here is the exact sequence of steps:

1. git clone https://github.com/dotnet/orleans.git
2. dotnet build
3. Deploy The Scanners!
4. Validate the results.

A note about ambiguous results:

We classify some results as ambiguous. This means there’s definitely some smoke, so we can’t immediately rule it out as a false negative after examining the metadata, but on the other hand, there’s enough uncertainty also to make us uncomfortable, considering it a true hit.

Example:

The vulnerability references “bootstrap” in the scan report, but the CVE description talks about “bootstrap-sass.” Maybe? Or in another case, the CVE description starts out with the words (in all caps) “DISPUTED.”

Results of scanning .NET and Nuget project for vulnerabilities:

I’ll save the best for first! Here’s what Finite State finds:

1. Finite State

18 vulnerabilities found (and two ambiguous hits).

Drop the scanner into the Orleans subdirectory and the results are pretty straightforward: 2 critical CVE’s, 5 high ones, and 11 mediums. A quick spot-check of the metadata looked good (no false positives and two ambiguous results).

2. Github Dependabot

Zero vulnerabilities found.

Dependabot not doing too much here, despite being a Microsoft product (albeit recently acquired):

3. Dotnet Retire

Two vulnerabilities found: CVE-2018-8292 and CVE-2018-8416. Finite State also found these two among the 18 vulnerabilities it identified.

4. Sonatype

Zero .NET vulnerabilities found!

Sonatype does detect a small handful of JavaScript vulnerabilities (since Orleans contains things like “VotingWeb/wwwroot/lib/jquery/jquery.min.js”), but nothing for .NET. To be fair, their scanner instructions did say “you must copy all .NET packages you depend on into the zip file you are scanning beforehand.” I typed “dotnet build” and zipped the result (660MB). As far as I’m concerned, I was doing them a favour by even zipping up orleans post-build in the first place – no other scanner required that.

Note-to-self: Probably Finite State should also scan those JavaScript packages! (Our current logic looks for NPM and Yarn lock files, but maybe it’s time to roll up our sleeves and consider scanning raw *.js and *.min.js files, too.)

5. Snyk

“It’s Complicated!” The problem with Snyk is that there’s two different ways to invoke the Snyk scanner, and each way returns wildly different results.

Snyk Approach #1 – Github Integration:

15 vulnerabilities found. 5 of those are false positives (all because Microsoft.NETCore.App was flagged as a dependency, but it’s not). 3 are ambiguous. 7 are true hits.

A few NPM and Docker vulnerabilities also found, but seeing as this bakeoff is only about .NET we ignored those.

Snyk Approach #2 – Command Line Invocation:

7 vulnerabilities found. 3 are ambiguous, leaving 4 true hits, including 1 true hit that Snyk approach #1 above did not find (CVE-2020-1469).

No NPM or Docker vulnerabilities found via this approach.

6. Whitesource Bolt

• 12 true CVE vulns.
• 3 ambigs.

7. OWASP Dependency Check

• 12 true CVE vulns.
• 3 true NON-CVE vulns.
• 2 ambigs.
• 17 falses

Unfortunately, OWASP Dependency Check is currently unable to handle .NET’s property substitution (e.g., when a *.csproj file references “Directory.Build.props”), a common convention for developers maintaining these files. This causes some frustrating false positives, such as reporting that “Google.Protobuf:$(GoogleProtobufVersion)” is vulnerable to CVE-2015-523.

OWASP Dependency Check also considers version 0.61.0 of the .NET MySqlConnector package to be vulnerable to 14 CVE’s – these are certainly all false positives. This is probably happening because Dependency Check considers version “0.61.0” to come before releases from MySQL’s popular version 5.x series, against which many CVEs have been filed over the years. However, version “0.61.0” of this package is less than 10 months old, making it impossible that it’s vulnerable to these ancient CVE’s.

Scanning .NET and Nuget Conclusion

Our own offering looks compelling in the .NET space. We were one of the top performers in this mini-benchmark, with Snyk, Mend (WhiteSource), and the open source OWASP Dependency Check tool also providing reasonable results. We were surprised to see Sonatype and Dependabot perform so poorly here. Software developers currently using the popular open source “Dotnet Retire” tool for this problem should definitely consider other options.

As always, with benchmarks such as these and security tools in general, your mileage can vary a lot based on the tools you’re using and your own particular context. I think a lot of companies become complacent with their existing tools. Similar to the smoke detectors in your house, it’s a good idea to benchmark your existing tools periodically just to ensure that they are still working properly!

Last piece of advice: Have .NET software? Give Finite State a closer look!