SBOM Challenge: Analysis a Day After

The S4x23 SBOM Challenge is all wrapped up! If you didn’t catch our play-by-play blog post on Day 1 or on Day 2, this challenge is the first of its kind technology challenge issued to vendors to identify the current state of innovation for generating and managing SBOMs - at the S4x23 Conference in Miami, FL. 

It was quite the experience between competitors and to an extent non-competitors, and it brought some much-needed visibility to the topic of the SBOM as well as the technology and the data analysis necessary to produce an SBOM. 

The challenge provided a window into the capabilities available, feature sets / capabilities still needed and the different approaches that each participating organization took in order to show the market what they can do. 

At the heart of this challenge, let’s face it - the SBOM has emerged as a critical component in the fight against hackers and to make up for the cybersecurity deficiencies in how software is composed today. 

And let’s not forget that while Executive Orders from the current administration, regulatory requirements and customer requirements are driving adoption, we still have a long way to go to get better, faster, and more comprehensive. Because it is not just the SBOM that matters - it is about driving down software supply chain risk. 

What does that mean? That means managing software and firmware vulnerabilities. It is about leveraging other data sets to synthesize analysis across tooling. It is delivering an accurate, reliable risk score with context to continuously manage risk across the AppSec stack and the Product organization’s roadmap to release connected products.

Day 3 found us with a much more relaxing day, with no specific milestones to be met for the SBOM Challenge. This, however, allows us an opportunity to wax philosophical, all without deadlines! We also received a briefing on the results of the challenge by INL - we appreciate all the work the INL team did to run this challenge and to capture where the market is, in terms of maturity and adoption. 

That said, in the end, Finite State was the ONLY platform that could perform in all aspects of the challenge. We were able to ingest binaries, generate SBOMs for all applicable software, identify CVEs from the SBOMs, identify non-CVE vulnerabilities in the binaries, apply vendor-provided VEX documents to those vulnerabilities, and ingest and share all of that data in any of the standard formats. Importantly, we identified more vulnerabilities than the other participants.

SBOM/VDR/VEX/CSAF as part of the CI/CD pipeline

Over the last two days we had lots of great questions about Finite State, but one of the themes that stood out was “Can I benefit from integration of an enhanced SBOM (with VDR and/or VEX) into my CI/CD pipeline?” The answer: Yes!

While it may not be feasible or reasonable to do a full firmware analysis for every build cycle in the pipeline, it makes a LOT of sense to analyze the individual binaries of the things that do change.

Setting the baseline up front with a full analysis and the resulting SBOM + VDR with the Finite State platform is almost like a penetration test of the firmware; the resulting data can open your eyes to a whole bunch of low hanging fruit to resolve. 

Resolving as many of these findings on the first round really sets the stage for subsequent, more focused analysis for specific changes. Analyzing individual build components, combined with regular full analysis provides a great way to track security improvement with documentation. Providing this documentation to those in the supply chain, effectively proves the level of commitment to evolving security improvement in your organization.

Oh, did we mention that the continuous analysis can pick up when older libraries or insecure methods are re-introduced? Continuous SBOM + VDR generation is also great to find when security improvements go backwards instead of forward; when a developer reintroduces a component, previously identified as being risky or insecure,  that had been eliminated in earlier builds. 

INL’s Presentation on SBOM challenge Findings

Today at 1:00 pm, INL presented the results from their perspective on the SBOM Challenge. 

One of the first conclusions from Ginger Wright and Hannah Kleinheider (the presenters from INL) was that they thought this would be able to issue an “apples to apples” comparison of the 5 vendors in the challenge.  

It turned out that they couldn’t do that, because each of the vendors had very different feature sets.  For example, Cybeats participated even though they do not generate SBOMs from binaries, and SRA used their consultants and open source tools (including FACT) to try and generate SBOMs. Thus, the challenge needed to be adapted to accommodate these different approaches to the market.

INL then revealed that the three artifacts that were tested (and were unknown to the participants) were:

• TPU 2000R - a legacy, bare metal firmware provided by Hitachi Energy (Artifact 2).
• OpenWRT (compiled for ARM) - a Linux-based open source router (Artifact 3).
• OSIsoft PI ProcessBook - a Windows-based software product (Artifact 1).

Let’s dive into the results for each of these to see how everyone stacked up against each other.

TPU 2000R

The INL team revealed that none of the competitors were able to produce an SBOM from the TPU 2000R binary. They noted that this was a bare metal firmware binary that was stripped of symbols (data that is useful for SBOM generation) and did not run any operating system. They chalked up the lack of results to a general lack of capability from the industry in this subset of software.

We have a different perspective on this, though. INL was able to provide everyone with an SBOM for the TPU 2000R, which was generated during the build process (possibly by hand).  The screenshot below shows that SBOM inside of the Finite State Next Generation Platform.

For anyone who has written firmware or C/C++ software, you might notice that this SBOM contains a set of header files from the C Standard Library. Those files are used to link to that library, but in general, this SBOM tells you very little about the software in the product.  You don’t actually know anything about any third-party software in this device, because there is no third-party software here.  

No matter how sophisticated the binary analysis tool is, there would be no meaningful SBOM that could be constructed. Even with the provided SBOM that was generated from the C header files, there is no useful information that can be gleaned from this.  The software identifiers tied to these header files will never link to known vulnerabilities.

So, what does this really mean? 

It’s important to keep in mind that SBOMs are only a part of the overall software supply chain security picture. They are a meaningful step forward in software transparency, and they can be incredibly useful for managing the risk of third-party software inside of devices, packages, or applications. 

But, what if there is no third-party software? In that case, other security testing and analysis, such as the static binary analysis and CWE detection in the Finite State Platform, will be necessary to find vulnerabilities. That will uncover vulnerabilities related to memory management (such as buffer overflows) and other vulnerabilities like hard coded credentials (which the Finite State team also found in this artifact).

In short, we respectfully disagree with the decision to include a software artifact with no meaningful third-party components in a challenge designed to test the ability of the participants to identify third-party components and manage vulnerabilities associated with them. However, we recognize that the INL team had challenges in gathering these artifacts from willing vendors, and we appreciate all of their efforts in running this challenge.

Finally, we want to be clear on one point. SBOMs absolutely can be generated for bare metal firmware, since oftentimes, third-party software (including open source components, third-party real-time operating systems, and IP stacks) are included.  Finite State has analyzed thousands of these types of firmware artifacts, and we’ve successfully generated SBOMs and found 0-day vulnerabilities in almost every case.  

This capability is an important part of software supply chain security, since many of our most critical devices are running embedded firmware, and Finite State’s capabilities are market leading in this field, with support for more than 50 different instruction set architectures, dozens of operating systems and chipsets, and hundreds of binary formats.

OpenWRT

The OpenWRT-based Linux binary artifact is much more aligned with the type of software where SBOMs can be very useful, and thus, this artifact was processed successfully by most of the participants. 

It should be noted, however, that aDolus has limited capabilities for embedded Linux firmware, and Cybeats does not conduct binary analysis for SBOM generation at all.  The results of the vendors’ analysis can be seen in the table below.

* identified non-CVE vulnerabilities as well

These results are interesting and show just how different the vendors’ capabilities are.  As you can see, Finite State was able to identify the most unique packages in the artifact.  We, of course, don’t know the details of what the other vendors reported, but you can see the highlights of our analysis in yesterday’s blog post.  

With a massive database of known software built out over the last several years and deep binary analysis capabilities that can identify software across architectures, operating systems, and platforms, we know that we have the best detection rates in the industry.

The unique files reported column has some interesting results.  In our opinion, the divergence here is due to the flexibility that vendors have in generating SBOMs rather than strong differences in the analysis capabilities.  

For example, today, the Finite State Platform only reports “File” components in an SBOM when they are a binary executable or library that is not part of an identified package.  We also will add “File” components into the dependency tree whenever we identify a statically linked subcomponent within a binary.  

Thus, we don’t include files such as text files, configuration files, symlinks, and others in an SBOM.  With the highest package identification rate and Finite State’s criteria for including “Files” in our SBOMs, we reported less total files. It appears the other vendors may have included all files inside the artifact (after their unpacking process) or no files at all. This might be an area we could better standardize on as an industry.

Finally, we were able to see the differences in the total number of vulnerabilities reported by each vendor for this artifact.  The Finite State Platform was able to find more vulnerabilities than the other vendors, and we also were able to identify non-CVE vulnerabilities in our analysis.

OSIsoft PI ProcessBook

As we analyzed the Windows artifact from INL, we were able to extract the files and we identified two common components that contained vulnerabilities. Many of the components we found, after further analysis, had no vulnerabilities or first-party code. You can see the results in the chart below. 

Final Conclusions from INL Briefing 

During INL’s debrief presentation, Ginger and Hannah made some comments that stuck with us:

• SBOMs are the beginning of a broader software transparency risk management ecosystem - The key word here is beginning. And software transparency. Oh, and risk management ecosystem. We can envision that the initial SBOM adoption across many verticals will be due to Cybersecurity Executive Order 14028, but real acceptance will be found when organizations realize that they can be incredibly useful for integrating into an overall risk management program  It will take time for that value to be realized, outside of the supply chain security industry to the border cybersecurity industry.
• Opportunity to work toward standardization in interoperability - We currently have two standards for SBOM format, CycloneDX and SPDX. And that is OK. Arguably, neither is complete, and each has their own unique challenges. However, regardless of the fact that two dominant formats exist, the market has to work toward interoperability so that SBOMs existing in both formats can coexist and can be used effectively in parallel without overlapping data or incompatible data formats that don’t provide the best results.  
• SBOMs do not include vulnerabilities - Taking this statement at its face value, it is completely accurate. An SBOM, is just that: A Software Bill of Materials, which starts to provide that “broader software transparency”. However, the further ecosystem around SBOM and supplemental data (VDR, VEX, etc.) DO contain vulnerabilities (at least vulnerability information which is what we think was the intent of the speakers’ comments). The enhanced SBOM ecosystem adds significant value to any organization to those that leverage the data: from improving the security during the development lifecycle, to audit before being released to manufacture, or to those implementing the affected binaries/products/firmware in their environment so that they can make better informed risk decisions. Importantly, VEX and VDR provide machine-readable vulnerability disclosures that will eventually be the primary way that software consumers get their vulnerability data. 

Conclusions

It’s clear from the results that the SBOM ecosystem is still pretty young, and we have more work to do as an industry. Different vendors had different approaches to solving challenges in reducing the risk that manufacturers ultimately take on when they build and ship connected products - and the same might be said for asset owners who need to know they are deploying secure products. 

As most markets mature from early tech to massive-scale adoption, more innovators will enter the “SBOM market” to solve specific challenges that surround the SBOM: generation, ingestion, data aggregation, software vulnerability management and remediation. We saw this first-hand in the SBOM Challenge.  

I think that all of us at Finite State were really excited to see the buzz generated not only to the entire sold out crowd at S4x23, but to those who came through the SBOM Pavilion and consumed the output of the SBOM Challenge.  

It was incredible to help bring understanding of the value of SBOMs to S4 attendees, especially the value outside of the need to come into compliance with Cybersecurity Executive Order 14028, or overall supply chain security. There is so much more value to an SBOM, especially when enhanced with VDR, VEX and CSAF data!

What came through pretty clearly in this challenge was that there are some fundamental shifts necessary in order to continue to push the SBOM into a must-have for teams looking to reduce software supply chain risk in order to play the long game in reducing risk across the AppSec and product security lifecycle. 

New Platform Release!

In short, with Finite State, you get the industry's only end-to-end SBOM solution inside the industry's most comprehensive software supply chain security solution. (so you're not just limited to SBOMs and CVEs)

We released an all new version of the Finite State platform on Day 1 and timed it so we could compete well in the SBOM Challenge. The new features and new data analysis capabilities were on full display and you can read about that announcement here. If you missed us at S4, you can easily request a demo here.