Monthly Quality Report for SBOM Tools (October 2023)

Introduction

In our continuous effort to enhance the quality of Software Bill of Materials (SBOM) content, we are excited to present our monthly SBOM quality report. This report aims to provide insights into the performance of various SBOM tools, shedding light on their capabilities, strengths, and weaknesses. For the October edition, we have conducted extensive testing using a diverse set of tools to analyze the SBOM quality, license detection, vulnerability scanning, and pipeline support.

This time we use the new docker container image that contains already many of the SBOM and vulnerability scanner tools: codenotary/sbom.sh:v1.3 

The nice thing about that container image is, that it automatically ships the output to https://sbom.sh where the result is visualized and can be shared using a unique URL.

Tools

• • Aqua Trivy 0.46.0
  • Anchore Syft 0.93.0
  • Anchore Grype 0.71.0

In this blog post, we check the container image of Jenkinfis: jenkins/jenkins

Goal

The primary objective of this report is to create a baseline and gain comprehensive understanding of how various SBOM tools function. Additionally, we aim to assess the accuracy and comparability of their results. For this purpose, we have chosen a popular Maven Java Open Source application, Apache Pulsar, to test the SBOM tools against. Our evaluation goes beyond SBOM generation to include advanced functionality such as license information, vulnerability scanning, and pipeline support.

Overall Result

Findings:

1. There is a dependency gap between all used products. vcn has the lowest number as duplicates are not shown. Therefore, the number is misleading.
2. There is a vulnerability gap between Grype and other products.
3. CycloneDX exhibits the best SBOM quality, followed by vcn.
4. None of the tools differentiate between direct and transitive dependencies; all treat them as flat. That's an important area we work on as well.
5. None of the tools produce results similar to mvn dependency:tree, which may be acceptable in certain cases.

Testcase

For our evaluation, we selected the Jenkins container image, version 2.414.3. We aimed to establish a baseline using the Maven dependency tree, covering both used and unused but existing dependencies.

We assessed SBOM quality using SBOM Quality Score.

Important: the quality score measures the overall structure of the SBOM file, not the completeness!

Aqua Trivy 0.46.0

Command:

• Generate SBOM with the command: docker run -it --rm codenotary/sbom.sh:v1.3 trivyimage vulnscan jenkins/jenkins

Result:

• 172 components
• 58 vulnerabilities
• SBOM quality 6.3/10
• Command duration: 41 seconds

View SBOM

Anchore Syft 0.93.0

Command:

• Generate SBOM with the command: docker run -it --rm codenotary/sbom.sh:v1.3 syftimage jenkins/jenkins

Result:

• 479 components
• 0 vulnerabilities
• SBOM quality 7.56/10
• Command duration: 17 seconds

View SBOM

Anchore Grype 0.71.0

Integration:

• Download and install using wget and dpkg.

Command:

• Generate SBOM with the command: docker run -it --rm codenotary/sbom.sh:v1.3 grypeimage jenkins/jenkins

Result:

• 478 components
• 123 vulnerabilities
• SBOM quality 7.11/10
• Command duration: 18 seconds

View SBOM

Note: Grype and Syft work best in combination, and the Grype result was used.

 

 

Conclusion

This monthly SBOM quality report has provided valuable insights into the performance of various SBOM generation tools. It is essential to choose the right tool based on your specific requirements, as each tool comes with its own strengths and limitations. That's where Trustcenter comes in with its ability to consume all of the tools we looked into for this report. 

We encourage organizations to prioritize SBOM generation and leverage these reports to make informed decisions about tool selection and integration into their software development pipelines.