Codenotary Trustcenter Blog

Monthly Quality Report for SBOM Tools

Written by Mahrukh | Sep 21, 2023 2:35:12 PM

Introduction

In our continuous effort to enhance the quality of Software Bill of Materials (SBOM) content, we are excited to present our monthly SBOM quality report. This report aims to provide insights into the performance of various SBOM tools, shedding light on their capabilities, strengths, and weaknesses. For this edition, we have conducted extensive testing using a diverse set of tools to analyze the SBOM quality, license detection, vulnerability scanning, and pipeline support.

Tools
    • Aqua Trivy 0.45.0
    • CycloneDX Core (Java) 7.3.2
    • CycloneDX Maven Plugin 2.7.9
    • Anchore Syft 0.89.0
    • Anchore Grype 0.66.0
    • vcn version v1.0.5
    • CycloneDX cdxgen 9.8.0

In this report, we delve into the world of SBOM generation tools, evaluating their performance in generating accurate and reliable SBOMs. This initiative is part of our ongoing commitment to promoting transparency, security, and the adoption of best practices in software development.

Goal

The primary objective of this SBOM quality report is to create a baseline and gain a comprehensive understanding of how various SBOM tools function. Additionally, we aim to assess the accuracy and comparability of their results. For this purpose, we have chosen a popular Maven Java Open Source application, Apache Pulsar, to test the SBOM tools against. Our evaluation goes beyond SBOM generation to include advanced functionality such as license information, vulnerability scanning, and pipeline support.

Overall Result

Findings:

  1. There is a dependency gap between all used products. vcn has the lowest number as duplicates are not shown. Therefore, the number is misleading.
  2. There is a vulnerability gap between Grype and other products.
  3. CycloneDX exhibits the best SBOM quality, followed by vcn.
  4. None of the tools differentiate between direct and transitive dependencies; all treat them as flat. That's an important area we work on as well.
  5. None of the tools produce results similar to mvn dependency:tree, which may be acceptable in certain cases.

Testcase

For our evaluation, we selected the Apache Pulsar project, available at https://github.com/apache/pulsar. We aimed to establish a baseline using the Maven dependency tree, covering both used and unused but existing dependencies.

The command used to obtain the baseline:

mvn dependency:tree | grep -Eo '[^ ]+:[^ ]+:[^:]+:[^:]+:[^:]+$' | sort | uniq | wc -l

Result: 2585

We assessed SBOM quality using SBOM Quality Score. Important: the quality score measures the overall structure of the SBOM file not the completeness!

Tools

We evaluated the following SBOM generation tools:

Aqua Trivy 0.45.0

Integration:

  • Download and install using wget and dpkg.

Command:

  • Generate SBOM with the command: trivy fs . --timeout 5m -f cyclonedx --scanners vuln --output sboms/trivy-sbom.json inside the application directory.

Result:

  • 1209 components
  • 32 vulnerabilities
  • 2016 licenses
  • SBOM quality issues
  • 2 errors in Maven constraints detected
  • Command duration: 2 minutes

View SBOM

CycloneDX Core (Java) 7.3.2

Integration:

  • Add the following section to pom.xml:
<dependency>
<groupId>org.cyclonedx</groupId>
<artifactId>cyclonedx-core-java</artifactId>
<version>7.3.2</version>
</dependency>
 

Command:

  • Generate SBOM with the command: mvn org.cyclonedx:cyclonedx-maven-plugin:makeBom inside the application.

Result:

  • 16 components
  • 0 vulnerabilities
  • Incomplete CycloneDX format (no serial)
  • Command duration: 15 minutes

View SBOM

CycloneDX Maven Plugin 2.7.9

Integration:

  • Add the following section to pom.xml:
<plugins>
<plugin>
<groupId>org.cyclonedx</groupId>
<artifactId>cyclonedx-maven-plugin</artifactId>
<configuration>
<!-- Configuration options -->
</configuration>
</plugin>
</plugins>
 

Command:

  • Generate SBOM with the command: mvn org.cyclonedx:cyclonedx-maven-plugin:makeBom inside the application.

Result:

  • 2 components (makeBom), 1031 (makeAggregateBom)
  • 0 vulnerabilities
  • 1102 licenses (makeAggregateBom)
  • Incomplete CycloneDX format (no serial)
  • Command duration: 6 minutes (makeBom), 32 seconds (makeAggregateBom)

View makeAggregateBom SBOM

Anchore Syft 0.89.0

Integration:

  • Download and install using wget and dpkg.

Command:

  • Generate SBOM with the command: syft . -o cyclonedx-json > sboms/syft-sbom.json inside the application directory.

Result:

  • 977 components
  • 0 vulnerabilities
  • Good SBOM quality
  • Command duration: 5 seconds

View SBOM

Anchore Grype 0.66.0

Integration:

  • Download and install using wget and dpkg.

Command:

  • Generate SBOM with the command: syft . -o cyclonedx-json > sboms/syft-sbom.json inside the application directory.

Result:

  • 977 components
  • 2085 vulnerabilities
  • Good SBOM quality
  • Command duration: 5 seconds

View SBOM

Note: Grype and Syft work best in combination, and the Grype result was used.

vcn version v1.0.5

Integration:

Command:

  • Generate SBOM with the command: vcn bom . --experimental --vuln-scan --bom-cdx-json vcnbom.json inside the application directory.

Result:

  • 597 components
  • 0 vulnerabilities
  • Good SBOM quality
  • Command duration: 30 seconds

View SBOM

CycloneDX cdxgen 9.8.0

Integration:

Command:

  • Generate SBOM with the Docker command: docker run --rm -v /tmp:/tmp -v $(pwd):/app:rw -t ghcr.io/cyclonedx/cdxgen -r /app -o /app/sboms/cdxgen-bom.json inside the application directory.

Result:

  • 1394 components
  • 0 vulnerabilities
  • 1411 licenses
  • Parse error
  • Command duration: 10 minutes

View SBOM

 

 

Conclusion

This monthly SBOM quality report has provided valuable insights into the performance of various SBOM generation tools. It is essential to choose the right tool based on your specific requirements, as each tool comes with its own strengths and limitations. That's where Trustcenter comes in with its ability to consume all of the tools we looked into for this report. 

We encourage organizations to prioritize SBOM generation and leverage these reports to make informed decisions about tool selection and integration into their software development pipelines.
View full post