Generating SBOM for Python Projects.

[MohammedAziz02]

Hi @prabhu,
When I generate the SBOM for the same Python repository dbt-oracle using the command cdxgen -t python -o sbom.json --spec-version 1.4, even with cdxgen -t python -o sbom.json --spec-version 1.4 --deep, I encounter different results in the dependencies graph. Here are the observations:

• SBOM 1 (example of a dependency in the graph in the first run)
  {
  "ref": "pkg:pypi/[email protected]",
  "dependsOn": [
  "pkg:pypi/[email protected]",
  "pkg:pypi/[email protected]", // this one does not exist in 2
  "pkg:pypi/[email protected]", // this one does not exist in 2
  "pkg:pypi/[email protected]",
  "pkg:pypi/[email protected]", // this one does not exist in 2
  "pkg:pypi/[email protected]",
  "pkg:pypi/[email protected]", // this one does not exist in 2
  "pkg:pypi/[email protected]", // this one does not exist in 2
  "pkg:pypi/[email protected]",
  "pkg:pypi/[email protected]", // this one does not exist in 2
  "pkg:pypi/[email protected]", // this one does not exist in 2
  "pkg:pypi/[email protected]",
  "pkg:pypi/[email protected]", // this one does not exist in 2
  "pkg:pypi/[email protected]", // this one does not exist in 2
  "pkg:pypi/[email protected]", // this one does not exist in 2
  "pkg:pypi/[email protected]",
  "pkg:pypi/[email protected]", // this one does not exist in 2
  "pkg:pypi/[email protected]",
  "pkg:pypi/[email protected]" // this one does not exist in 2
  ]
  }
• SBOM 2 (example of the same dependency in the graph in the second run)
  {
  "ref": "pkg:pypi/[email protected]",
  "dependsOn": [
  "pkg:pypi/[email protected]",
  "pkg:pypi/[email protected]",
  "pkg:pypi/[email protected]",
  "pkg:pypi/[email protected]",
  "pkg:pypi/[email protected]",
  "pkg:pypi/[email protected]", // this one does not exist in 1
  "pkg:pypi/[email protected]", // this one does not exist in 1
  "pkg:pypi/[email protected]",
  "pkg:pypi/[email protected]" // this one does not exist in 1
  ]
  }
  I examined the Python script here and noticed that the order of dependencies impacts the results. the code does not handle cycles effectively. For instance, if we have a cycle such as A -> B and B -> A, the resulting graph will be incorrect.

[prabhu]

@MohammedAziz02 This is a good observation. Are there any errors reported by cdxgen with the DEBUG environment variable? I remember adding a message or a comment that essentially says some SBOM is better than no SBOM, since the state of the virtual environment could be erroneous.

[prabhu]

@MohammedAziz02 Can you create a new virtual environment and run the piptree script directly to see how this could be improved?

[MohammedAziz02]

@prabhu , What I found is that we can directly use the path to handle such a situation. Let's suppose we have A -> B, E and B -> C and C -> A. here we will have the problem of cyclic dependencies, If we know the path of each route, we can check if the current node already exists in the previous route. If it does, we continue. In this way, we will properly handle the cyclic problem. so we can just modify some pieces in the code, it will be like this : what do you think 🤔

import importlib.metadata as importlib_metadata
import json
import sys

from pip._internal.metadata import pkg_resources


def frozen_req_from_dist(dist):
    try:
        from pip._internal.operations.freeze import FrozenRequirement
    except ImportError:
        from pip import FrozenRequirement
    try:
        from pip._internal import metadata

        dist = metadata.pkg_resources.Distribution(dist)
        try:
            fr = FrozenRequirement.from_dist(dist)
        except TypeError:
            fr = FrozenRequirement.from_dist(dist, [])
        return str(fr).strip()
    except ImportError:
        pass


def get_installed_distributions():
    dists = pkg_resources.Environment.from_paths(None).iter_installed_distributions(
        local_only=False,
        skip=(),
        user_only=False,
    )
    return [d._dist for d in dists]


def find_deps(idx, path, reqs, traverse_count):
    freqs = []
    for r in reqs:
        d = idx.get(r.key)
        if not d:
            continue
        r.project_name = d.project_name if d is not None else r.project_name
        if r.key in path:
            print(f"Cycle detected: {' -> '.join(current_path)}")
            continue
        current_path = path + [r.key]
        specs = sorted(r.specs, reverse=True)
        specs_str = ",".join(["".join(sp) for sp in specs]) if specs else ""
        dreqs = d.requires()
        freqs.append(
            {
                "name": r.project_name,
                "version": importlib_metadata.version(r.key),
                "versionSpecifiers": specs_str,
                "dependencies": find_deps(idx, current_path, dreqs, traverse_count + 1) if dreqs and traverse_count < 200 else [],
            }
        )
    return freqs


def main(argv):
    out_file = "piptree.json" if len(argv) < 2 else argv[-1]
    tree = []
    pkgs = get_installed_distributions()
    idx = {p.key: p for p in pkgs}
    traverse_count = 0
    for p in pkgs:
        fr = frozen_req_from_dist(p)
        if not fr.startswith('# Editable'):
            tmpA = fr.split("==")
        else:
            fr = p.key
            tmpA = [fr,p.version]
        name = tmpA[0]
        if name.startswith("-e"):
            name = name.split("#egg=")[-1].split(" ")[0].split("&")[0]
        version = "latest"
        if len(tmpA) == 2:
            version = tmpA[1]
        tree.append(
            {
                "name": name.split(" ")[0],
                "version": version,
                "dependencies": find_deps(idx, [p.key], p.requires(), traverse_count + 1),
            }
        )
    all_deps = {}
    for t in tree:
        for d in t["dependencies"]:
            all_deps[d["name"]] = True
    trimmed_tree = [
        t for t in tree if t["name"] not in all_deps
    ]
    with open(out_file, mode="w", encoding="utf-8") as fp:
        json.dump(trimmed_tree, fp)


if __name__ == "__main__":
    main(sys.argv)

[prabhu]

@MohammedAziz02 this is super cool research! Is it possible to make this change to piptree.js and share a pull request? You can add the oracle repo to the repotests too.

[prabhu]

Related bug: AppThreat/atom#145

[prabhu]

@MohammedAziz02 I just tried this patch and it worked like a breeze! Could you kindly send a PR by updating both piptree.py and piptree.js? Alternatively, let me know how my branch looks. Bonus points if you can also fix the atom bug.

[prabhu]

Please test with the master branch.