[IDEA] feat: support PEP-770 - bundled (phantom) dependencies

[jkowalleck]

based on https://sethmlarson.dev/early-promising-results-with-sboms-and-python-packages
contact @sethmlarson

PEP: https://peps.python.org/pep-0770/
PEP discussion: https://discuss.python.org/t/pep-770-improving-measurability-of-python-packages-with-software-bill-of-materials/76308

---

goal

gather the declaration of bundled dependencies of a package, by reading its shipped SBOMs.

Warning

the PEP 770 is stil a draft, so it is unclear how declared shipped SBOMs may be detected ...

expected outcome:

• bundled dependencies are listed as sub-components of their component in the SBOM result
• each declared bundled dependency is present in the dependency graph-
• dependency graph, if present, of such bundled dependencies is carried over into the SBOM result

followup

after implementing this, update the benchmark call andresults in https://github.com/psf/sboms-for-python-packages/tree/main/benchmark

---

example result

JSON based on a demo-SBOM for Pillow==11.1.0 https://gist.github.com/sethmlarson/9b87245c99147815e8e18901f4a10444

example JSON

{
    "$schema": "http://cyclonedx.org/schema/bom-1.6.schema.json",
    "bomFormat": "CycloneDX",
    "specVersion": "1.4",
    "metadata": {
        "component": {
            "type": "application",
            "name": "my-app",
            "version": "0.13.37",
            "bom-ref": "my-app"
        }
    },
    "components": [
        {
            "type": "library",
            "bom-ref": "pillow==11.1.0",
            "name": "Pillow",
            "version": "11.1.0",
            "components": [
                {
                    "type": "library",
                    "bom-ref": "pillow==11.1.0|pkg:rpm/almalinux/libXau@1.0.9-3.el8?distro=almalinux-8",
                    "name": "libXau",
                    "version": "1.0.9-3.el8",
                    "purl": "pkg:rpm/almalinux/libXau@1.0.9-3.el8?distro=almalinux-8"
                },
                {
                    "type": "library",
                    "bom-ref": "pillow==11.1.0|pkg:rpm/almalinux/jbigkit-libs@2.1-14.el8?distro=almalinux-8",
                    "name": "jbigkit-libs",
                    "version": "2.1-14.el8",
                    "purl": "pkg:rpm/almalinux/jbigkit-libs@2.1-14.el8?distro=almalinux-8"
                },
                {
                    "type": "library",
                    "bom-ref": "pillow==11.1.0|pkg:rpm/almalinux/libtiff@4.0.9-33.el8_10?distro=almalinux-8",
                    "name": "libtiff",
                    "version": "4.0.9-33.el8_10",
                    "purl": "pkg:rpm/almalinux/libtiff@4.0.9-33.el8_10?distro=almalinux-8"
                },
                {
                    "type": "library",
                    "bom-ref": "pillow==11.1.0|pkg:rpm/almalinux/libxcb@1.13.1-1.el8?distro=almalinux-8",
                    "name": "libxcb",
                    "version": "1.13.1-1.el8",
                    "purl": "pkg:rpm/almalinux/libxcb@1.13.1-1.el8?distro=almalinux-8"
                },
                {
                    "type": "library",
                    "bom-ref": "pillow==11.1.0|pkg:rpm/almalinux/openjpeg2@2.4.0-5.el8?distro=almalinux-8",
                    "name": "openjpeg2",
                    "version": "2.4.0-5.el8",
                    "purl": "pkg:rpm/almalinux/openjpeg2@2.4.0-5.el8?distro=almalinux-8"
                },
                {
                    "type": "library",
                    "bom-ref": "pillow==11.1.0|pkg:rpm/almalinux/libjpeg-turbo@1.5.3-12.el8?distro=almalinux-8",
                    "name": "libjpeg-turbo",
                    "version": "1.5.3-12.el8",
                    "purl": "pkg:rpm/almalinux/libjpeg-turbo@1.5.3-12.el8?distro=almalinux-8"
                },
                {
                    "type": "library",
                    "bom-ref": "pillow==11.1.0|pkg:rpm/almalinux/lcms2@2.9-2.el8?distro=almalinux-8",
                    "name": "lcms2",
                    "version": "2.9-2.el8",
                    "purl": "pkg:rpm/almalinux/lcms2@2.9-2.el8?distro=almalinux-8"
                },
                {
                    "type": "library",
                    "bom-ref": "pillow==11.1.0|pkg:rpm/almalinux/bzip2-libs@1.0.6-26.el8?distro=almalinux-8",
                    "name": "bzip2-libs",
                    "version": "1.0.6-26.el8",
                    "purl": "pkg:rpm/almalinux/bzip2-libs@1.0.6-26.el8?distro=almalinux-8"
                },
                {
                    "type": "library",
                    "bom-ref": "pillow==11.1.0|pkg:rpm/almalinux/libpng@1.6.34-5.el8?distro=almalinux-8",
                    "name": "libpng",
                    "version": "1.6.34-5.el8",
                    "purl": "pkg:rpm/almalinux/libpng@1.6.34-5.el8?distro=almalinux-8"
                },
                {
                    "type": "library",
                    "bom-ref": "pillow==11.1.0|pkg:rpm/almalinux/freetype@2.9.1-9.el8?distro=almalinux-8",
                    "name": "freetype",
                    "version": "2.9.1-9.el8",
                    "purl": "pkg:rpm/almalinux/freetype@2.9.1-9.el8?distro=almalinux-8"
                },
                {
                    "type": "library",
                    "bom-ref": "pillow==11.1.0|pkg:rpm/almalinux/libwebp@1.0.0-9.el8_9.1?distro=almalinux-8",
                    "name": "libwebp",
                    "version": "1.0.0-9.el8_9.1",
                    "purl": "pkg:rpm/almalinux/libwebp@1.0.0-9.el8_9.1?distro=almalinux-8"
                },
                {
                    "type": "library",
                    "bom-ref": "pillow==11.1.0|pkg:rpm/almalinux/libwebp@1.0.0-9.el8_9.1?distro=almalinux-8",
                    "name": "libwebp",
                    "version": "1.0.0-9.el8_9.1",
                    "purl": "pkg:rpm/almalinux/libwebp@1.0.0-9.el8_9.1?distro=almalinux-8"
                },
                {
                    "type": "library",
                    "bom-ref": "pillow==11.1.0|pkg:rpm/almalinux/libwebp@1.0.0-9.el8_9.1?distro=almalinux-8",
                    "name": "libwebp",
                    "version": "1.0.0-9.el8_9.1",
                    "purl": "pkg:rpm/almalinux/libwebp@1.0.0-9.el8_9.1?distro=almalinux-8"
                }
            ],
        }
    ],
    "dependencies": [
        {
            "ref": "my-app",
            "dependsOn": [
                "pillow==11.1.0"
            ]
        },
        {
            "ref": "pillow==11.1.0",
            "dependsOn": [
                "pillow==11.1.0|pkg:rpm/almalinux/libXau@1.0.9-3.el8?distro=almalinux-8",
                "pillow==11.1.0|pkg:rpm/almalinux/jbigkit-libs@2.1-14.el8?distro=almalinux-8",
                "pillow==11.1.0|pkg:rpm/almalinux/libtiff@4.0.9-33.el8_10?distro=almalinux-8",
                "pillow==11.1.0|pkg:rpm/almalinux/libxcb@1.13.1-1.el8?distro=almalinux-8",
                "pillow==11.1.0|pkg:rpm/almalinux/openjpeg2@2.4.0-5.el8?distro=almalinux-8",
                "pillow==11.1.0|pkg:rpm/almalinux/libjpeg-turbo@1.5.3-12.el8?distro=almalinux-8",
                "pillow==11.1.0|pkg:rpm/almalinux/lcms2@2.9-2.el8?distro=almalinux-8",
                "pillow==11.1.0|pkg:rpm/almalinux/bzip2-libs@1.0.6-26.el8?distro=almalinux-8",
                "pillow==11.1.0|pkg:rpm/almalinux/libpng@1.6.34-5.el8?distro=almalinux-8",
                "pillow==11.1.0|pkg:rpm/almalinux/freetype@2.9.1-9.el8?distro=almalinux-8",
                "pillow==11.1.0|pkg:rpm/almalinux/libwebp@1.0.0-9.el8_9.1?distro=almalinux-8",
                "pillow==11.1.0|pkg:rpm/almalinux/libwebp@1.0.0-9.el8_9.1?distro=almalinux-8",
                "pillow==11.1.0|pkg:rpm/almalinux/libwebp@1.0.0-9.el8_9.1?distro=almalinux-8"
            ]
        },
        {
            "ref": "pillow==11.1.0|pkg:rpm/almalinux/libXau@1.0.9-3.el8?distro=almalinux-8"
        },
        {
            "ref": "pillow==11.1.0|pkg:rpm/almalinux/jbigkit-libs@2.1-14.el8?distro=almalinux-8"
        },
        {
            "ref": "pillow==11.1.0|pkg:rpm/almalinux/libtiff@4.0.9-33.el8_10?distro=almalinux-8"
        },
        {
            "ref": "pillow==11.1.0|pkg:rpm/almalinux/libxcb@1.13.1-1.el8?distro=almalinux-8"
        },
        {
            "ref": "pillow==11.1.0|pkg:rpm/almalinux/openjpeg2@2.4.0-5.el8?distro=almalinux-8"
        },
        {
            "ref": "pillow==11.1.0|pkg:rpm/almalinux/libjpeg-turbo@1.5.3-12.el8?distro=almalinux-8"
        },
        {
            "ref": "pillow==11.1.0|pkg:rpm/almalinux/lcms2@2.9-2.el8?distro=almalinux-8"
        },
        {
            "ref": "pillow==11.1.0|pkg:rpm/almalinux/bzip2-libs@1.0.6-26.el8?distro=almalinux-8"
        },
        {
            "ref": "pillow==11.1.0|pkg:rpm/almalinux/libpng@1.6.34-5.el8?distro=almalinux-8"
        },
        {
            "ref": "pillow==11.1.0|pkg:rpm/almalinux/freetype@2.9.1-9.el8?distro=almalinux-8"
        },
        {
            "ref": "pillow==11.1.0|pkg:rpm/almalinux/libwebp@1.0.0-9.el8_9.1?distro=almalinux-8"
        },
        {
            "ref": "pillow==11.1.0|pkg:rpm/almalinux/libwebp@1.0.0-9.el8_9.1?distro=almalinux-8"
        },
        {
            "ref": "pillow==11.1.0|pkg:rpm/almalinux/libwebp@1.0.0-9.el8_9.1?distro=almalinux-8"
        }
    ]
}

[jkowalleck]

per PEP770, pyrthon packages could ship their own SBOM - in any format.
so either we woudl need to "understand" any format, and wuld transform it into CDX, o the component can be enhanced with the info given by its shipped SBOM,. ...

— OR —

we could see whether to put that component's SBOM as a "attachment"
what we could do today:
add an external reference of type "bom" to that component, and the url woud not be a http-url, but a data url.

the mime-type is disputable. maybe add some logic for proper detection?

e.g. compressed + base64ed the SBOM from the original ticket above: data:aapplication/zip;base64,eJztWF1v2jAUfedXVGiPjUOAsoKEhrRpr9vTVGmaKicYapoPKzErbOO/7zomCVACtkmqVC1P4HtzzvHRzcW+rb+tK/i0PyTeAwlwe3TVfuCcjWzbW3t+FJLpCkXx3JZh240Cy0EDJH+iRRKF7WuJAKGvURxgLjA+y4e/3GXRhBHvB4kTCg9A3EH9LBIQjqeYC2qpJV31ooABQMj3ltMQXzMiMDBjPvUwp7mGPCXEQZoSrC3IOoz+LnR0kNNDvY+HGWKbMZntQOTxTfptsxWfy0wg92eeVKLYp26M4/UJNkZ9P3oajx0HOahTtq3vadaJbR1/vETscdGK4s9v4h97nI9iFtjYD7BPw+XKBqg7vJxAEA2tHiL+7acpTXgcjfMc67aMKDNBgpRl7ZpR0JRls2Xsp8oNpT5D3Twnejl7Fy6dP1JuAWQy6SLHcvqGHu8iKThdcOkbrSG6WW4DFKez2aQvSyStkXunY1bRAknB6AMuo7JWFd04t1eeC+8j9G2oEfPeAShKvaPgMXJZRWyzHI4YCReMzLvwFkJ5WDeGHuc4So0jZ9J3WVlws3wGKKHa4svYjaBGblDPcrrmBV2AKdV1QWdU1xrSG2a7FySiUoaWsdcCQamotxwGBp8X2SxX3T+UdeV/tzgqDazuwNDdAknxaJdx6busIbpZbgMUC+cTcRHr9Y0btERRcrngMWoWKmKb5fAsJkTAincQ/rqHhhZnMGrdIiPS91hVbrNMBqgn4rL05etI0fcg36yUBZJix9jhMipnVdHvbr+7/Xbc3lv5VXDLwHahPSVwWp+S0KPk5Mhsf/p2MMiSIMm38Ogc63CIti+s0HV9hvv0LE5LQ4UzsJrHPzXPO2q84Nd6s63/OlfTzaXmo3uNZ9U6D2k19+/XBF9NdzRucBUSmkyQK9mm9iS1ElatyeLljPpTtkp2aTJnqoBYY/5yOZvJHKISc7Xu5Zczat9SK9mkdiN8U6zycN7atP4D7RY7SA==

— OR —

@stevespringett

Page 55 of the SBOM Guide has an example you might be interested in. The example is for a threat model, but changing the external reference to a bom would also be possible.

{
  "bomFormat": "CycloneDX",
  "specVersion": "1.6",
  "serialNumber": "urn:uuid:3e671687-395b-41f5-a30f-a58921a69b79",
  "version": 1,
  "components": [
    {
      "bom-ref": "acme-application",
      "type": "application",
      "name": "Acme Application",
      "version": "1.0.0",
      "externalReferences": [
        {
          "type": "threat-model",
          "url": "urn:cdx:3e671687-395b-41f5-a30f-a58921a69b79/1#acme-threatmodel"
        }
      ]
    },
    {
      "bom-ref": "acme-threatmodel",
      "type": "data",
      "name": "Acme Threat Model",
      "scope": "excluded",
      "data": [
        {
          "type": "other",
          "contents": {
            "attachment": {
              "encoding": "base64",
              "contentType": "application/pdf",
              "content": "VGhyZWF0IG1vZGVsIGdvZXMgaGVyZQ=="
            }
          }
        }
      ]
    }
  ]
}

---

asked in slack for alternatives: https://cyclonedx.slack.com/archives/CVA0G10FN/p1736440976834859