Generate SBOMs is taking a ridiculous amount of time

[MichaelSimons]

Generating SBOMs in the official builds is taking a ridiculously long time IMO. This should be investigated/reported to 1ES.

Illustrative Build (Microsoft internal link)

• Linux_amd64 src-azurelinux-3.0-net8.0-android-graph 4h 51m
  • Build Images 2h 49m
  • Generate SBOMs 1h 56m

Looking at the logs, there are numerous exceptions happening.

[richlander]

It is worth noting that this is a larger problem than it seems. Our CI was down for 24 hours due to an Azure Linux break. It took much of that time to re-build our Azure Linux images with this cost being a major contributor. This cost represents work stoppage for tens of people and has an indirect effect on many more. It is very costly and will eventually represent a security and/or reliability problem when we have to re-build these images quickly in an emergency. It is guaranteed to happen.

[mthalman]

I've analyzed the scan results. The main issue is the number of images and their size. Nothing seems out of the ordinary with the scanning itself other than the errors (which I'm guessing is related to the image size).

For example, in the Linux_amd64 src-azurelinux-3.0-net10.0-crossdeps-graph build leg, there are 25 images being built. The average SBOM scan time per image is 6.8 minutes which seems reasonable considering the sizes. The smallest image is 450 MB which takes 1.5 minutes to scan. The majority of them are about 2 GB. The largest is azurelinux-3.0-net10.0-crossdeps-builder-amd64 which is 13 GB (see #1446).

We should investigate ways to reduce the image sizes. See #1447

[richlander]

Are we using the SBOMs for anything? They may be required by some policy, but ...

[mthalman]

They're required by policy.

[richlander]

We should ask for an exception. This seems like a waste.

[ericstj]

Generate SBOMs should not take that amount of time, something is wrong here. It should be on a similar order to CG which completes in seconds to minutes.

[ericstj]

This looks sus:

│ Linux             │ 1.2e+02 seconds   │ 0                 │ 0                │

https://dev.azure.com/dnceng/internal/_build/results?buildId=2682591&view=logs&j=5451b624-d93a-5e9f-b2ab-9f4f54898441&t=5f5789cc-a604-5657-2dd3-a86dd06a6248&l=317

│ Linux             │ 6e+02 seconds     │ 0                 │ 0                │

https://dev.azure.com/dnceng/internal/_build/results?buildId=2682591&view=logs&j=5451b624-d93a-5e9f-b2ab-9f4f54898441&t=5f5789cc-a604-5657-2dd3-a86dd06a6248&l=144

[mthalman]

Generate SBOMs should not take that amount of time, something is wrong here. It should be on a similar order to CG which completes in seconds to minutes.

This is different than CG. We're using the Docker image scan functionality of SBOM generation which requires running the container and it runs different tools than you would in CG, notably syft.

[mthalman]

We can consider moving the SBOM generation to be post-publish. The only requirement is that SBOMs are added in the pipeline, not when that occurs.