CI/CD status

[prabhu]

Discussion to track our build status, downtime etc.

self-hosted agents

We have two Mac Minis (64GB RAM each) and one Mini PC (32 GB RAM) powering cdxgen.

Todo

• Move to BDC1 (Upcoming Bude data centre) and invest in a physical lock.
• Publish OBOMs for the compute units.
• Harden the OS a bit based on the OBOM and lynis scans.
• Investigate multi-arch builds with docker rootless

Learnings

• Docker rootless and containerd-based builds have low reliability. Running docker as root magically makes things better. This is because most official docker steps, including the build-and-push action, expect sole root access.
• Macs continue to suck in headless mode. After the recent software update, tailscale got disabled, requiring physical access to toggle the switch. Why?
• Docker builds from an ARM64 host machine usually work only on sunny days. I don't think QEMU is that well tested on ARM64. So we need AMD64 machines with working KVM to reliably build for ARM64.
• Multi-arch builds with Docker are possible only due to mere luck. Ubuntu Server, images such as tonistiigi/binfmt, multiarch/qemu-user-static, moby/buildkit:buildx-stable-1 wrap a very old version of QEMU 8.2.2 with several known bugs. Upgrading the entire stack to use QEMU 10 is not for the faint-hearted. Even with QEMU 10, you are met with regression bugs with the only solution being to downgrade to QEMU v8.1.5!

[prabhu]

Thursday 3 Apr BST 23:00 (UTC+1)

Noticed errors with our container builds due to installation problems. Identified issues with the pip install blint command. Created and released blint 2.4.1 to fix. Re-ran the container builds to make the master branch green. Total DevOps time spent: 4 hours. This is, however, non-chargeable since I am also responsible for the bug originating from the blint 2.4.0 release.

[prabhu]

Tuesday 8 Apr BST 11:00 (UTC+1)

Noticed errors with our container builds due to installation problems. Identified issues with the pip install blint command. Have created this upstream ticket to investigate the problem.

[prabhu]

pip install has started working by simply re-running the workflows a few times. If I come across any RCA from the PyPI team, I will link it here.

[prabhu]

Thursday 10 Apr BST 10:20 (UTC+1)

Noticed timeouts in the upload base images workflow with the macos-hosted agent. Suspected the issue to be disk space, so re-ran the workflow after doing docker system prune -a.

[prabhu]

docker system prune did the trick. A couple of failures with sle .NET builds, which is like the one we have seen before.

#5 29.73 To install missing framework, download:
#5 29.73 https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=8.0.15&arch=x64&rid=linux-x64&os=sles.15.6
#5 ERROR: process "/bin/sh -c zypper refresh && zypper --non-interactive update && zypper --non-interactive install -l --no-recommends git-core nodejs20 npm20 python311 python311-pip wget zip unzip make gawk java-21-openjdk-devel     && dotnet --list-sdks     && dotnet workload install android wasm-tools wasm-tools-net6 wasm-tools-net7     && dotnet workload list     && npm install -g corepack     && zypper clean -a" did not complete successfully: exit code: 150

[prabhu]

My attempts to build sle dotnet images without performing zypper refresh and update did not work.

148.8 Running post-transaction scripts [....done]
150.4 8.0.407 [/usr/share/dotnet/sdk]
151.7
156.1 Failed to update the advertising manifest microsoft.net.workload.emscripten.net6: Value cannot be null or an empty string. (Parameter 'value').
156.1 Failed to update the advertising manifest microsoft.net.sdk.maccatalyst: Value cannot be null or an empty string. (Parameter 'value').
156.1 Advertising manifest not updated. Manifest package for microsoft.net.sdk.maccatalyst doesn't exist.
156.1 Advertising manifest not updated. Manifest package for microsoft.net.workload.emscripten.net6 doesn't exist.
156.2 Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.current: Value cannot be null or an empty string. (Parameter 'value').
156.2 Advertising manifest not updated. Manifest package for microsoft.net.workload.mono.toolchain.current doesn't exist.
156.2 Failed to update the advertising manifest microsoft.net.sdk.tvos: Value cannot be null or an empty string. (Parameter 'value').
156.2 Advertising manifest not updated. Manifest package for microsoft.net.sdk.tvos doesn't exist.
156.3 Failed to update the advertising manifest microsoft.net.workload.emscripten.net7: Value cannot be null or an empty string. (Parameter 'value').
156.3 Advertising manifest not updated. Manifest package for microsoft.net.workload.emscripten.net7 doesn't exist.
156.3 Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.net7: Value cannot be null or an empty string. (Parameter 'value').
156.3 Advertising manifest not updated. Manifest package for microsoft.net.workload.mono.toolchain.net7 doesn't exist.
156.4 Failed to update the advertising manifest microsoft.net.sdk.maui: Value cannot be null or an empty string. (Parameter 'value').
156.4 Advertising manifest not updated. Manifest package for microsoft.net.sdk.maui doesn't exist.
156.5 Failed to update the advertising manifest microsoft.net.workload.emscripten.current: Value cannot be null or an empty string. (Parameter 'value').
156.5 Advertising manifest not updated. Manifest package for microsoft.net.workload.emscripten.current doesn't exist.
156.5 Failed to update the advertising manifest microsoft.net.sdk.android: Value cannot be null or an empty string. (Parameter 'value').
156.5 Advertising manifest not updated. Manifest package for microsoft.net.sdk.android doesn't exist.
156.6 Failed to update the advertising manifest microsoft.net.sdk.macos: Value cannot be null or an empty string. (Parameter 'value').
156.6 Advertising manifest not updated. Manifest package for microsoft.net.sdk.macos doesn't exist.
156.6 Failed to update the advertising manifest microsoft.net.sdk.aspire: Value cannot be null or an empty string. (Parameter 'value').
156.6 Advertising manifest not updated. Manifest package for microsoft.net.sdk.aspire doesn't exist.
159.3 Failed to update the advertising manifest microsoft.net.sdk.ios: Unable to get repository signature information for source https://api.nuget.org/v3-index/repository-signatures/5.0.0/index.json..
159.3 Advertising manifest not updated. Manifest package for microsoft.net.sdk.ios doesn't exist.
159.3 Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.net6: Unable to get repository signature information for source https://api.nuget.org/v3-index/repository-signatures/5.0.0/index.json..
159.3 Advertising manifest not updated. Manifest package for microsoft.net.workload.mono.toolchain.net6 doesn't exist.
159.3 Installing pack Microsoft.Android.Sdk.Linux version 34.0.43...
159.5 Workload installation failed. Rolling back installed packs...
159.5 Rolling back pack Microsoft.Android.Sdk.Linux installation...
159.5 Workload installation failed: Value cannot be null or an empty string. (Parameter 'value')

SLE issues usually resolve themselves in a day or two. Let's check again tomorrow without wasting any more DevOps time.

[prabhu]

SLE dotnet 8 and 9 images are still broken. The error message lists two conflicting versions, 8.0.14 and 8.0.15, indicating upstream issues. Raised a query here.

#5 35.12 8.0.408 [/usr/share/dotnet/sdk]
#5 35.12 You must install or update .NET to run this application.
#5 35.12 
#5 35.12 App: /usr/share/dotnet/sdk/8.0.408/dotnet.dll
#5 35.12 Architecture: x64
#5 35.12 Framework: 'Microsoft.NETCore.App', version '8.0.15' (x64)
#5 35.12 .NET location: /usr/share/dotnet/
#5 35.12 
#5 35.12 The following frameworks were found:
#5 35.12   8.0.14 at [/usr/share/dotnet/shared/Microsoft.NETCore.App]
#5 35.12 
#5 35.12 Learn more:
#5 35.12 https://aka.ms/dotnet/app-launch-failed
#5 35.12 
#5 35.12 To install missing framework, download:
#5 35.12 https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=8.0.15&arch=x64&rid=linux-x64&os=sles.15.6
#5 ERROR: process "/bin/sh -c zypper refresh && zypper --non-interactive update && zypper --non-interactive install -l --no-recommends git-core nodejs20 npm20 python311 python311-pip wget zip unzip make gawk java-21-openjdk-devel     && dotnet --list-sdks     && dotnet workload install android wasm-tools wasm-tools-net6 wasm-tools-net7     && dotnet workload list     && npm install -g corepack     && zypper clean -a" did not complete successfully: exit code: 150

[prabhu]

Have removed the dotnet workload install commands with this commit, since it's been days. Recommendation to deprecate the SLE dotnet images in favour of the Debian ones, which have been stable as well as support multiple architectures.

[prabhu]

To vent my frustration with the reliability of SLE Docker builds, below is an error from failed Python 3.11 builds. I have to now wait a while for all other builds to finish before restarting the failed ones.

16.22 Retrieving: bzip2-1.0.8-150400.1.122.aarch64 (SLE_BCI) (1/46),  45.8 KiB    
16.27 Retrieving: bzip2-1.0.8-150400.1.122.aarch64.rpm [....error]
17.50 Download (curl) error for 'https://public-dl.suse.com/SUSE/Products/SLE-BCI/15-SP6/aarch64/product/aarch64/bzip2-1.0.8-150400.1.122.aarch64.rpm':
17.50 Error code: Curl error 35
17.50 Error message: OpenSSL/3.1.4: error:0A000438:SSL routines::tlsv1 alert internal error
17.50 
17.50 Abort, retry, ignore? [a/r/i/...? shows all options] (a): a
17.52 Problem occurred during or after installation or removal of packages:
17.52 Installation has been aborted as directed.
17.52 Please see the above error message for a hint.
------
Dockerfile.python311:21

[prabhu]

Wednesday 16 Apr BST 11:00 (UTC+1)

Power is down, so the self-hosted agents are unavailable.

(Power must be back in six hours. The good news is that we have an EcoFlow 1KWh battery backup. The bad news is that I am travelling in the opposite direction, so I couldn't physically deliver the battery to the co-location centre. Note to self: we need to distribute compute units across locations.)

[prabhu]

Power is back at 4 p.m. Have to revisit my plans to scale entirely with in-house computes.

[prabhu]

Thursday 17 Apr BST 17:00 (UTC+1)

Noticed that the "Upload base images" had errors. Apparently, this is a known issue with colima which requires a workaround after every macOS update.

brew uninstall colima qemu lima
rm -rf ~/.colima
brew install colima
brew services restart colima

[prabhu]

For reminder, we cannot use Docker Desktop for Mac since it requires access to the keychain and doesn't really work well in a headless environment. Things like Rancher Desktop and nerdctl require rework in the workflow to replace Docker-based steps with custom scripts.

[prabhu]

Could have tried this simpler workaround too.

rm -rf ~/.colima/_lima/_networks/user-v2

[malice00]

FYI: keychain can be interacted with from CLI. Not sure what you need, but I use it in CI all the time!

…

On Thu, Apr 17, 2025, 18:41 prabhu ***@***.***> wrote: For reminder, we cannot use Docker Desktop for Mac since it requires access to the keychain and doesn't really work well in a headless environment. Things like Rancher Desktop and nerdctl require rework in the workflow to replace Docker-based steps with custom scripts. — Reply to this email directly, view it on GitHub <#1720 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADC7EIPPXZPRAUMWKWXEBMT2Z7KUZAVCNFSM6AAAAAB2NKT6X2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOBWHE4DQOA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[prabhu]

Interesting. I don't have the exact error with me, but it only worked when run from a screensharing session, since macOS wanted to show a password prompt. Upon restart, nothing docker related will work unless it was unlocked. In contrast, colima works in clear text password mode, but does have other issues.

[malice00]

I believe the correct command would be `keychain unlock`. It needs credentials, but these can be given on the cli.

…

On Thu, Apr 17, 2025, 18:59 prabhu ***@***.***> wrote: Interesting. I don't have the exact error with me, but it only worked when run from a screensharing session, since macOS wanted to show a password prompt. Upon restart, nothing docker related will work unless it was unlocked. In contrast, colima works in clear text password mode, but does have other issues. — Reply to this email directly, view it on GitHub <#1720 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADC7EIJKXQIQR45ZMJ3ELWL2Z7MV3AVCNFSM6AAAAAB2NKT6X2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOBXGAYDKMI> . You are receiving this because you commented.Message ID: ***@***.***>

[prabhu]

Will keep in mind. So far we are ok with colima.

[prabhu]

Saturday 26 Apr BST 17:00 (UTC+1)

swift-actions/setup-swift step has stopped working due to this issue. This step is quite flaky. Need to prioritise #1711 in case this GPG issue is not resolved in a day or two.

[prabhu]

Our workflows worked after multiple restarts. It is frustrating that setup-swift is not a first-party step similar to setup-java, owned and distributed by GitHub. Relying on the free Ubuntu keyserver for verifying the signature of Swift keys can only scale so much. Open-source devs can also do only so much free support for the world.

[prabhu]

Thursday 1 May BST 15:00 (UTC+1)

Happy Labour Day (International Workers' Day)!

Today a brand new server asin joined our infra. It is currently at a highly-secure and undisclosed location (under my desk 10 minutes from GCHQ in Bude).

I would love to share an HBOM and OBOM. Sadly, there are no HBOM tools I could find. (cc: @malice00). So, just some output from a couple of commands and a decent OBOM.

Output from lscpu

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          52 bits physical, 57 bits virtual
  Byte Order:             Little Endian
CPU(s):                   48
  On-line CPU(s) list:    0-47
Vendor ID:                AuthenticAMD
  Model name:             AMD Ryzen Threadripper 7960X 24-Cores
    CPU family:           25
    Model:                24
    Thread(s) per core:   2
    Core(s) per socket:   24
    Socket(s):            1
    Stepping:             1
    CPU(s) scaling MHz:   12%
    CPU max MHz:          5665.0000
    CPU min MHz:          545.0000
    BogoMIPS:             8386.78
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulq
                          dq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwait
                          x cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveop
                          t xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc amd_ibpb_ret arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassis
                          ts pausefilter pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap
Virtualization features:
  Virtualization:         AMD-V
Caches (sum of all):
  L1d:                    768 KiB (24 instances)
  L1i:                    768 KiB (24 instances)
  L2:                     24 MiB (24 instances)
  L3:                     128 MiB (4 instances)
NUMA:
  NUMA node(s):           1
  NUMA node0 CPU(s):      0-47
Vulnerabilities:
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Mitigation; Safe RET
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                  Not affected
  Tsx async abort:        Not affected

lshw.txt

OBOM

asin-obom.cdx.json

Bugs identified

• [obom] track third-party PPA in ubuntu/debian #1767

[prabhu]

Thursday 8 May BST 00:00 (UTC+1)

Self-hosted agents ran out of disk space. It turned out, due to some known bugs in cdxgen, the /tmp folder was full of docker-images-*, atom-usages-*, and atom-reachables-* directories. Manually cleared the /tmp folder for now. Don't know which one is easier - fixing cdxgen to properly clear the temp directories or simply periodically checking the self-hosted agents.

[malice00]

Idea: setup a cron to clean temp dir every so often

[prabhu]

Added this line in the repotests workflow. BTW, I had an old Intel MacBook Pro with 32GB RAM lying around. I removed macOS and installed Ubuntu Server on this. This server is called simran. It ran the repotests successfully last night.

Meanwhile, I upgraded the asin server to 256GB RAM. Initially, one 64GB is reserved for cdxgen, while the rest can be used for any experiments, testing, etc. Thank you PC specialist for the generous discount on the upgrade RAM pricing!

[prabhu]

Ok, looks like it wasn't that successful. Taking a look.

[prabhu]

Adding || true to the various pip install commands did the trick. I think this hybrid setup of self-hosted and github-hosted will make our workflows and tests more generic and portable. Something other projects could try and copy to reduce our reliance of certain large enterprises.

[prabhu]

Thursday 8 May BST 13:00 (UTC+1)

Despite the new self-hosted machines, our CI usage is going up well beyond the 50K minutes that GitHub offers for Enterprise users. We are now beginning to consume minutes from other OWASP orgs as well (CycloneDX is a sub-org under OWASP).

Thanks to the excellent work from @malice00, we have a new matrix-based workflow for building base images. However, building Ruby 3.4.3 steps still takes too much time, so we are at risk of exceeding 50K minutes this month as well.

I think the next step is to look into multi-stage builds and layer caching, so that $HOME/.rbenv gets mounted and copied over from a cache for various architectures. Other ideas to reduce usage are welcome.

[prabhu]

Wednesday 14 May BST 12:00 (UTC+1)

New Raspberry Pi 5 build agent. If it works well for our needs, we can set up a mini-cluster for testing and get it to a stage where users could run cdx at industrial scale and reduce cloud costs.

OBOM

pi5-obom.cdx.json

[prabhu]

Thursday 15 May BST 12:00 (UTC+1)

Currently, fighting the CI due to multiple failures (all of them with github-hosted).

Installing dotnet on debian isn't reliable

#10 [linux/arm64 2/5] COPY ci/base-images/debian/install.sh /tmp/
#10 ERROR: failed to copy: httpReadSeeker: failed open: unexpected status code https://mcr.microsoft.com/v2/dotnet/sdk/blobs/sha256:1df4174e1641c24bac819a993f5ed3d2ae6facbada6f7a943bed15d22e64851b: 503 Service Unavailable

Node 20 builds continue to fail for mysterious reasons

102.7 update-alternatives: using /usr/bin/g++-13 to provide /usr/bin/g++ (g++) in auto mode
112.6 npm error code EEXIST
112.6 npm error path /usr/lib64/node_modules/npm20/bin/npm
112.6 npm error EEXIST: file already exists
112.6 npm error File exists: /usr/lib64/node_modules/npm20/bin/npm
112.6 npm error Remove the existing file and try again, or run npm
112.6 npm error with --force to overwrite files recklessly.
112.6 npm error A complete log of this run can be found in: /root/.npm/_logs/2025-05-15T11_00_43_489Z-debug-0.log

I am also seeing random qemu crashes.

corepack command not found errors

A number of java builds are failing, despite us installing corepack explicitly!

0.127 /bin/sh: corepack: command not found
------
Dockerfile.java17-slim:20
--------------------
  19 |     
  20 | >>> RUN cd /opt/cdxgen && corepack enable && corepack pnpm install --config.strict-dep-builds=true --prod --package-import-method copy --frozen-lockfile && corepack pnpm cache delete \
  21 | >>>     && mkdir -p /opt/cdxgen-node-cache \
  22 | >>>     && node /opt/cdxgen/bin/cdxgen.js --help \
  23 | >>>     && rm -rf /root/.cache/node \
  24 | >>>     && chmod a-w -R /opt
  25 |     WORKDIR /app

The rolling image is looking for corepack-default. WHY?

> [linux/arm64 3/4] RUN cd /opt/cdxgen && corepack enable && corepack pnpm install --config.strict-dep-builds=true --prod --package-import-method copy --frozen-lockfile && corepack pnpm cache delete     && mkdir -p /opt/cdxgen-node-cache     && node /opt/cdxgen/bin/cdxgen.js --help     && pip install --upgrade --no-cache-dir blint atom-tools --target /opt/pypi     && rm -rf /root/.cache/node     && chmod a-w -R /opt:
0.183 /usr/bin/corepack-default: No such file or directory

opensuse continues to do opensuse things

ERROR: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: registry.opensuse.org/opensuse/bci/python:3.10: failed to resolve source metadata for registry.opensuse.org/opensuse/bci/python:3.10: failed to do request: Head "https://registry.opensuse.org/v2/opensuse/bci/python/manifests/3.10": dial tcp 195.135.223.221:443: i/o timeout

ERROR: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: registry.opensuse.org/opensuse/bci/python:3.9: failed to resolve source metadata for registry.opensuse.org/opensuse/bci/python:3.9: failed to do request: Head "https://registry.opensuse.org/v2/opensuse/bci/python/manifests/3.9": dial tcp 195.135.223.221:443: i/o timeout

[prabhu]

Like most things with software and IT, a mere restart is making all these builds pass again. Guess we will never know the reason. Probably there is a way to make a job retry once if the previous job failed.

[malice00]

The node image was failing because we were trying to install npm although it was already there. Fixed in #1801

[prabhu]

Very good catch!

[prabhu]

Wednesday 21 May BST 10:00 (UTC+1)

Multiple build failures since macos-hosted was out of disk space. Retrying the builds after running the command docker system prune -a

Noticed another issue with attaching sboms to the nightly images.

node bin/cdxgen.js -t docker -o sbom-oci-base-image.cdx.json ghcr.io/cyclonedx/debian-dotnet8:nightly
node bin/verify.js -i sbom-oci-base-image.cdx.json --public-key contrib/bom-signer/public.key

Retrying with sha256:6dd7e99d793d70ff2ce73260d0cdd66ed32cbaebf468163457cfdfcbe5da1d89
Manifest file /home/runner/work/_temp/cdxgen-sboms/docker-images-Ees8wm/manifest.json was not found after export at /home/runner/work/_temp/cdxgen-sboms/docker-images-Ees8wm
OCI BOM generation has failed due to problems with exporting the image ghcr.io/cyclonedx/debian-dotnet8:nightly.
sbom-oci-base-image.cdx.json is invalid!

Possible bug in cdxgen?

[prabhu]

@malice00 I think we need docker run --rm --privileged multiarch/qemu-user-static --reset -p yes -c yes and the docker/setup-buildx-action@v3 for macos-hosted as well. Also I think the qemu workaround step needs to be above the buildx step to make the buildx agents use the correct qemu static binaries. Finally we need load: true in the docker/build-push-action@v5 for cdxgen sbom generation to work correctly.

[malice00]

load doesn't work on multi-platform builds, but I've been working on a solution in a branch. Hope to finish it this weekend.

[malice00]

Also, about the disk space: A lot of that is probably our build caches. We should remove those periodically -- I'll see if I can add that to the branch as well!

[prabhu]

Please rebase from master since have merged a change to split the ruby build per architecture.

[prabhu]

colima on prabhus-mini seems to be running with just 100GB disk. The template, however, shows 500GB indicating this VM probably got created with default settings before the template was set correctly.

prabhu@colima:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        96G   85G   12G  89% /
tmpfs            20G     0   20G   0% /dev/shm
tmpfs           7.9G  972K  7.9G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
efivarfs         56K  2.6K   54K   5% /sys/firmware/efi/efivars
vz-rosetta      927G  589G  339G  64% /mnt/lima-rosetta
/dev/vda16      891M   32M  797M   4% /boot
/dev/vda15       98M  6.4M   92M   7% /boot/efi
tmpfs           4.0G  4.0K  4.0G   1% /run/user/501
/dev/vdb         47M   47M     0 100% /mnt/lima-cidata
:/Users/prabhu  927G  589G  339G  64% /Users/prabhu
:/tmp/colima    927G  589G  339G  64% /tmp/colima

[prabhu]

Thursday 22 May BST 10:00 (UTC+1)

load: true doesn't support multi-arch builds in build-push-action. The error was ERROR: Docker exporter does not currently support exporting manifest lists. When a built image is not loaded, attempting to generate an SBOM is resulting in cdxgen downloading the image again from the registry!

Related: #1606

[prabhu]

Friday 23 May BST 09:00 (UTC+1)

Happy Friday! mini-dev-1 (that gray Beelink ESR8 one since we don't have the HBOM!) required some servicing today. When I set up this machine a while ago, my thinking was to set enough UMA Frame Buffer size so that we could invoke cdx1 and other mini ML models with llama.cpp as part of CI pipelines. For some reason, I went with 16GB (out of 32GB RAM). This not only reduced the amount of memory available for CI, but also reduced the system performance a bit as per AMD support document.

Changing this setting to auto required physically rebooting to BIOS, etc. With auto, Ubuntu Server can now see and access all 32GB RAM! However, as you can imagine, anything with the word auto never really works on Linux. So, llama.cpp and others can no longer access the GPU memory!

The plan is to run this as a CPU-only build agent for a bit more, till we reach a point where we need some GPU.

[prabhu]

Note to self: can a HBOM tool capture the BIOS settings?