Unable to build the 1.33 FIPS Snap Package

[AdeepKrishnaKeelar]

Summary

Was attempting to build the Microk8s Snap Package using the 1.33 branch and after adding the FIPS commit on it. We are using the Go v1.24 to build the Snap Package. During the snap build process, particularly at helm, the build breaks, possibly guessing there isn't a FIPS commit or addition for this version.
I tried to manually add it but yeah, that doesn't work 😅

What Should Happen Instead?

The stage should complete, and proceed forward.

Reproduction Steps

• Clone Microk8s Repo.
• Checkout 1.33 branch.
• Apply FIPS commit -- git cherry-pick $(git log -n 1 remotes/origin/fips --pretty=format:"%H")
• Resolve the merge conflict manually in snap/snapcraft.yaml
• Change the variables -- KUBE_VERSION=v1.33.3 in build-scripts/components/kubernetes/version.sh
• Change go version to 1.24/stable in snap/snapcraft.yaml in build-deps
• Change FIPS Env variables (Uncomment OpenSSL and LD library and Go_FIPS=1) in microk8s-resources/default-args/
• Ensure CGO=1 in the build for all components, and comment export=openssl as Go 1.24 has native FIPS support.
• sudo SNAPCRAFT_BUILD_ENVIRONMENT=host snapcraft

The same build-package workflow provided in microk8s works for the same instead of manual building.

Logs

Here's the sample output from the build workflow, during the snap package build.

+ git clone https://github.com/helm/helm --depth 1 -b v3.17.2 /root/parts/helm/build/helm
Cloning into '/root/parts/helm/build/helm'...
Note: switching to 'cc0bbbd6d6276b83880042c1ecb34087e84d41eb'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
  git switch -c <new-branch-name>
Or undo this operation with:
  git switch -
Turn off this advice by setting config variable advice.detachedHead to false
+ cd /root/parts/helm/build/helm
+ git config user.name 'MicroK8s builder bot'
+ git config user.email microk8s-builder-bot@canonical.com
+ '[' -e /root/project/build-scripts/components/helm/pre-patch.sh ']'
++ python3 /root/project/build-scripts/print-patches-for.py helm v3.17.2
+ for patch in $(python3 "${DIR}/print-patches-for.py" "${COMPONENT_NAME}" "${GIT_TAG}")
+ git am /root/project/build-scripts/components/helm/patches/v3.17.2/0002-enable-cgo.patch
Applying: enable cgo
+ for patch in $(python3 "${DIR}/print-patches-for.py" "${COMPONENT_NAME}" "${GIT_TAG}")
+ git am /root/project/build-scripts/components/helm/patches/v3.17.2/0001-disable-warnings-for-kubeconfig-permissions.patch
error: patch failed: cmd/helm/root.go:206
Applying: disable warnings for kubeconfig permissions
error: cmd/helm/root.go: patch does not apply
hint: Use 'git am --show-current-patch' to see the failed patch
Patch failed at 0001 disable warnings for kubeconfig permissions
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Failed to run 'override-build': Exit code was 128.
Run the same command again with --debug to shell into the environment if you wish to introspect this failure.
Error: Process completed with exit code 2.

Can you suggest a fix?

Think the fix is adding a patch and port it to the fips branch ?
https://github.com/canonical/microk8s/tree/fips/build-scripts/components/helm/patches (and any other necessary component)

Are you interested in contributing with a fix?

Tried to manually add it, but that doesn't work. Sorry.

[Hamedmt]

I have a similar problem too. I have updated all related components/addons versions to be compatible, but the patches may not be correct. Moreover, it seems that the env "GOEXPERIMENT=opensslcrypto" is not a valid option in Go 1.21+. I tried "GOFIPS=1", but it needs FIPS-enabled kernel to access the backend openssl libraries.

[AdeepKrishnaKeelar]

Hey @Hamedmt, to build the 1.32 Snap Package, I used Go 1.24 which has native FIPS support, had no issues.
Like mentioned, I feel it may be the missing patch not ported for the FIPS branch for version 3.17.3 for helm (https://github.com/canonical/microk8s/tree/fips/build-scripts/components/helm/patches)
You might find this previous issue of mine helpful -- #5046
I did build the 1.33 snap package with a downgraded helm version.

[Hamedmt]

Hi Adeep,
Thanks for your message. I tried that approach you mentioned too. I use Go1.24-fips for the build and removed the "GOEXPERIMENT" env, and just enabled it via GOFIPS=1. Although I have a FIPS-enabled OS, the cluster-agent is not healthy and complaining about crypto backend.

 panic: FIPS mode requested (environment variable GOFIPS) but no supported crypto backend is enabled

[AdeepKrishnaKeelar]

Hi @Hamedmt oh yes..
If you would try another attempt, why not try passing the CGO variable set to 1 in all the necessary components ?
Now in the build-components, where the FIPS environment is present right, pass this like so -- export CGO_ENABLED=1
Eg snippet for cluster-agent

#!/bin/bash

export INSTALL="${1}/bin"
mkdir -p "${INSTALL}"

# export GOEXPERIMENT=opensslcrypto
export CGO_ENABLED=1
go build -ldflags '-s -w' -o cluster-agent ./main.go

cp cluster-agent "${INSTALL}"

think this might help.. I faced something similar for helm

microk8s helm
panic: FIPS mode requested (environment variable GOFIPS=1) but no supported crypto backend is enabled

after attempting the above, 'twas working pretty fine..

microk8s helm
The Kubernetes package manager

Common actions for Helm:

- helm search:    search for charts
- helm pull:      download a chart to your local directory to view
- helm install:   upload the chart to Kubernetes
- helm list:      list releases of charts

Environment variables:

Hope this might help.. :)

[Hamedmt]

Thanks a lot for your comment.
I have added export CGO_ENABLED=1 in the build.sh of the components which are built by go, including the cluster-agent, but I still get the panic error.

[AdeepKrishnaKeelar]

Hi, requesting for help on this please..?
Not a change I can port from my side, or if steps can be shared, I could attempt to raise a PR ?