Update dependencies

[stefanprodan]

Updates:

• helm.sh/helm/v3 v3.18.0
• sigs.k8s.io/controller-runtime v0.21.0
• cloud.google.com/go/storage v1.54.0
• github.com/sigstore/cosign/v2 v2.5.0

Part of: fluxcd/flux2#5363

[stefanprodan]

Lots of issues with Helm 3.18 due to the upgrade of ORAS from v1 to v2. Login to container registries is broken:

failed to construct Helm client: failed to decode config file at /tmp/credentials3039474744: invalid config format: EOF

We'll postpone to upgrade to Helm 3.18 for Flux 2.7

[TerryHowe]

Does the error indicate that since the login failed, there is nothing in the creds file (or it wasn't created) causing a EOF.

[matheuscscp]

Reopening just to test some fixes in CI

[TerryHowe]

That change related to the floats may not be required because there was a change similar to that that was reverted for 3.18.1

[TerryHowe]

Ideally, you wouldn't need to make these changes with 3.18.1, but I don't yet understand all the problems

[TerryHowe]

I'd be concerned that 3.18.1 will be broken with this change.

[matheuscscp]

Tests are passing now, but I still prefer not to ship 3.18.0

[matheuscscp]

I'll keep this branch around though, it's documenting a few things we need to do:

• Remove old assertion library from tests
• Remove a flaky .Update() in the ObjectLevelWorkloadIdentity feature gate test outside an Eventually block
• Remove the oras v1 workaround passing an empty tmp file for not storing docker creds at ~/.docker/config.json (the big culprit of tests not passing)

[TerryHowe]

I'd still like to understand the file issue

[matheuscscp]

I'd still like to understand the file issue

It's actually simple: oras v1 was writing to ~/.docker/config.json. In order to avoid that in a Flux controller, which cannot keep such state across reconciliations of different objects, we were creating an empty tmp file and passing its name down to oras v1 so it could have somewhere else to write that was not ~/.docker/config.json. Then we delete the file at the end of the reconciliation of the object.

The first thing that oras v2 does with this empty file is try to parse a JSON from it, which obviously fails with EOF as the file has 0 bytes of content.

The fix for our CI to pass here is just not creating and passing down this empty file to helm/oras anymore.

We were never reusing this file anywhere else in the reconciliation code except for deleting it at the end, so I think in our case just removing the code that creates it, passes it down to helm/oras and deletes it at the end is enough.

[stefanprodan]

We were never reusing this file anywhere else in the reconciliation code except for deleting it at the end, so I think in our case just removing the code that creates it, passes it down to helm/oras and deletes it at the end is enough.

I hope ORAS v2 can take the auth data directly without having to write files on disk. If that's not the case, then we should look into replacing ORAS with fluxcd/pkg/oci and fluxcd/pkg/auth.

[matheuscscp]

We were never reusing this file anywhere else in the reconciliation code except for deleting it at the end, so I think in our case just removing the code that creates it, passes it down to helm/oras and deletes it at the end is enough.

I hope ORAS v2 can take the auth data directly without having to write files on disk. If that's not the case, then we should look into replacing ORAS with fluxcd/pkg/oci and fluxcd/pkg/auth.

I ran this test step by step on the debugger to investigate:

source-controller/internal/controller/helmchart_controller_test.go

Lines 2412 to 2434 in 98b2edc

	{
	name: "HTTP with basic auth secret",
	want: sreconcile.ResultSuccess,
	insecure: true,
	registryOpts: registryOptions{
	withBasicAuth: true,
	},
	secretOpts: secretOptions{
	username: testRegistryUsername,
	password: testRegistryPassword,
	},
	secret: &corev1.Secret{
	ObjectMeta: metav1.ObjectMeta{
	Name: "auth-secretref",
	},
	Type: corev1.SecretTypeDockerConfigJson,
	Data: map[string][]byte{},
	},
	assertConditions: []metav1.Condition{
	*conditions.TrueCondition(meta.ReconcilingCondition, meta.ProgressingReason, "building artifact: pulled 'helmchart' chart with version '0.1.0'"),
	*conditions.UnknownCondition(meta.ReadyCondition, meta.ProgressingReason, "building artifact: pulled 'helmchart' chart with version '0.1.0'"),
	},
	},

Looks like helm and oras v2 are keeping the username/password entirely in memory with this:

https://github.com/helm/helm/blob/cc58e3f5a3aa615c6a86275e6f4444b5fdd3cc4e/pkg/registry/client.go#L269

https://github.com/oras-project/oras-go/blob/05a2b09cbf2eab1df691411884dc4df741ec56ab/registry/remote/auth/client.go#L68-L82

So I don't think oras v2 is writing this to disk, @TerryHowe can probably confirm?

Maybe we should use fluxcd/pkg/oci to have the same OCI code everywhere and not three different codes like right now, but it doesn't look like writing the creds to disk is going to be an issue. Btw this helm OCI code already uses fluxcd/pkg/auth

[scottrigby]

PS the empty config file issue for oras v2 is now fixed in Helm 3 helm/helm#30932 it will be in the next patch release 3.18.2 which we will be releasing very soon.

Although I agree flux should no longer need to create a temp creds file at all anymore.

[matheuscscp]

TLDR: We have a workaround in the current version passing an empty tmp file per reconciliation for Helm/ORAS v1 to write creds to. In Helm 3.18 the file is no longer written to, only read from. Since it's empty, we get an error. Helm fixed this in a 3.18.x patch release, but now that we no longer need to generate this empty tmp file we prefer to remove this code.

@hiddeco has a strong opinion about implementing this with a custom Authorizer that bypasses any kind of disk persistence logic (which is fine by me) and left these pointers:

https://github.com/helm/helm/blob/v3.18.2/pkg/registry/client.go#L198

https://github.com/helm/helm/blob/v3.18.2/pkg/registry/client.go#L147-L160

import (
	"context"
	"fmt"
	"io"

	"github.com/hashicorp/go-cleanhttp"
	"helm.sh/helm/v3/pkg/registry"
	"oras.land/oras-go/v2/registry/remote"
	"oras.land/oras-go/v2/registry/remote/auth"
	"oras.land/oras-go/v2/registry/remote/credentials"

	"github.com/akuity/kargo/pkg/x/version"
)

type EphemeralAuthorizer struct {
	authorizer *auth.Client
	store      credentials.Store
}

func NewEphemeralAuthorizer() (*EphemeralAuthorizer, error) {
	store := credentials.NewMemoryStore()
	authorizer := &auth.Client{
		Client:     cleanhttp.DefaultClient(),
		Cache:      auth.NewCache(),
		Credential: credentials.Credential(store),
	}
	authorizer.SetUserAgent("Kargo/" + version.GetVersion().Version)

	return &EphemeralAuthorizer{
		authorizer: authorizer,
		store:      store,
	}, nil
}

func (a *EphemeralAuthorizer) Login(ctx context.Context, host, username, password string) error {
	reg, err := remote.NewRegistry(host)
	if err != nil {
		return err
	}
	reg.Client = a.authorizer

	cred, err := a.authorizer.Credential(ctx, host)
	if err != nil {
		return fmt.Errorf("fetching credentials for %q: %w", host, err)
	}

	if err = reg.Ping(ctx); err != nil {
		return fmt.Errorf("authenticating to %q: %w", host, err)
	}

	// NB: Deal with "special" behavior around docker.io and its subdomains.
	key := credentials.ServerAddressFromRegistry(host)
	key = credentials.ServerAddressFromHostname(key)

	return a.store.Put(ctx, key, cred)
}

func NewRegistryClient(authorizer auth.Client) (*registry.Client, error) {
	opts := []registry.ClientOption{
		registry.ClientOptWriter(io.Discard),
		registry.ClientOptAuthorizer(authorizer),
		// NB: Options like ClientOptCache and ClientOptHTTPClient do not have
		// an effect on the registry client when using the authorizer, as they
		// are only set when NewClient constructs a new authorizer internally.
	}

	return registry.NewClient(opts...)
}