How can I use kubeconfig resource

[NicoFgrx]

Hello there 👋

TLDR : I can't use the NewKubeconfig() output as input for kubernetes.NewProvider().

The long story :
I'm trying to bootstrap a Talos cluster with Pulumi (pulumi-proxmoxve for VM creation) and pulumi-talos to bootstrap the cluster.
Then, I want to automatically deploy helm chart on the cluster.

The Talos cluster is configured correctly with this extract of code (i'm using Go) :

// ...
secrets, err := machine.NewSecrets(ctx, "secrets", &machine.SecretsArgs{..})
// ...

for _, item := range kubernetesNodes {
    configuration := machine.GetConfigurationOutput(ctx, machine.GetConfigurationOutputArgs{...})
    // ...
    configurationApply, err := machine.NewConfigurationApply(...)
    // ...
}
//...
bootstrap, err := machine.NewBootstrap(...)
//...

Then, I want to retrieve the kubeconfig file with :

// Create Kubeconfig resource 
k, err := cluster.NewKubeconfig(ctx, "kubeconfig", &cluster.KubeconfigArgs{
    ClientConfiguration: cluster.KubeconfigClientConfigurationArgs{
        CaCertificate:     secrets.ClientConfiguration.CaCertificate(),
        ClientCertificate: secrets.ClientConfiguration.ClientCertificate(),
        ClientKey:         secrets.ClientConfiguration.ClientKey(),
    },
    Node: pulumi.String(item.Networks[0].Address),
}, pulumi.DependsOn([]pulumi.Resource{bootstrap}))
if err != nil {
    return nil, fmt.Errorf("error while create kubeconfig on node %s, got %v", item.Name, err)
}

Next, I use the k variable to create a kubernetes Provider with this code :

// Create kubernets provider based on kubeconfig
p, err := kubernetes.NewProvider(ctx, "kube-provider", &kubernetes.ProviderArgs{
    Kubeconfig: k.KubeconfigRaw,
}, pulumi.DependsOn([]pulumi.Resource{k}))
if err != nil {
    return errors.Wrap(err, "create kubernetes provider")
}

kopts := []pulumi.ResourceOption{}
kopts = append(kopts, pulumi.Provider(p))
kopts = append(kopts, pulumi.DependsOn([]pulumi.Resource{p}))

Finally, I try to deploy helm chart using this code

a, err := helmv3.NewRelease(ctx, "metallb", &helmv3.ReleaseArgs{
    Chart: pulumi.String("metallb"),
    Name:  pulumi.String("metallb"),
    RepositoryOpts: helmv3.RepositoryOptsArgs{
        Repo: pulumi.String("https://metallb.github.io/metallb"),
    },
    Namespace: pulumi.String("metallb-system"),
    Version:   pulumi.String("v0.14.9"),
    Values: pulumi.Map{
        "frr": pulumi.Map{
            "enabled": pulumi.Bool(false),
        },
    },
}, kopts...)
if err != nil {
    return nil, err
}

But i got this error message :

error: can't create Helm Release with unreachable cluster: unable to load schema information from the API server: Get "https://cp.x.x.x:6443/openapi/v2?timeout=32s": dial tcp y.y.y.y:6443: connect: connection refused

k.KubeconfigRaw is also exported as Pulutmi output, and when I'm using it in file I can reach the cluster, so I don't think it's a network issue here. And the cluster is healthy with all Pod Running correctly.
I have no idea where this can came from, i tried with several wait between the apply and the bootstrap but I got the same error again and again. Any idea to debug ?

[kylepl]

Have you tried putting something other than a Helm release into the cluster, to see if it's just any access doesn't work? e.g. a small deployment/job - just to see if it can start?

Also I've found it useful to sometimes dump out the k8s config, and just run kubectl commands for debugging.

I've done something like:

kubeConfig.KubeconfigRaw.ApplyT(func(kubeconfig string) (interface{}, error) {
		err := os.WriteFile("/tmp/talos_k8s_config_to_use", []byte(kubeconfig), 0o600)
		if err != nil {
			return nil, err
		}
		return nil, nil
	})

Then you could see if kubectl works at all.

[NicoFgrx]

Hey 👋

Sorry for the slow reply... I didn’t have my PC handy.

Have you tried putting something other than a Helm release into the cluster, to see if it's just any access doesn't work? e.g. a small deployment/job - just to see if it can start?

From what I remember, no, I’ve only tried doing a Helm release using this plugin: https://www.pulumi.com/registry/packages/kubernetes/api-docs/helm/v4/chart/ with Golang.

Also I've found it useful to sometimes dump out the k8s config, and just run kubectl commands for debugging.

Yes, that’s what I meant when I said:

k.KubeconfigRaw is also exported as Pulumi output, and when I'm using it in file I can reach the cluster, so I don’t think it’s a network issue here. And the cluster is healthy with all Pods Running correctly.

I used pulumi output to get the kubeconfig value and ran kubectl commands in the CLI.

Actually, I haven’t touched that part in a long time. I set up a workaround where I export the kubeconfig in my CI, then use an Ansible playbook in a step to handle the Helm releases.

I’ll try updating the project dependencies that use this to see if I still get the error, and I’ll update you on my progress once it’s done.