🌱 E2E: Install independent Metal3 IPAM after upgrade #2382

lentzi90 · 2025-03-07T13:25:40Z

What this PR does / why we need it:

The Metal3 IPAM was previously bundled with CAPM3. Now we deploy it
separately as a CAPI IPAM provider. In clusterctl upgrade tests going
from a version where IPAM is bundled, to a version where it is not, we
must install it after the upgrade.
This commit adds a post upgrade hook to the clusterctl upgrade tests
that installs the Metal3 IPAM.

NOTE: This also includes #2380 we can merge that first or we can close it. I just wanted to make sure I don't hit that issue when testing the upgrade.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

lentzi90 · 2025-03-07T13:34:24Z

/test metal3-e2e-clusterctl-upgrade-test-main

peppi-lotta · 2025-03-10T07:43:01Z

You are using metal3 as provider name but currently it's actually metal3ipam and this is why the test is not passing probably.

I created a PR to change it to Metal3 because that is a better and more correct name.
PR is here: metal3-io/metal3-dev-env#1509

adilGhaffarDev · 2025-03-10T09:22:23Z

/test metal3-e2e-clusterctl-upgrade-test-main

adilGhaffarDev · 2025-03-10T09:25:24Z

let's change the provider name here too:

cluster-api-provider-metal3/test/e2e/pivoting.go

Line 39 in 6fef657

Metal3ipamProviderName = "metal3ipam"

lentzi90 · 2025-03-10T11:04:04Z

/test metal3-e2e-clusterctl-upgrade-test-main

tuminoid · 2025-03-10T17:13:32Z

/test metal3-e2e-clusterctl-upgrade-test-main

tuminoid · 2025-03-12T05:36:02Z

/test metal3-e2e-clusterctl-upgrade-test-main

lentzi90 · 2025-03-12T15:39:05Z

/test metal3-e2e-clusterctl-upgrade-test-main

tuminoid · 2025-03-13T06:43:32Z

It is still complaining about target provider components for IPAM, even with the new name. I don't see open PRs anymore in any repo related to this, besides this one. What are we missing?

lentzi90 · 2025-03-13T07:02:29Z

There is something wrong with the clusterctl-config. It doesn't have the Metal3 IPAM provider listed. 🤔

tuminoid · 2025-03-13T07:42:01Z

There is something wrong with the clusterctl-config. It doesn't have the Metal3 IPAM provider listed. 🤔

We added it here and adjusted that here ... hmm.

adilGhaffarDev · 2025-03-13T07:52:08Z

There is something wrong with the clusterctl-config. It doesn't have the Metal3 IPAM provider listed. 🤔

We can move the following code to ci-e2e.sh

https://github.com/metal3-io/metal3-dev-env/blob/d831cba0825498a685d3b116183ee79dad00e358/03_launch_mgmt_cluster.sh#L499C1-L505C4

and try again. But I am not sure why dev-env is not setting it.
It should be fine to move it there for the main branch.

lentzi90 · 2025-03-13T08:07:53Z

Dev-env is setting it but possibly not in the place that is used by this test. There could also be something with the overrides structure that need to be adjusted.

lentzi90 · 2025-03-13T08:12:43Z

This is from earlier in the test. Does it look correct? I'm getting very confused with all the overrides and provider=... 😵

 + clusterctl init --core cluster-api:v1.9.5 --bootstrap kubeadm:v1.9.5 --control-plane kubeadm:v1.9.5 --infrastructure=metal3:v1.10.99 -v5 --ipam=metal3:v1.10.99
 Using configuration file="/home/metal3ci/.config/cluster-api/clusterctl.yaml"
 Installing the clusterctl inventory CRD
 Creating CustomResourceDefinition="providers.clusterctl.cluster.x-k8s.io"
 Fetching providers
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/cluster-api/v1.9.5/core-components.yaml" provider="cluster-api" version="v1.9.5"
 Fetching file="core-components.yaml" provider="cluster-api" type="CoreProvider" version="v1.9.5"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/bootstrap-kubeadm/v1.9.5/bootstrap-components.yaml" provider="bootstrap-kubeadm" version="v1.9.5"
 Fetching file="bootstrap-components.yaml" provider="kubeadm" type="BootstrapProvider" version="v1.9.5"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/control-plane-kubeadm/v1.9.5/control-plane-components.yaml" provider="control-plane-kubeadm" version="v1.9.5"
 Fetching file="control-plane-components.yaml" provider="kubeadm" type="ControlPlaneProvider" version="v1.9.5"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/infrastructure-metal3/v1.10.99/infrastructure-components.yaml" provider="infrastructure-metal3" version="v1.10.99"
 Using override="infrastructure-components.yaml" provider="infrastructure-metal3" version="v1.10.99"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/ipam-metal3/v1.10.99/ipam-components.yaml" provider="ipam-metal3" version="v1.10.99"
 Using override="ipam-components.yaml" provider="ipam-metal3" version="v1.10.99"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/cluster-api/v1.9.5/metadata.yaml" provider="cluster-api" version="v1.9.5"
 Fetching file="metadata.yaml" provider="cluster-api" type="CoreProvider" version="v1.9.5"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/bootstrap-kubeadm/v1.9.5/metadata.yaml" provider="bootstrap-kubeadm" version="v1.9.5"
 Fetching file="metadata.yaml" provider="kubeadm" type="BootstrapProvider" version="v1.9.5"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/control-plane-kubeadm/v1.9.5/metadata.yaml" provider="control-plane-kubeadm" version="v1.9.5"
 Fetching file="metadata.yaml" provider="kubeadm" type="ControlPlaneProvider" version="v1.9.5"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/infrastructure-metal3/v1.10.99/metadata.yaml" provider="infrastructure-metal3" version="v1.10.99"
 Using override="metadata.yaml" provider="infrastructure-metal3" version="v1.10.99"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/ipam-metal3/v1.10.99/metadata.yaml" provider="ipam-metal3" version="v1.10.99"
 Using override="metadata.yaml" provider="ipam-metal3" version="v1.10.99"

adilGhaffarDev · 2025-03-13T08:22:11Z

This is from earlier in the test. Does it look correct? I'm getting very confused with all the overrides and provider=... 😵

 + clusterctl init --core cluster-api:v1.9.5 --bootstrap kubeadm:v1.9.5 --control-plane kubeadm:v1.9.5 --infrastructure=metal3:v1.10.99 -v5 --ipam=metal3:v1.10.99
 Using configuration file="/home/metal3ci/.config/cluster-api/clusterctl.yaml"
 Installing the clusterctl inventory CRD
 Creating CustomResourceDefinition="providers.clusterctl.cluster.x-k8s.io"
 Fetching providers
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/cluster-api/v1.9.5/core-components.yaml" provider="cluster-api" version="v1.9.5"
 Fetching file="core-components.yaml" provider="cluster-api" type="CoreProvider" version="v1.9.5"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/bootstrap-kubeadm/v1.9.5/bootstrap-components.yaml" provider="bootstrap-kubeadm" version="v1.9.5"
 Fetching file="bootstrap-components.yaml" provider="kubeadm" type="BootstrapProvider" version="v1.9.5"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/control-plane-kubeadm/v1.9.5/control-plane-components.yaml" provider="control-plane-kubeadm" version="v1.9.5"
 Fetching file="control-plane-components.yaml" provider="kubeadm" type="ControlPlaneProvider" version="v1.9.5"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/infrastructure-metal3/v1.10.99/infrastructure-components.yaml" provider="infrastructure-metal3" version="v1.10.99"
 Using override="infrastructure-components.yaml" provider="infrastructure-metal3" version="v1.10.99"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/ipam-metal3/v1.10.99/ipam-components.yaml" provider="ipam-metal3" version="v1.10.99"
 Using override="ipam-components.yaml" provider="ipam-metal3" version="v1.10.99"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/cluster-api/v1.9.5/metadata.yaml" provider="cluster-api" version="v1.9.5"
 Fetching file="metadata.yaml" provider="cluster-api" type="CoreProvider" version="v1.9.5"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/bootstrap-kubeadm/v1.9.5/metadata.yaml" provider="bootstrap-kubeadm" version="v1.9.5"
 Fetching file="metadata.yaml" provider="kubeadm" type="BootstrapProvider" version="v1.9.5"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/control-plane-kubeadm/v1.9.5/metadata.yaml" provider="control-plane-kubeadm" version="v1.9.5"
 Fetching file="metadata.yaml" provider="kubeadm" type="ControlPlaneProvider" version="v1.9.5"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/infrastructure-metal3/v1.10.99/metadata.yaml" provider="infrastructure-metal3" version="v1.10.99"
 Using override="metadata.yaml" provider="infrastructure-metal3" version="v1.10.99"
 Potential override file searchFile="/home/metal3ci/.config/cluster-api/overrides/ipam-metal3/v1.10.99/metadata.yaml" provider="ipam-metal3" version="v1.10.99"
 Using override="metadata.yaml" provider="ipam-metal3" version="v1.10.99"

this seems fine to me, here --ipam=metal3 is working fine. This is coming from dev-env, which is launching a management cluster. What seems wrong here?

lentzi90 · 2025-03-13T08:32:36Z

It is probably correct. I'm just getting confused by at all.
I found another thing to check now... do we need to add it in the e2e config? We have config for all the other things there (core, control-plane, bootstrap and infra).

adilGhaffarDev · 2025-03-13T08:36:10Z

I found another thing to check now... do we need to add it in the e2e config? We have config for all the other things there (core, control-plane, bootstrap and infra).

yes we need to add there, I forgot about that. For e2e it gets it from there.

adilGhaffarDev · 2025-03-27T09:40:05Z

I was planning to use the release artifacts instead of taking it from main. Should we do that first? I'm happy either way. We can merge first and then I do it in a follow up or the other way

I think we can fix it in this PR. its a small change

kashifest · 2025-03-27T09:40:42Z

I was planning to use the release artifacts instead of taking it from main. Should we do that first? I'm happy either way. We can merge first and then I do it in a follow up or the other way

It should be there now

lentzi90 · 2025-03-27T10:17:02Z

/test metal3-e2e-clusterctl-upgrade-test-main
/test metal3-centos-e2e-integration-test-main
/test metal3-ubuntu-e2e-integration-test-main

kashifest · 2025-03-27T10:19:23Z

/lgtm

adilGhaffarDev · 2025-03-27T10:29:17Z

let's also unhold
/unhold

lentzi90 · 2025-03-27T10:29:45Z

Wait, let's see that the clusterctl upgrade works also after the last changes
/hold

lentzi90 · 2025-03-27T10:49:08Z

I forgot to set type: "url".

lentzi90 · 2025-03-27T10:49:29Z

/test metal3-e2e-clusterctl-upgrade-test-main
/test metal3-centos-e2e-integration-test-main
/test metal3-ubuntu-e2e-integration-test-main

lentzi90 · 2025-03-27T11:46:37Z

Ugh, we still need to get the metadata.yaml...

lentzi90 · 2025-03-27T11:53:52Z

Let's see if this works...
/test metal3-e2e-clusterctl-upgrade-test-main
/test metal3-centos-e2e-integration-test-main
/test metal3-ubuntu-e2e-integration-test-main

The Metal3 IPAM was previously bundled with CAPM3. Now we deploy it separately as a CAPI IPAM provider. In clusterctl upgrade tests going from a version where IPAM is bundled, to a version where it is not, we must install it after the upgrade. This commit adds a post upgrade hook to the clusterctl upgrade tests that installs the Metal3 IPAM. Signed-off-by: Lennart Jern <[email protected]>

lentzi90 · 2025-03-27T12:56:25Z

/test metal3-e2e-clusterctl-upgrade-test-main

lentzi90 · 2025-03-27T13:55:30Z

/test metal3-centos-e2e-integration-test-main
/test metal3-ubuntu-e2e-integration-test-main

lentzi90 · 2025-03-27T15:12:43Z

Looks like a flake pulling ironic. Both tests passed and then it failed in AfterEach.

/test metal3-e2e-clusterctl-upgrade-test-main

lentzi90 · 2025-03-27T17:23:24Z

/hold cancel
All good now!

kashifest · 2025-03-27T19:26:55Z

/lgtm

metal3-io-bot requested review from adilGhaffarDev and mquhuy March 7, 2025 13:25

metal3-io-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Mar 7, 2025

lentzi90 force-pushed the lentzi90/clusterctl-upgrade-ipam branch from f25f534 to ef7f5cc Compare March 7, 2025 13:30

lentzi90 mentioned this pull request Mar 10, 2025

🌱 E2E: Ensure cert-manager webhook is available #2380

Merged

lentzi90 force-pushed the lentzi90/clusterctl-upgrade-ipam branch from ef7f5cc to 03dd3e7 Compare March 10, 2025 10:59

lentzi90 force-pushed the lentzi90/clusterctl-upgrade-ipam branch from 03dd3e7 to 3612a78 Compare March 12, 2025 13:52

metal3-io-bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 12, 2025

lentzi90 closed this Mar 13, 2025

lentzi90 force-pushed the lentzi90/clusterctl-upgrade-ipam branch from 3612a78 to 5dd8f6e Compare March 13, 2025 08:55

metal3-io-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Mar 13, 2025

lentzi90 force-pushed the lentzi90/clusterctl-upgrade-ipam branch from 2cc9596 to a8a4cff Compare March 27, 2025 10:16

metal3-io-bot removed the lgtm Indicates that a PR is ready to be merged. label Mar 27, 2025

metal3-io-bot added the lgtm Indicates that a PR is ready to be merged. label Mar 27, 2025

metal3-io-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 27, 2025

metal3-io-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 27, 2025

lentzi90 force-pushed the lentzi90/clusterctl-upgrade-ipam branch from a8a4cff to 53a8957 Compare March 27, 2025 10:48

metal3-io-bot removed the lgtm Indicates that a PR is ready to be merged. label Mar 27, 2025

lentzi90 force-pushed the lentzi90/clusterctl-upgrade-ipam branch from 53a8957 to c70e361 Compare March 27, 2025 11:53

lentzi90 force-pushed the lentzi90/clusterctl-upgrade-ipam branch from c70e361 to 4c02c7e Compare March 27, 2025 12:56

metal3-io-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 27, 2025

metal3-io-bot added the lgtm Indicates that a PR is ready to be merged. label Mar 27, 2025

metal3-io-bot merged commit 40a76be into metal3-io:main Mar 27, 2025
15 checks passed

metal3-io-bot deleted the lentzi90/clusterctl-upgrade-ipam branch March 27, 2025 19:27

metal3-io-bot added this to the CAPM3 - v1.10 milestone Mar 27, 2025

🌱 E2E: Install independent Metal3 IPAM after upgrade #2382

🌱 E2E: Install independent Metal3 IPAM after upgrade #2382

Uh oh!

Conversation

lentzi90 commented Mar 7, 2025

Uh oh!

lentzi90 commented Mar 7, 2025

Uh oh!

peppi-lotta commented Mar 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adilGhaffarDev commented Mar 10, 2025

Uh oh!

adilGhaffarDev commented Mar 10, 2025

Uh oh!

lentzi90 commented Mar 10, 2025

Uh oh!

tuminoid commented Mar 10, 2025

Uh oh!

tuminoid commented Mar 12, 2025

Uh oh!

lentzi90 commented Mar 12, 2025

Uh oh!

tuminoid commented Mar 13, 2025

Uh oh!

lentzi90 commented Mar 13, 2025

Uh oh!

tuminoid commented Mar 13, 2025

Uh oh!

adilGhaffarDev commented Mar 13, 2025

Uh oh!

lentzi90 commented Mar 13, 2025

Uh oh!

lentzi90 commented Mar 13, 2025

Uh oh!

adilGhaffarDev commented Mar 13, 2025

Uh oh!

lentzi90 commented Mar 13, 2025

Uh oh!

adilGhaffarDev commented Mar 13, 2025

Uh oh!

adilGhaffarDev commented Mar 27, 2025

Uh oh!

kashifest commented Mar 27, 2025

Uh oh!

lentzi90 commented Mar 27, 2025

Uh oh!

kashifest commented Mar 27, 2025

Uh oh!

adilGhaffarDev commented Mar 27, 2025

Uh oh!

lentzi90 commented Mar 27, 2025

Uh oh!

lentzi90 commented Mar 27, 2025

Uh oh!

lentzi90 commented Mar 27, 2025

Uh oh!

lentzi90 commented Mar 27, 2025

Uh oh!

lentzi90 commented Mar 27, 2025

Uh oh!

lentzi90 commented Mar 27, 2025

Uh oh!

lentzi90 commented Mar 27, 2025

Uh oh!

lentzi90 commented Mar 27, 2025

Uh oh!

lentzi90 commented Mar 27, 2025

Uh oh!

kashifest commented Mar 27, 2025

Uh oh!

Uh oh!

Uh oh!

peppi-lotta commented Mar 10, 2025 •

edited

Loading