HostClaim Implementation

[pierrecregut]

As discussed during nov 2025 MTM, this is a discussion on the implementation of HostClaims covering at least:

• task breakdown and progress
• details not covered by the blueprint especially when they interact with other bmo or capm3 new features
• new needs (eg: data templating for HostClaims used without capm3)
• test strategy, documentation, etc.

[pierrecregut]

[New Component] HostClaim Data Templater

This is a summary of a requirement expressed by Didrik Koren and @lentzi90 on metal3 slack.
When a HostClaim is used without CAPM3, its configuration, especially the network configuration, will depend on the BareMetalHost it is associated to. So we need a template mechanism similar to Metal3DataTemplate except that it will limit the use of labels/annotations to the BareMetalHost (and potentially HostClaim) but that does not refer to Machine or Metal3Machine:

• We will defer this work item after the initial integration with CAPM3. It will probably require its own blueprint.
• How is the template associated to the HostClaim ?
  • an umbrella resource that associates such templates to HostClaims and will render it when the HostClaim is associated (risk: multiple umbrella resources targeting a single HostClaim unless the umbrella resource has the name of the HostClaim).
  • an annotation or field on the HostClaim that is only used for this use-case and not with the capm3 provider.
• Templates custom resources at the capm3 or HostClaim level will share 99% of the content (no cluster reference for HostClaims) but they will probably have to be duplicated. Probably the same for the rendering logic unless we push it in a separate package in baremetal-operator used by capm3 provider.

[nerok]

This covers a lot of my thoughts on the network config. I did start to wonder if there could be reasons to have networking both for the infra provider and the consumer, say infra knows that the machine needs to bond to specific interfaces, but might not know which vlan or IPs are in use. But after considering it, I think it would be to complex.

Though I see that the "vendor", and maybe "metadata" of cloud-init could really get to use here too, allowing the infra provider to create them.

For the actual network data templater I believe there are two approaches, the simple and the structured one. The simple would be a text template, say Go-template, just with access to all of hardware data and/or BMH. The structured one would require a go-library to generate a config-drive network config (openstack network config? Don't feel like I have a good name for that, I only notice that it is quite different from the one in cloud-init documentation), and possibly some selectors or something to specify the correct field in the BMH or HardwareData.

Which approach is better? Don't really know, I added both in my case, but used the structured (opinionated) one. That one required the physical links to match the links in HD, and got their mac-address from there, then it propagated the mac-address based on the "parent interface" (bond has members, vlan is on a existing interface etc).

[pierrecregut]

There was a discussion on this topic during MTM for preprovisionning data (from 20:30 in the recording) that evolved to a discussion on making network data more generic. To make more clear what I meant by templates with functions, a raw prototype in Kanod is here : https://gitlab.com/Orange-OpenSource/kanod/host-baremetal/-/blob/main/internal/controller/network_data_template.go?ref_type=heads . But for HostClaims without capm3 it would not be understandable to have a solution that is not similar to the standard case with capm3.

[pierrecregut]

[Detail] Status of the HostClaim

• Expose as much as possible through Kubernetes conditions (no reason to support obsolete capi conditions).
• No Phase or FailureMessage/Reason
• Initial proposal: distinguish at least Associated and Synchronized condition (most errors were used after the synchronization and could easily be captured by the Synchronization condition). Both summarize in a Ready condition.
• We should also expose the status of the BareMetalHost as conditions. There are two major stable states that can be exposed as conditions : Provisioned and Available, the others can be seen as intermediate steps toward those goals and expressed as reasons why the condition is (not yet) fulfilled. With CapM3, we are only interested in the Provisioned condition: the BMH was available when it was chosen and the HostClaim will be dropped when the Metal3Machine is replaced. But it may not be the case for HostClaims used without CAPM3 and we could also think of alternate implemetnation of node-reuse. So Available should probably also be exposed.
• We must not expose to many details at that level. There must be a clean separation between what consumers and hardware teams see.

[nerok]

An issue I experienced when trying this myself was that one machine failed, but I had multiple other machines matching the same selector (the selector chose any machine of a particular kind, on purpose). I added support for failing to another machine if deploy to one failed (and tracked both the machine currently in use and all that had previously failed, and if all failed, we rolled back to the first and retried that again).

If this is something we want in HostClaim, it might need to be configurable, as this might not be what everyone wants, and it might also depend from machine to machine or differ based on HostDeployPolicy.

Another issue which occured in the same situation was when we created multiple machines at once, they tried to take the same machine, but that quickly got resolved after using consumerRef. Not that important, just as a note.

[pierrecregut]

The Kanod hostclaim prototype and the coming metal3 version use the same choice/locking logic as the metal3 capi provider between a metal3machine and a bareMetalHost. The main question is how to avoid duplicating the code despite the few important variations.

[pierrecregut]

On failure recovery, groups of machines, etc. HostClaim is about binding a single workload to a BareMetalHost. There may be a need for an equivalent of MachineSets, MachineDeployments at the level of HostClaim but then there is also a need for a generic liveness probe to implement it correctly and this is a different topic.
My opinion is that when you really need that, you are better off pretending your machine is a cluster or integrate it into an existing cluster. The added kubelet will be the liveness probe and it will host all your telemetry needs without reinventing the wheel even if the main workload is not a container.

[pierrecregut]

[Tests] HardwareData e2e tests

HostClaims rely a lot on the separation BareMetalHost/HardwareData to hide the BMH from the end user. Current e2e test only look at the hardwareDetails in status (potentially fed by an annotation of the resource). We probably should add e2e tests on HardwareData and provide the HardwareData with inspection disabled in the e2e tests of HostClaims.

[lentzi90]

We recently added a simple test case that makes use of HardwareData for external inspection. This should hopefully be useful:

baremetal-operator/test/e2e/external_inspection_test.go

Line 251 in 11b070e

It("should skip inspection and become available when HardwareData exists and BMH has inspection disabled", func() {

.

[pierrecregut]

[Task Breakdown]

• Implementation of HostClaim resource and controller with companion HostDeployPolicy resource in baremetal-operator project
  • Definition of custom resources
  • Controller implementation and unit testing
  • E2E tests
• Using HostClaims in cluster-api-provider-metal3
  • modifications of the Metal3Machine CRD (namespace in selector, identityRef, etc.)
  • implementation of the Metal3Machine reconciliation with HostClaims as a Metal3MachineManager, hook to attach it to the Metal3MachineController.
  • metal3data rendering when the m3m is attached to a hostclaim abstraction of BMH behind a data getter.
  • support of remediation with HostClaims.
  • Tests
• Documentation of HostClaims and their use
  • hostclaim resource and association with baremetalhost
  • usage with the cluster api provider
• Using hostclaims in Metal3 devenv
• Templating for HostClaims (without capm3).

[pierrecregut]

@Rozzii may be, I could open issues for the main items in the relevant projects and sub-issues for level 2 items so that that the discussion can stay organized when there is some kind of agreement on the break-down ? What is the current practice in Metal3 projects ? I would modify the plan above to point to those issues.

[pierrecregut]

#2856 defines the hostclaim resource. The full implementation in baremetal operator is visible on branch hostclaim: https://github.com/pierrecregut/baremetal-operator/tree/hostclaim

Commits are huge but splitting them does not make a lot of sense.

An embryo of documentation in the book is here metal3-io/metal3-docs#641

[pierrecregut]

Node Reuse
The first implementation of the m3machine controller with hostclaim delegated all the work to the hostclaim controller. The implementation in the hostclaim controller was an exact copy of the implementation of the m3m controller targeting bmh. This worked... but if anything did go wrong, the user managing the cluster could not remove the node reuse marks on the bmh (the whole purpose of hostclaims is that he does not manage the bmh). On the other hand, the infra manager who handles the BMH has no vision on the validity of those annotations...

We have completely changed the implementation to do all the work in the capm3 provider. When a m3machine associated to a hostclaim marked for node reuse, is deleted, we do not delete the hostclaim. Instead, we clear its fields (image, userdata, etc.) turn it off and remove the consumerRef (https://github.com/pierrecregut/cluster-api-provider-metal3/blob/hostclaim/baremetal/metal3machine_host_manager.go#L194-L217). When we create the new m3machine, we try to associate it with a hostclaim with no consumerRef and the same node-reuse label (https://github.com/pierrecregut/cluster-api-provider-metal3/blob/hostclaim/baremetal/metal3machine_host_manager.go#L606C1-L633C3). If it fails, we create a new hostclaim. If it succeeds, we wait until the node is marked available (deprovisioning takes time) using an available condition which reflects BMH condition (#2888).

This does not solve the issue that no label will be removed if the size of the deployment shrinks. But this problem existed in the standard capm3 implementation without hostclaim. At least the cluster manager can delete the hostclaim and we added an annotation to record the time when the consumerRef was removed (it may not be necessary: we could use the transition time of the Available condition which should be true for such a HostClaim). The time can be used to garbage collect hostClaims that are unbound for too long.

Comments are welcome on the choice of approach and its implementation (@lentzi90, the original authors of the node-reuse implementation)

[lentzi90]

I am hardly the original author, but I probably touched the code 😅
From CAPM3 point of view I do not have an issue with this approach. But I am wondering about "standalone" use-cases. Would this mean that in order to use HostClaims, CAPM3 is mandatory. I mean even when used without CAPI, for non-kubernetes things, we would still need to deploy CAPM3 then to have the HostClaim feature?

[pierrecregut]

It just means that if you want to reuse a node in the standalone case, you should keep the hostclaim, clear its field to push it to available state and reprovision it with your new workload, rather than relying on a propagated label shared between different hostclaims. Without hostclaims, the notion of node reuse does not make really sense for standalone use of baremetal hosts. Worse there is a good chance that an external operator consuming BMH will not honor the node-reuse label set by the capm3 operator (as the consumerRef of a reusable bmh is empty until it is reassociated). There is also an empty consumerRef for HostClaims but the purpose of the consumerRef is slightly different.

[dtantsur]

I don't read the proposal as requiring CAPM3 for HostClaims (that would be a no-go from my perspective), just that the HostClaims controller will support "empty" (or better "non-provisioning") host claims, which can be used to reserve hosts that are currently available or deprovisioning.

[lentzi90]

Ah ok now I think I get it. This sounds good to me!

Without hostclaims, the notion of node reuse does not make really sense for standalone use of baremetal hosts.

Of course, I don't know what I was thinking. 😂

[pierrecregut]

Automated Cleaning Mode As we agree on the way to handle node-reuse, a side question is how we handle cleaning. @dtantsur has rightfully pointed out that end-users should not be able to do whatever they want with cleaning. A solution would be to say that with hostclaims, cleaning is always metadata (forced on the bmh while removing the finalizer) when the hostclaim is deleted, and could be whatever the user wants (propagated from hostclaim to bmh during Associate/Update), when the hostclaim is just deprovisioned. Usually, this is disabled because of node-reuse. @lentzi90 do you agree there are no valid use case where somebody may want to disable cleaning when we delete the hostclaim.

[dtantsur]

I don't mind extending consumerRef as long as nobody gets confused (I assumed consumerRef is pretty standard in k8s)

[lentzi90]

Could we maybe rely on external tools for this instead? If the server admin decides that certain cleaning modes must be used or must not be used, they could use Kyverno or some other policy agent to enforce it. Either use mutating webhook to change it to the preferred value, or deny the request if it is outside the policy. We would need to make sure the host claim controller can tolerate this though.

[diconico07]

This would solve the allowed values inside a host claim, but here the issue is that deprovisioning while holding a HostClaim may be fine with cleaning disabled, but when dropping the claim you probably want a clean. Not sure a Kyverno policy would be able to ensure that.

[zaneb]

It makes sense to me that a consumer should be able to override the cleaning mode to disable it as long as it holds the consumer ref (and no longer). This presumably means we'll need to store the UID of the consumer in the status. I think we can implement this by kicking the host back from Available to Preparing when the consumer ref changes?
We should probably fix #2687 to make things simpler to reason about.
And we'll need either an extra field or an annotation for the consumer to indicate its desire to disable cleaning. As much as I don't love putting the field inside the consumer ref, it does have the advantage that existing consumers will automatically clear it without any code changes.

[pierrecregut]

There is an on-going experiment. The approach chosen is to put a constraint in the CRD so that the override can only exist if there is a consumerRef. But the user still has to destroy both fields. I did not fix #2687 but if the first consumer is a hostclaim, automatedCleaning will probably be disabled on the provisioning, so cleaning will not be triggered.

[dtantsur]

On a somewhat related note to https://github.com/orgs/metal3-io/discussions/2795#discussioncomment-15485815, we need to decide what we do about force-deleted HostClaims. We already have bad experience with users force-deleting BareMetalHosts, somebody will definitely try to "solve" their problems by kubectl --force. Doing so will created an orphaned instance that is no longer owned by anything.

On top of that, in many of the cleaning suggestions, forced deletion of a host claim will leave the BMH with whatever automatedCleaningMode was last set, which could be the value from the HostClaim. This is even more dangerous in combination with the empty HostClaim. Imagine, I set automatedCleaningMode=disabled and deprovision the HostClaim. My data, which may include some sort of exploits, is still on the disk. This is the same problem as in https://github.com/orgs/metal3-io/discussions/2795#discussioncomment-15531762, but now I also force-delete my HostClaim. Now only the stale consumerRef is preventing this host from being used by someone else.

Any ideas here?

[diconico07]

We may want to have support for that then, an annotation akin to the reboot one to trigger cleaning of a BMH in "available" state.
If we also add more details about cleaning phase in the status, the HostClaim controller (or whatever prune the stale HostClaim reference) can add the annotation when removing the link if needed (i.e. if BMH is available and last clean is not the expected kind).

[dtantsur]

I've drafted an RFE for that, comments welcome: #2922

[dtantsur]

To circle back to this, currently the discussions are around the option to add automated clean mode to consumerRef, record the last mode used (in general or only through the override - we haven't fully agreed yet) and re-run cleaning through the preparation flow if the mode used is "lower" (disabled < metadata < future full cleaning) than the one configured on the BMH.

[dtantsur]

A potential issue: *Ref fields tend to have the same structure in Kubernetes: namespace, name, maybe kind. Shouldn't we add something like BMH.Spec.InstanceOverrides for automated cleaning (and maybe more fields with defaults if we need to)?

[pierrecregut]

We probably can with an admission policy that requires consumerRef == nil => instanceOverrides == nil. And yes it is better to keep consumerRef as a corev1.ObjectRef instead of a type extending it. I would call it consumerOverrides to insist on the fact it is linked to the consumerRef (and it is not technically an instance override has different bmh may have different default values).

[dtantsur]

Today we touched upon the interaction between HostFirmwareSettings/HostFirmwareComponents and HostClaims. While I don't think we need to solve this problem in the first iteration of HostClaims, @jacob-anders and I have discussed it briefly, here are our thoughts:

• We don't see a point for allowing untrusted tenants to upgrade and especially downgrade firmware. New versions can be buggy, old versions can have vulnerability.
• We may add a curated list of safe firmware settings that a user can set on a HostClaim (or something like HostClaimFirmwareSettings, the exact API tbd). This needs to be opt-in. Maybe we extend the existing HostUpdatePolicy with something like SafeSettings. FirmwareSchema objects are probably safe to become readable by tenants.
• Trusted tenants can be allowed to edit HFS/HFC directly, I don't see a need for API additions for this case.

[pierrecregut]

Totally agree on HostFirmwareComponents this is clearly the infra admin responsibility.. On HFS (and may be raid configuration), there is a difference between what is authorized (and should be handled by the HostClaim controller) and what is the default (when you get a new bmh, we should expect a given configuration unless we change it) Override fields handle the last part and only the last part.
Do we need overrides for HFS and Raid ? It is more a convenience. Users should expect a default configuration unless they play with the settings. But it is not a security issue as it is for cleaning (not applying the default cleaning may expose sensitive data of the previous tenant).

[pierrecregut]

Selection within a namespace for HostDeployPolicy
This is one of the two requests for improvements from a team managing hardware at Orange with respect to HostClaims.

They would like to have a minimal description of the BMH, (bmc address and credential), launch inspection and label the BMH on the result of the inspection. So far, so good. The problem is that all BMH will end up in the same namespace independently of the characteristics. So they would like to be able to tell in the HDP, customer A (ie namespace A or with label user=A) can use the BMH of this namespace with label size value to small or medium but not large because he is not a premium customer.

This could be implemented as an added selector put in the HDP object but the semantics is really tricky.

The main issue is that now we may have several selectors coming from all the HDP policies in the namespace of the BMH that are satisfied by the user. Different HDP tend to be interpreted as a disjunction of requirements (you need to satisfy one of them).

• But, here should we not interpret it as a conjunction (you must satisfy all added selectors). For example if you are premium in disk and only regular in cpu, you should not end up with a high end cpu because the machine has disks.
• On the other hand, if you have just one category label, you may end up with HDP stating if you are a type a user, you can use category A machines, and another that if you are a type b, you can use category B. If you have both labels you should be able to use A and B which means we make a disjunction.

[diconico07]

So basically you add a LabelSelector to filter the exposed BMH using labels.

The main issue is that you may want both ways (disjunction and conjunction) in parallel, so you would basically need a mechanism similar to taints for Nodes.
The way it could work would be BMO get configured with a label prefix for taints, if a BMH has taints labels (i.e starting with said prefix), then it can only be used if they are tolerated by a combination of HDP that would select this host.

So adding two LabelSelectors one that "select" BMH and one that "tolerate" taints labels on them.
So during BMH selection it goes through all HDP matching the Claim's namespace, for each add all BMH matching to a map with their taints, removes the taints it tolerates from them, then potential BMH are all BMH in the list with no taints at the end.

So with your example:

• Namespace has labels
  • CategoryA
  • CategoryB
  • PremiumCPU
• HDP "catA"
  • For namespaces with "CategoryA" label
  • Select BMH with label "Category=A"
• HDP "catB"
  • For namespaces with "CategoryB" label
  • Select BMH with label "Category=B"
• HDP "premiumCPU"
  • For namespaces with "PremiumCPU" label
  • Tolerates "CPU=high-end"
• HDP "premiumDisk"
  • For namespaces with "PremiumDisk" label
  • Tolerates "hasDisks"
• BMH 1has labels
  • Category=A
  • taint/CPU=high-end
• BMH 2has labels
  • Category=B
• BMH 3has labels
  • Category=A
  • taint/hasDisk=true

would end up selecting either BMH 1 or 2 but excludes BMH 3
I think this would work without making the HDP too complex, nor making the BMH selection too complex.

[pierrecregut]

That is really elegant. My guess is that most infra managers will only choose either standard labels based (flavors) or taint labels based selections (independent characteristics) but not both especially if they give limited information on their inventory (next point). But it is great that it can handle a mix of the two.

[pierrecregut]

Hide the inventory
This is the second request for enhancement from the hardware teams.
In the current version of hostclaims, for each end-user we give r/w rights for hostclaim (and associated secrets) in a given namespace and read right on HardwareData. User see the full inventory and select from it. They even see the labels/annotations associated to each BMH.

Hardware teams do not want to share their inventory with customers. So there should be at least a mode where user do not need to have read right on HardwareData.

This can be easily achieved by copying/synchronizing hardwareData to the namespace of the user and pointing (hardwareData field in hostclaim status) to this resource. Copy of BMH metadata should be done on this copy.

[dtantsur]

You mean, copy a subset of HardwareData? If you copy the entire resource, you're not achieving anything.

I wonder if there are existing k8s solutions for partially copying resources.

[pierrecregut]

You only have access to the HardwareData of the BMH associated to your HostClaim, not all the other ones. You do not have a map of the datacenter.

[lentzi90]

So the request is to limit what users see to only the HardwareData they can actually use, right? Do you then have multiple HardwareData in the same namespace or are they already in separate namespaces? If it is ok to have them in separate namespaces, the user RBAC could be adjusted to only cover those.

On the other hand, if there are multiple HDs in the same namespace, I don't see other options than to copy/synchronize.

[nikParasyr]

Hello,

Went through the blueprint and the discussion here and this feature is very interesting for us. would like to add some questions/remarks about it, not sure if this is the right place.

Disclaimer:

• we are currently investigating rolling out metal3 for our infra, so no production experience and some of the remarks might be "wrong" due to lack of understanding
• we have about ~2000 baremetal nodes spread over 3 DCs (in close proximity, can be considered AZs), 6-7 service teams using them
• each service team will probably require multiple namespaces ( to seperate dev/stag/prod/... etc )
• we want both baremetal k8s but also non-k8s nodes (ratio atm is about 50-50, so considerable non k8s)
• we want k8s clusters spread over 3 DC

We have seen the following "problems" that the current proposal seem to tackle:

1. BMH has to be in the same namespace as Metal3Machine => HostClaim seems to solve this very nicely
2. for non-k8s nodes we do not want to give access to BMH to users => too much "sensitive" data and in general doesnt support seperation of concerns between baremetal infra team and service teams => HostClaim seems to supports this nicely, before this we were considering writing a custom controller just to hide options from service teams.
3. No meta/networkData templating for non-k8s nodes => not user-friendly to have to write networkData per node, to my understanding templating will be part of HostClaim which would help a lot

So the above points, very nice to see this proposal tackling them.

We are concerned about 1 more aspect that we are not sure how much it is tight to this proposal, specifically scaling. Due to the number of baremetal nodes and the fact we want multiple provisioning networks (due to multi-DC and secuirty reasons) we believe a single BMO and single ironic through IrSO will not cut it. For this we are considering:

1. Don't install ironic through IrSO => Install it a different way, add multiple conductors, rabbitmq etc etc <= Not super fan of this, this will create a single big ironic installation for all our users dependant on rabbitmq and other things, can be a SPOF
2. Try and implement something similar to the diagram mentioned here. <= we sort of like this as we prefer to have 20 "simple" ironic installations that are independant of each other, than one big, IrSO makes it also relatively easy to deploy them. But it introduces all kinds of other problems, namely:

• we need at least 1 ironic per DC(multiple either for scaling or for seperating provisioning networks for security reasons), BMO ties to 1 ironic currently and can be either cluster OR namespaced scope => if we do one BMO per namespace and one ironic, we cannot have clusters spread over 3 DCs (nodes will be on a different namespaces/ironics, metal3 does not support this).

In the current proposal, the HostClaim controller is part of BMO which means:

• if we deploy BMO as namespaced scope, HostClaim will also be namespaced scope, we get similar problems as mentioned above. Even if we can claim from different namespaces not sure if we really want multiple HostClaim controllers claiming the same BMH from different BMO.

The idea I had while going through the proposal and discussion was: if HostClaim is on its own controller-manager, we can have muliple ironics/BMO on the same cluster namespaced-scoped, but a single cluster-wide HostClaim controller that offers BMH to users. Not sure about other problems of such a design but i think it will help in:

• enabling multi-Ironic/BMO topologies (for scaling/security reasons) by retaining a single "interface" towards the end users and metal3
• if this is the supported way of scaling within the metal3 ecosystem, issues like multi-conductor ironic via IrSO or/and BMO talking to multiple ironics can be considered as out of scope.

Sidenote: we can potentially have similar behavior if hostClaim controller is part of BMO but will require a "golden" BMO that runs he hostClaim controller and some very sketchy RBACs. Also having this as a "recognized" pattern will help, instead of us doing downstream very sketchy changes.

Im not sure whether:

• I explain properly the limitations we face
• this is really the right spot to mention them
• im missing something due to limited knowledge and experience with metal3

If something is unclear or more input needed, let me know (i can also join the office hours to discuss there if needed)

( and also thanks a lot for all the work, as i mentioned above HostClaim already tackles multiple issues we had noted and it will help us a lot )

[dtantsur]

Don't install ironic through IrSO

IrSO actually supports HA mode, there have been proposals for more enhancements.

dependant on rabbitmq

RabbitMQ is not needed for scaling (and may even hurt), you probably meant a real database.

if we do one BMO per namespace and one ironic, we cannot have clusters spread over 3 DCs (nodes will be on a different namespaces/ironics, metal3 does not support this)

I don't quite get the concern. K8s namespaces don't have to match physical placement of pods on nodes.

[dtantsur]

metal3-io/ironic-standalone-operator#263 is another idea for more conductors in IrSO.

[lentzi90]

For problem 3 (meta/network-data). Yes it keeps coming up. I think everyone agrees that we should do something about it. We just haven't gotten to it yet. 😅

Focusing on remaining aspect. Let's see if I got it.

• You need one Ironic per DC
• BMO can only work with one Ironic, so this means 3 BMOs also
• Multiple BMOs basically means you need namespaced mode, or they would conflict
• We end up with at least one namespace per DC so that the BMHs belonging to it would be handled by the correct BMO/Ironic pair
• You want to create clusters across all 3 DCs => BMHs from all three namespaces
• ❌ CAPM3 cannot do this since it would mean Metal3Machines in different namespaces belonging to the same cluster

I am not aware of any current work addressing this, but I do see two paths forward.

1. The issue you identified already: One BMO controls multiple Ironic #1667
2. The CAPM3 multi-tenancy contract proposal. In its current form it would not allow this, but it could easily be extended. If implemented as written, it would already allow you to have the control-plane in one DC and a MachineDeployment in another. Let's say we extend and combine this with failure domains, then the control-plane could pick BMHs from all three DCs.

[nikParasyr]

( thanks for the replies and the info given during office hours )

Focusing on remaining aspect. Let's see if I got it.

• You need one Ironic per DC
• BMO can only work with one Ironic, so this means 3 BMOs also
• Multiple BMOs basically means you need namespaced mode, or they would conflict
• We end up with at least one namespace per DC so that the BMHs belonging to it would be handled by the correct BMO/Ironic pair
• You want to create clusters across all 3 DCs => BMHs from all three namespaces
• ❌ CAPM3 cannot do this since it would mean Metal3Machines in different namespaces belonging to the same cluster

This is indeed a very good summary, much easier to understand :)

So my "idea" is that if HostClaim is a seperate operator, then you can easily have multiple Ironic<->BMO pairs but one HostClaim operator giving a single point of entry for users/capm3. The multiple Ironic<->BMO pairs can be due to scaling, different DCs, security concerns for the provisioning network, ...
In this scenario, issues like One BMO controls multiple Ironic or even [RFE] Multiple Ironic instances per pod can be considered as out-of-scope, as in the "official scaling" architecture is just deploying more ironic<->BMO pairs, IrSO does not need to support multiple ironic instances per pod or multiple conductors, similarly BMO doesnt need to talk to multiple ironics. Instead its deploying multiple "simpler" ironics and BMOs and have the HostClaim operator work across them.

This was my idea while going through related open issues, this discussion and the related proposal. Not sure how valid it is as I lack knowledge/experience since we are still investigating metal3.

The CAPM3 multi-tenancy contract proposal. In its current form it would not allow this, but it could easily be extended. If implemented as written, it would already allow you to have the control-plane in one DC and a MachineDeployment in another. Let's say we extend and combine this with failure domains, then the control-plane could pick BMHs from all three DCs.

Indeed, i went through it and the Metal3Cluster and identityRef will allow us to tackle it, probably with 1 "child metal3 cluster" per DC and one parent cluster connecting to all of them.

[dtantsur]

When an explicit HostDeployPolicy is missing, does it make sense to assume that HostClaims from a namespace can use BareMetalHosts in the same namespace?