design: Scale provisioning of EMT-S nodes #250
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Thank you for your contribution! Please make sure to review our Contributing Guide.
| The provisioning of EMT-S at scale will be driven by a local orchestrator instance that will be slimmed down to include only necessary components. In a nutshell, | ||
| it will consist of: | ||
|
|
||
| - Foundational Platform Services that are required to deploy and run the orchestrator instance. |
There was a problem hiding this comment.
Argo? How do we handle the deployment of the orchestrator?
There was a problem hiding this comment.
yes, let's assume for now this is deployed via the EIM-Standalone profile you are working on @daniele-moro so effectively we leverage the orchestrator on prem installer.
There was a problem hiding this comment.
I strongly advise against creating a dedicated SKU for this, like we previously did in a previous release.
In a parallel work effort, I proposed that we make Orchestrator configurable at initial deployment time AND runtime.
In this effort, I propose that instead of a separate variant of Orchestrator, we utilize this proposed dynamically configurable Orchestrator and explicitly configure it to disable unnecessary features).
There was a problem hiding this comment.
yes, no variant is intended, a profile is what we are going to use for now, further optimizations can be achieved in the future.
There was a problem hiding this comment.
There is already a way to do that via Argo CD profile, but it is not as straightforward as it can be. I will make "the ability to enable/disable components easily" an explicit requirement for the new installer that we are working on.
There was a problem hiding this comment.
@charlesmcchan @se-chris-thach I want to test EIM-only/standalone profile, can I start creating a profile or should I use a different approach?
| The provisioning of EMT-S at scale will be driven by a local orchestrator instance that will be slimmed down to include only necessary components. In a nutshell, | ||
| it will consist of: | ||
|
|
||
| - Foundational Platform Services that are required to deploy and run the orchestrator instance. |
There was a problem hiding this comment.
yes, let's assume for now this is deployed via the EIM-Standalone profile you are working on @daniele-moro so effectively we leverage the orchestrator on prem installer.
ajaythakurintel
added
the
Proposal
osinstom
requested review from
rad-szulim,
yi-tseng-intel,
ashridatta,
teone and
callumnobleintel
as code owners
| - Deploy the provisioning service on the local network that support PXE Boot (BIOS/UEFI with DHCP + TFTP) boot and iPXE with HTTPs. | ||
| - Have a UX to pre-register BareMetal edge nodes using Serial number or UUID or MAC address. | ||
| - Provision different OS profiles to different edge nodes selected based on Serial number or UUID or MAC address. | ||
| - Provision default OS when a device on the LAN boots over PXE and is not pre-registered. |
There was a problem hiding this comment.
What is the meaning of default OS here, Are we installing supporting the Ubuntu also in EMT-S?
There was a problem hiding this comment.
no, but there can be multiple versions of EMT-S, we will have. default one in this case. Aslo this can be extended to other immutable ISO images.
| - Deploy the provisioning service on the local network that support PXE Boot (BIOS/UEFI with DHCP + TFTP) boot and iPXE with HTTPs. | ||
| - Have a UX to pre-register BareMetal edge nodes using Serial number or UUID or MAC address. | ||
| - Provision different OS profiles to different edge nodes selected based on Serial number or UUID or MAC address. | ||
| - Provision default OS when a device on the LAN boots over PXE and is not pre-registered. |
There was a problem hiding this comment.
no, but there can be multiple versions of EMT-S, we will have. default one in this case. Aslo this can be extended to other immutable ISO images.
| In other words, the only difference between the new PXE-based boot and HTTP-based boot is how the OS provisioning is triggered. The subsequent workflow remains the same - | ||
| it leverages Micro-OS, device discovery and Tinkerbell workflow to complete OS provisioning. | ||
|
|
||
| **NOTE1**: Customers should provide their own local DHCP server for dynamic IP address assignment. |
There was a problem hiding this comment.
this needs ample documentation
There was a problem hiding this comment.
also I can't see it in the diagram below, even though the DHCP request is intercepted by SMEE, we ought to clarify and add it in the diagram.
|
|
||
| **NOTE1**: Customers should provide their own local DHCP server for dynamic IP address assignment. | ||
|
|
||
| **NOTE2**: The workflow assumes that the EIM Standalone is deployed locally on customers' premises and ENs have direct connectivity with EIM services. |
There was a problem hiding this comment.
shall we say something about the interaction with external (not managed by us) DHCP servers?
There was a problem hiding this comment.
I'll add a diagram with a sample network topology but we're not really making any changes to network requirements. We had been relying on customer-managed DHCP server for long time already
There was a problem hiding this comment.
I believe we need to tell the user to change few things in their dhcp server too? Dont we need the next-server option?
There was a problem hiding this comment.
next-server is provided by SMEE's DHCP server which is configured in proxy DHCP mode
There was a problem hiding this comment.
If we enforce any change to DHCP servers, this design proposal wouldn't make sense to me
There was a problem hiding this comment.
ok if you believe is not necessary - but this is what I found to make them coexist:
- A PXE client sends a DHCP request (broadcast).
- The external DHCP server responds with: An IP address, The next-server (SMEE server IP), The filename (iPXE binary).
- Tinkerbell SMEE (in proxy DHCP mode) responds with PXE-specific options (e.g., boot filename and TFTP/HTTP server details).
- The PXE client downloads the iPXE binary and executes the boot script provided by Boots.
Not sure if this is suggested for network race conditions
There was a problem hiding this comment.
let me do some more experiments to confirm once I have my target on-prem environment running
|
|
||
| **NOTE2**: The workflow assumes that the EIM Standalone is deployed locally on customers' premises and ENs have direct connectivity with EIM services. | ||
|
|
||
| The high-level PXE-based provisioning workflow is as follows: |
There was a problem hiding this comment.
also in the sequence say something about external DHCP interactions. Or create a small paragraph with the assumptions and the checklist user should comply:
- Boot works in proxy dhcp mode only
- External DHCP server uses DHCP options....
|
|
||
| 3. **Expose Tinkerbell SMEE's DHCP/TFTP server via K8s External IP** | ||
|
|
||
| Tinkerbell SMEE's DHCP/TFTP server must be reachable from a local, on-prem L2 network to handle DHCP/TFTP requests. This requires modifications to FPS services. |
There was a problem hiding this comment.
is it not enough use hostNetwork for port 67-69? Which modifications do we expect?
There was a problem hiding this comment.
Perhaps, I'll work on another PoC to make SMEE work on the on-prem deployment
| #### Alternative workflow with managed EMF | ||
|
|
||
| In some cases it may be desirable for users to scale EMT-S provisioning via legacy PXE, but the EMF (or EIM) is deployed in a central location. | ||
| This solution also enables the PXE boot when the entire EMF (or EIM Standalone) is deployed as remote, cloud-based, managed solution (not on-prem). | ||
|
|
||
| In this alternative there is only a small local piece of EIM (called "EIM-local" hereinafter) deployed on site (via installation script provided by EIM team). | ||
| The EIM-local assists in the initial PXE boot and make possible for ENs to initiate boot via PXE. Once booted into Micro-OS, the provisioning process is taken over by the cloud-based orchestrator. | ||
|
|
||
| The EIM-local consists of the following components: | ||
| 1. **Standalone Tinkerbell SMEE** providing DHCP/TFTP server to support legacy PXE boot. | ||
| 2. **Local HTTP server** (e..g, Nginx) storing mirrors of `boot.ipxe` and Micro-OS image. | ||
| 3. (OPTIONAL) **K8s cluster with MetalLB extension** to make Standalone SMEE's DHCP/TFTP servers accessible from a local network. Only needed if EIM-local is deployed on top of Kubernetes. | ||
| Note that the EIM-local can also be deployed as standalone OS services or Docker containers with `--network=host`. | ||
|
|
||
| **NOTE1:** Local HTTP server providing `boot.ipxe` and Micro-OS image is needed to overcome HTTPS issue as | ||
| SMEE's built-in iPXE doesn't include EMF's CA certificate. This alternative design assumes no modifications to Tinkerbell SMEE, for simplicity. |
There was a problem hiding this comment.
Honestly, I argue that improving DKAM seems to me a better option here - we basically empower DKAM to curate for any arbitrary orchestrator (PROVISIONING_URL, ORCH_CA).
Of course I prefer the 2nd option below
| - No Kyverno for policies | ||
| - No UI deployed, the EN pre-registration is done via Bulk Import Tool, API or CLI tool. | ||
| - Limited observability stack that should only provide logs. No alerting, SRE, Prometheus. | ||
| - Abandon HA requirements - EIM-S is primarily a single-node, on-prem deployment. Any data loss can be backed by hardware-level redundancy. Think of EIM-S as Rancher Desktop or Virtualbox. |
There was a problem hiding this comment.
we dont have any HA mechanism when deployed locally
There was a problem hiding this comment.
Even in the observability stack ?
| 2. **Expose Provisioning Nginx via HTTP for on-prem EIM standalone only** | ||
|
|
||
| Currently, Provisioning Nginx is deployed behind HTTPS and the iPXE binary must be built with orchestrator's CA certificate for successful handshake. | ||
| Tinkerbell SMEE obviously doesn't include the EMF's CA certificate. Therefore, any communication over HTTPS is not feasible. |
There was a problem hiding this comment.
Good to check with SAFE team once to go ahead with HTTP provided local network is protected with firewall.
There was a problem hiding this comment.
Had a chat with @haribabug. He will provide his feedback on this ADR. I'm going to expand the network requirements section. Basically we rely on being in the local L2 network that is isolated (without any inbound connections from outside)
There was a problem hiding this comment.
Is it worth using Smee if it is going to cause a divergence in security policy and deployment process like this?
There are many other off-the-shelf DHCP servers out there (dnsmasq, udhcpd, Kea, etc.) that could fill this gap, or we could see if Smee could be modified.
Co-authored-by: Andrea Campanella <[email protected]>
Co-authored-by: Andrea Campanella <[email protected]>
Co-authored-by: Andrea Campanella <[email protected]>
Co-authored-by: Andrea Campanella <[email protected]>
…ge-manageability-framework into sen-scale-proposal
| - Deploy the provisioning service on the local network that support PXE Boot (BIOS/UEFI with DHCP + TFTP) boot and iPXE with HTTPs. | ||
| - Have a UX to pre-register BareMetal edge nodes using Serial number or UUID. | ||
| - Provision different OS profiles to different edge nodes selected based on Serial number or UUID. | ||
| - Provision default OS when a device on the LAN boots over PXE and is not pre-registered. |
There was a problem hiding this comment.
Clarify that the "default OS when not pre-registered" is different from the "bottom up" registration workflow - the EN would not be registered at the end of this process, just having the default OS installed.
Maybe call this "Anonymous OS install" or something to specify that it isn't registered.
Also, in this case there non-authenticate OS install would not send logs to EMF, correct?
There was a problem hiding this comment.
@krishnajs is this requirements still valid with pre-registration flow? I think we should remove it.
| In other words, the only difference between the new PXE-based boot and HTTP-based boot is how the OS provisioning is triggered. The subsequent workflow remains the same - | ||
| it leverages Micro-OS, device discovery and Tinkerbell workflow to complete OS provisioning. | ||
|
|
||
| **NOTE1**: Customers should provide their own local DHCP server for dynamic IP address assignment. |
There was a problem hiding this comment.
What configuration will be needed to perform this? Typically any DHCP servers requires defining a subnet (or multiple subnets, which can have different config), so we would need the user to provide this information.
Ideally we could reconfigure this if IP ranges change, without requiring a reinstall.
| 2. **Expose Provisioning Nginx via HTTP for on-prem EIM standalone only** | ||
|
|
||
| Currently, Provisioning Nginx is deployed behind HTTPS and the iPXE binary must be built with orchestrator's CA certificate for successful handshake. | ||
| Tinkerbell SMEE obviously doesn't include the EMF's CA certificate. Therefore, any communication over HTTPS is not feasible. |
There was a problem hiding this comment.
Is it worth using Smee if it is going to cause a divergence in security policy and deployment process like this?
There are many other off-the-shelf DHCP servers out there (dnsmasq, udhcpd, Kea, etc.) that could fill this gap, or we could see if Smee could be modified.
| note over microos,inv: OS provisioning continues with the standard flow, through Micro-OS and Tinkerbell workflow | ||
| ``` | ||
|
|
||
| 1. Users perform standard EN registration via UI or Bulk Import Tool. For each EN we can define OS profile and additonal settings (e.g., site, local account). |
There was a problem hiding this comment.
Users perform standard EN registration via UI
The registration flow is not the standard one as whether the Cluster is installed or not entirely depend on the selection of the OS profile (via cloud-init). We should make sure that the user only has the <project_id>_ir/irw roles so that the UI can hide the Cluster creation steps from the registration/provisioning flow.
| 1) **Support legacy PXE boot to scale EMT-S provisioning** - the EIM will be extended with a local DHCP/TFTP server that helps initiate OS provisioning via legacy PXE boot. | ||
| The legacy PXE boot uses the devices' PXE firmware to bootstrap into iPXE, which then continues the EN provisioning process. | ||
| 2) **Extend usage of Platform Bundle to be compatible with USB-based EMT-S** - to be consistent with USB-based EMT-S, the EIM will consume Platform Bundle that contains all the scripts and files to install standalone K8s cluster and other customizations. | ||
| 3) **Use EIM standalone deployment** - the proposed solution involves deploying a specific EMF profile, referred to as EIM standalone profile. |
There was a problem hiding this comment.
In 3.1 we deploy the whole orchestrator in the OXM flow reducing the resource requirements (TODO: measure resources needed for 100 within 5 minutes, also unlimited resources as good as we can get numbers.) and use roles NOT to show App and CO elements in UI, default MT user, projects and org created automatically via a module during installation, with a PoC of Infra-only profile of EMF with "extreme" simplification but still using the orchestrator as a whole, not part of 3.1. Reason behind decision to keep the whole of EMF is limited resource/performance/time and UX gain compared to the effort we need to invest to maintain. PoC of Daniele will be revisited for a possible 3.2 implementation.
as per the discussion please move this lower in the ADR and advertise as PoC.
| Also, an alternative to using SMEE was considered to avoid divergence in security policies (HTTP vs. HTTPS for Provisioning Nginx). | ||
| This direction requires more development effort, but is still left as future improvement. It's further elaborated in a separate [design proposal](https://github.com/open-edge-platform/edge-manageability-framework/pull/309). | ||
|
|
||
| ## Affected components and Teams |
There was a problem hiding this comment.
@daniele-moro to add the role to avoid User accessing App and CO.
Description
This PR adds design proposal for Scale provisioning of EMT-S edge node supporting OXM workflow
Any Newly Introduced Dependencies
N/A
How Has This Been Tested?
N/A
Checklist: