design: Extract Micro-OS from PV storage #217
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Thank you for your contribution! Please make sure to review our Contributing Guide.
osinstom
requested review from
Andrea-Campanella,
se-chris-thach,
mdbalvin,
rad-szulim,
charlesmcchan,
yi-tseng-intel,
ashridatta,
teone,
callumnobleintel,
daniele-moro,
SeanCondon and
pierventre
as code owners
osinstom
requested review from
cgoea,
emmakenny,
PalashGoelIntel,
palade,
pperycz,
ltalarcz,
jakubsikorski,
sys-orch-approve,
ahalimx86,
johnoloughlin and
garyloug
as code owners
…latform/edge-manageability-framework into proposal-eim-decouple-hookos
|
|
||
| Last updated: 24.04.2025 | ||
|
|
||
| ## Abstract |
There was a problem hiding this comment.
another benefit to list is also the startup time of DKAM - given all the issues we faced with the readiness probe implementation
| DKAM should not do any of the steps above. Therefore, the main challenge to solve is how to provide CA certs and environment variables to the Micro-OS image downloaded from an external source, | ||
| without sacrificing the security. | ||
|
|
||
| ### Solution |
There was a problem hiding this comment.
are the last 2 steps impacted if we move from hookos to emt-tinker ?
|
|
||
| dkam->>pa: Store signed iPXE EFI binary | ||
|
|
||
| user->>+rs: Download RS certificate & Micro-OS DER key |
There was a problem hiding this comment.
RS Certrificate might not be required use opt if the certificte is ss only
There was a problem hiding this comment.
I'm not sure about this. The signing of Micro-OS is used to add an additional security level when downloading via iPXE, which is only secured by HTTPS with one-way TLS.
There was a problem hiding this comment.
The iPXE client is not really rich like other clients to carry well known CAs. So, irrespective of the source, we might need to add the trust chain for TLS.
| Steps 4-8: At deployment time, DKAM reads the orchestrator CA certificate and cluster-wide variables, | ||
| and generates two artifacts: 1) the signed Micro-OS bundle, and 2) the signed iPXE script that downloads | ||
| the Micro-OS image and the Micro-OS bundle. | ||
| We assume that both artifacts are signed with the same key that is generated by DKAM. | ||
| Note that iPXE includes standard public certificates (like Microsoft or AWS) so we don't need to embed | ||
| these certificates, as long as it's not trusted by iPXE or a self-signed certificate that is used by a local storage. |
There was a problem hiding this comment.
can we skip certificate bundling if it has a public root ? Do we need to handle certificate rotation?
| - the orchestrator CA certificate (uploaded now) | ||
| - the secure key used to sign iPXE and orchestrator bundle (uploaded now) | ||
| - the secure key used to sign EMT (uploaded now) - only if EMT is used | ||
| - the secure key used to sign Micro-OS image (new). This key is downloaded from the external storage. |
There was a problem hiding this comment.
| - the secure key used to sign Micro-OS image (new). This key is downloaded from the external storage. | |
| - the secure key used to sign Micro-OS image (uploaded now). This key is downloaded from the external storage (new). |
There was a problem hiding this comment.
Are you thinking of the DER key that is currently used to sign EMT images? I'm not sure if we can use the same one.
| **Phase 1**: Work on a PoC by storing the current non-curated HookOS on the RS file server. The HookOS image is currently distributed as OCI artifact. | ||
| The PoC would require to store it as non-OCI on the S3 bucket. The PoC should give an answer if the solution described in this proposal is feasible. |
There was a problem hiding this comment.
can we keep it as OCI and use services as ECR ?
There was a problem hiding this comment.
Nope, we cannot download OCI scripts with iPXE
| 1. How to handle authentication to external storage? It must be an HTTPS endpoint and TLS certificates must be embedded into | ||
| the iPXE binary, but if an external storage requires some additional authentication (e.g., JWT like the internal RS) | ||
| iPXE is not able to support that. | ||
| 2. The design doesn't assume that the trust CA certifcates can be rotated/refreshed. |
There was a problem hiding this comment.
please elaborate more on .2
|
|
||
| note over en: EN powered on | ||
|
|
||
| en->>pa: Download iPXE binary |
There was a problem hiding this comment.
the signature verification key and the CA cert chain for orch still needs to be configured into the EN.
There was a problem hiding this comment.
@AdithyaBaglody, please help review if the SB flow remains intact.
| environment files that are needed for Micro-OS to work with the given Edge Orchestrator instance. The files package will be called a "Micro-OS bundle" hereinafter. | ||
| - The Micro-OS bundle is a signed archive. It is signed by the same keychain as iPXE script. | ||
| - iPXE script along with the Micro-OS bundle are still stored on the orchestrator PV. Since these are small files, the provisioning time would not be impacted. | ||
| - ENs will download the Micro-OS bundle and the Micro-OS image during the iPXE phase and inject the Micro-OS bundle to the Micro-OS image (via `initrd`), similarly to how the initramfs is loaded. |
There was a problem hiding this comment.
IHMO this is not going to work. Did someone do a TR??
There was a problem hiding this comment.
I did a quick PoC and I was able to inject a tarball containing a cert file along with the initramfs
There was a problem hiding this comment.
This is orchestrator CA certificate that is needed for Micro-OS services (device discovery, tink worker, ...) to connect to orchestrator
Can you please elaborate on the security risks so that we can think how we can fix them? |
@osinstom This proposal talks about signing the image on the edge. We have no way to trust the edge node at that time. You intend to extend the cpio archive of the initramfs by using a signed archive. This would create a new archive, effectively it will become an unsigned image. grub can only verify the complete image. If you are creating a cpio archive on the fly, there is no way to sign it and make it trusted. |
osinstom
changed the title
ajaythakurintel
added
the
Proposal
Description
This PR brings the design proposal to extract the Micro-OS image from PV storage in order to improve EN provisioning time.
Any Newly Introduced Dependencies
N/A
How Has This Been Tested?
N/A
Checklist: