Is use of talos.config.early possible with OAuth2 machine config download (talos.config)?

[ghouscht]

Hey,
So far we used a Talos custom build to include CA certificates in Talos. The CA certificates are needed for the OAuth2 authentication flow (Machine Configuration OAuth2 Authentication) to work. I was hoping that with Talos 1.12 we can ditch out custom build and instead use the new kernel parameter talos.config.early to load the CA certificates using a TrustedRootsConfig document. However, it seems this does not work. Is that intentional?

Example agrs

talos.config.early=CG== # zstd compressed base64 encoded `TrustedRootsConfig` document
talos.config=https://api.example.com/machineconfig
talos.config.oauth.client_id=42
talos.config.oauth.extra_variable=uuid
talos.config.oauth.extra_variable=mac

Result: Talos applies the early machine config but never starts the OAuth2 device flow and falls back to maintenance mode.

Please note: The custom CA certificates are needed for the OAuth2 device flow to work because the endpoint in talos.config is using a cert signed by this custom CA.

[ghouscht]

btw. I forgot to mention that platform is set to metal when generating the ISO using imager (talos.platform=metal kernel arg is present). According to https://docs.siderolabs.com/talos/v1.12/configure-your-talos-cluster/system-configuration/acquire#platform-configuration I would assume this is supposed to work.

[frezbo]

If you're using imager follow the embedded config https://docs.siderolabs.com/talos/v1.12/platform-specific-installations/boot-assets#example-adding-embedded-machine-configuration-with-imager

[frezbo]

and https://docs.siderolabs.com/talos/v1.12/reference/kernel#talos-config-early-and-talos-config-inline

[ghouscht]

This is what I do. When I boot the generated ISO I can see all the kernel params I added by pressing e in the boot loader. However, it seems that everything after talos.config.early is ignored. For example we also add net.ifnames=0, if put before the talos.config.early it is applied, but if its put after, it is not applied which is weird.

I'm also aware of the 4096 byte limit of the kernel cmdline, the TrustedRoots config itself if 2557 bytes:

$ cat ca.yaml | zstd --compress --ultra -22 | base64 -w 0 | wc -c

So I don't think we exceed the limit.

[frezbo]

you're better off using embedded config here since it would get loaded way early https://docs.siderolabs.com/talos/v1.12/configure-your-talos-cluster/system-configuration/acquire#methods-of-acquiring-machine-configuration I meant to share this

[ghouscht]

Yeah I can try that as well. To my understanding it shouldn't make a difference because the talos.config.early is supposed to be loaded before the platform configuration so I was expecting this to work as well.

[frezbo]

it should, but would need the console logs to debug more

[ghouscht]

I tried to create an ISO using this example. However, it fails with the following error:

skipped pulling overlay (no overlay)
profile ready:
arch: arm64
platform: metal
secureboot: false
version: v1.12.0
customization:
  embeddedMachineConfiguration: |
    # machine.yaml
    apiVersion: v1alpha1
    kind: WatchdogTimerConfig
    device: /dev/watchdog0
    timeout: 5m
input:
  kernel:
    path: /usr/install/arm64/vmlinuz
  initramfs:
    path: /usr/install/arm64/initramfs.xz
  sdStub:
    path: /usr/install/arm64/systemd-stub.efi
  sdBoot:
    path: /usr/install/arm64/systemd-boot.efi
  baseInstaller:
    imageRef: ghcr.io/siderolabs/installer-base:v1.12.0
  systemExtensions:
    - imageRef: ""
      tarballPath: /tmp/imager3062674697/embedded-config.tar
output:
  kind: iso
  isoOptions:
    sdBootEnrollKeys: if-safe
    bootloader: sd-boot
  outFormat: raw
◱ error setting selinux xattr for "/tmp/imager3062674697/extensions/0/rootfs/usr/local/etc/talos/config.yaml": xattr.LSet /tmp/imager3062674697/extensions/0/rootfs/usr/local/etc/talos/config.yaml security.selinux: operation not permitted
Error: error setting selinux xattr for "/tmp/imager3062674697/extensions/0/rootfs/usr/local/etc/talos/config.yaml": xattr.LSet /tmp/imager3062674697/extensions/0/rootfs/usr/local/etc/talos/config.yaml security.selinux: operation not permitted

(same with 1.12.1)

I found this check in the codebase for detecting if selinux is enabled/disabled. Apparently this is also trying to detect if it is run in a container here. This checks for presence of a file /usr/etc/in-container which I can't find in the imager container. Not sure if said file should be present or not...

Anyway the example seems to only work with the --privileged docker run flag which I thought is no longer needed to run the imager container?

[frezbo]

privileged is still needed, only talos 1.13 images will be rootless