Elemental with OS Micro stuck at 'Configuring bootstrap node(s) waiting for agent to check in and apply initial plan'

[zetneteork]

Setup

• Rancher version: v2.10.3
• Rancher UI Extensions: Elemental 3.0.1
• Browser type & version:
• OS Micro 6.0 ISO v2.1.3-4.17
• Cluster - Elemental RKE2
  -- Kubernetes v1.31.7+rke2r1
  -- Kubernetes v1.28.15+rke2r1

Describe the bug

Node or nodes with OS Micro ready to use for deployment at Elemental does not spin up RKE2 Cluster and stuck at:

Configuring bootstrap node(s) elmental2-pool1-lmkqd-gmtxl: waiting for agent to check in and apply initial plan

To Reproduce

• Install Rancher v2.10.3
• Expose rancher externally over https
• Install a plugin Elemental
• Create cloud init config for nodes
• Generate ISO
• Deploy ISO to 4 nodes
• Create with Elemental cluster with one node, three or four.

Result
The node is stuck on bootstrap and never finishes the process.

Expected Result

Find the root cause of the bootstrap deployment issue.

Screenshots

Additional context

[kkaempf]

Please share your MachineRegistration

[kkaempf]

Can you log in to the client node ? Please check client logs.

[zetneteork]

Inventory of machine:

apiVersion: elemental.cattle.io/v1beta1
kind: MachineInventory
metadata:
  annotations:
    elemental.cattle.io/auth: tpm
    elemental.cattle.io/os.cpe-name: cpe:/o:suse:sl-micro:6.0
    elemental.cattle.io/os.id: sl-micro
    elemental.cattle.io/os.image: registry.suse.com/suse/sl-micro/6.0/baremetal-os-container:2.1.3-4.17
    elemental.cattle.io/os.name: SL-Micro
    elemental.cattle.io/os.pretty-name: SUSE Linux Micro 6.0
    elemental.cattle.io/os.version: '6.0'
    elemental.cattle.io/os.version-id: '6.0'
    elemental.cattle.io/reg-version: 1.6.6
    elemental.cattle.io/registration-ip: 192.168.78.130
  creationTimestamp: '2025-04-08T23:59:55Z'
  finalizers:
    - machineinventory.elemental.cattle.io
  generation: 1
  labels:
    CPUModel: Intel-R-Core-TM-i5-7500T-CPU-2-70GHz
    CPUVendor: GenuineIntel
    CPUVendorTotalCPUCores: '4'
    create-cluster-selector: WbRHFcaTnpzOUvHzEhmXtQZO
  managedFields:
    - apiVersion: elemental.cattle.io/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:labels:
            f:create-cluster-selector: {}
      manager: rancher
      operation: Update
      time: '2025-04-09T00:14:20Z'
    - apiVersion: elemental.cattle.io/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:elemental.cattle.io/auth: {}
            f:elemental.cattle.io/os.cpe-name: {}
            f:elemental.cattle.io/os.id: {}
            f:elemental.cattle.io/os.image: {}
            f:elemental.cattle.io/os.name: {}
            f:elemental.cattle.io/os.pretty-name: {}
            f:elemental.cattle.io/os.version: {}
            f:elemental.cattle.io/os.version-id: {}
            f:elemental.cattle.io/reg-version: {}
            f:elemental.cattle.io/registration-ip: {}
          f:finalizers:
            .: {}
            v:"machineinventory.elemental.cattle.io": {}
          f:labels:
            .: {}
            f:CPUModel: {}
            f:CPUVendor: {}
            f:CPUVendorTotalCPUCores: {}
          f:ownerReferences:
            .: {}
            k:{"uid":"b143f270-bb39-43e8-a8b7-45368a7e50d6"}: {}
        f:spec:
          .: {}
          f:tpmHash: {}
      manager: elemental-operator
      operation: Update
      time: '2025-04-09T00:14:42Z'
    - apiVersion: elemental.cattle.io/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:conditions: {}
          f:plan:
            .: {}
            f:checksum: {}
            f:secretRef: {}
            f:state: {}
      manager: elemental-operator
      operation: Update
      subresource: status
      time: '2025-04-09T00:14:43Z'
  name: m-f44165e0-fcaf-4188-a5be-9eeac0efcebd
  namespace: fleet-default
  ownerReferences:
    - apiVersion: elemental.cattle.io/v1beta1
      controller: true
      kind: MachineInventorySelector
      name: elemental-cluster-1-pool1-nwpwc-c4mmd
      uid: b143f270-bb39-43e8-a8b7-45368a7e50d6
  resourceVersion: '106527'
  uid: 6037afa4-f815-42e8-9ba6-8864fe8bf2bc
spec:
  tpmHash: 694611c3d44342bc16bbc89f5b654ee3e3abacd40cff1172959de94c1eec1257
status:
  conditions:
    - lastTransitionTime: '2025-04-09T00:02:31Z'
      message: plan successfully applied
      reason: PlanSuccessfullyApplied
      status: 'True'
      type: Ready
    - lastTransitionTime: '2025-04-09T00:14:43Z'
      message: Successfully adopted
      reason: SuccessfullyAdopted
      status: 'True'
      type: AdoptionReady
  plan:
    checksum: 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
    secretRef:
      name: m-f44165e0-fcaf-4188-a5be-9eeac0efcebd
      namespace: fleet-default
    state: Applied

[zetneteork]

Registration endpoint: configuration1

apiVersion: elemental.cattle.io/v1beta1
kind: MachineRegistration
metadata:
  creationTimestamp: '2025-04-08T17:56:25Z'
  generation: 2
  managedFields:
    - apiVersion: elemental.cattle.io/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          .: {}
          f:config:
            .: {}
            f:cloud-config:
              .: {}
              f:users: {}
            f:elemental:
              .: {}
              f:install:
                .: {}
                f:device-selector: {}
                f:reboot: {}
                f:snapshotter:
                  .: {}
                  f:type: {}
              f:reset:
                .: {}
                f:reboot: {}
                f:reset-oem: {}
                f:reset-persistent: {}
          f:machineInventoryLabels:
            .: {}
            f:CPUModel: {}
            f:CPUVendor: {}
            f:CPUVendorTotalCPUCores: {}
      manager: rancher
      operation: Update
      time: '2025-04-08T18:52:18Z'
    - apiVersion: elemental.cattle.io/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:conditions: {}
          f:registrationToken: {}
          f:registrationURL: {}
          f:serviceAccountRef: {}
      manager: elemental-operator
      operation: Update
      subresource: status
      time: '2025-04-09T22:16:24Z'
  name: configuration1
  namespace: fleet-default
  resourceVersion: '467762'
  uid: 4f74f23a-024b-47b0-85d3-d8b4c0b312d3
spec:
  config:
    cloud-config:
      users:
        - name: root
          passwd: root
    elemental:
      install:
        device-selector:
          - key: Name
            operator: In
            values:
              - /dev/sda
              - /dev/vda
              - /dev/nvme0
              - /dev/nvme0n1
              - /dev/nvme0n1p1
          - key: Size
            operator: Gt
            values:
              - 25Gi
        reboot: true
        snapshotter:
          type: btrfs
      reset:
        reboot: true
        reset-oem: true
        reset-persistent: true
  machineInventoryLabels:
    CPUModel: ${System Data/CPU/Model}
    CPUVendor: ${System Data/CPU/Vendor}
    CPUVendorTotalCPUCores: ${System Data/CPU/Total Cores}
status:
  conditions:
    - lastTransitionTime: '2025-04-09T22:16:24Z'
      message: ''
      reason: SuccessfullyCreated
      status: 'True'
      type: Ready
  registrationToken: mpbvbwfqlmtdldhd9msrclj4lhsn2sk6cgfqzrg4rlrrx2jxnbkhh8
  registrationURL: >-
    https://rancher.turris.opentree.cz/elemental/registration/mpbvbwfqlmtdldhd9msrclj4lhsn2sk6cgfqzrg4rlrrx2jxnbkhh8
  serviceAccountRef:
    kind: ServiceAccount
    name: configuration1
    namespace: fleet-default

[zetneteork]

The node terminal:

[kkaempf]

Hmm, looks like a certificate issue.
Is the Rancher instance running with a self-signed CA ?
Then you need to pass it as ca-cert in the MachineRegistration

[zetneteork]

elemental-system-agent.service

Apr 09 00:02:31 rancher-9450 systemd[1]: Started Elemental System Agent.
Apr 09 00:02:31 rancher-9450 elemental-system-agent[1431]: time="2025-04-09T00:02:31Z" level=info msg="Rancher System Agent version dev (HEAD) is starting"
Apr 09 00:02:31 rancher-9450 elemental-system-agent[1431]: time="2025-04-09T00:02:31Z" level=info msg="Using directory /var/lib/elemental/agent/work for work"
Apr 09 00:02:31 rancher-9450 elemental-system-agent[1431]: time="2025-04-09T00:02:31Z" level=info msg="Starting remote watch of plans"
Apr 09 00:02:31 rancher-9450 elemental-system-agent[1431]: time="2025-04-09T00:02:31Z" level=info msg="Initial connection to Kubernetes cluster failed with error Get \"https://rancher.turris.opentree.cz/k8s/clusters/local/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority, removing CA data and trying again"
Apr 09 00:02:31 rancher-9450 elemental-system-agent[1431]: time="2025-04-09T00:02:31Z" level=info msg="Starting /v1, Kind=Secret controller"
Apr 09 00:02:31 rancher-9450 elemental-system-agent[1431]: time="2025-04-09T00:02:31Z" level=info msg="Detected first start, force-applying one-time instruction set"
Apr 09 00:02:31 rancher-9450 elemental-system-agent[1431]: time="2025-04-09T00:02:31Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"
Apr 09 00:02:31 rancher-9450 elemental-system-agent[1431]: time="2025-04-09T00:02:31Z" level=info msg="[K8s] updated plan secret fleet-default/m-f44165e0-fcaf-4188-a5be-9eeac0efcebd with feedback"
Apr 09 00:14:47 rancher-9450 elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e"
Apr 09 00:14:47 rancher-9450 elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/elemental/agent/work/20250409-001447/6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_0"
Apr 09 00:14:47 rancher-9450 elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[Applyinator] Running command: hostnamectl [set-hostname --transient m-f44165e0-fcaf-4188-a5be-9eeac0efcebd]"
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[Applyinator] Command hostnamectl [set-hostname --transient m-f44165e0-fcaf-4188-a5be-9eeac0efcebd] finished with err: <nil> and exit code: 0"
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/elemental/agent/work/20250409-001447/6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1"
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[Applyinator] Running command: bash [-c [ -f /var/lib/rancher/bootstrap_done ] || /var/lib/rancher/bootstrap.sh && touch /var/lib/rancher/bootstrap_done]"
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stdout]: [INFO]  --no-roles flag passed, unsetting all other requested roles"
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stdout]: [INFO]  CA strict verification is set to true"
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stdout]: [INFO]  Using default agent configuration directory /etc/rancher/agent"
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stdout]: [INFO]  Using default agent var directory /var/lib/rancher/agent"
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: [WARN]  /usr/local is read-only or a mount point; installing to /opt/rancher-system-agent"
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stdout]: [INFO]  Successfully downloaded CA certificate"
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stdout]: [INFO]  Value from https://rancher.turris.opentree.cz/cacerts is an x509 certificate"
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: curl: (60) SSL certificate problem: unable to get local issuer certificate"
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: More details here: https://curl.se/docs/sslcerts.html"
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: "
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: curl failed to verify the legitimacy of the server and therefore could not"
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: establish a secure connection to it. To learn more about this situation and"
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: how to fix it, please visit the web page mentioned above."
Apr 09 00:14:47 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:47Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: [ERROR]  000 received while testing Rancher connection. Sleeping for 5 seconds and trying again"
Apr 09 00:14:52 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:52Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: curl: (60) SSL certificate problem: unable to get local issuer certificate"
Apr 09 00:14:52 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:52Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: More details here: https://curl.se/docs/sslcerts.html"
Apr 09 00:14:52 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:52Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: "
Apr 09 00:14:52 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:52Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: curl failed to verify the legitimacy of the server and therefore could not"
Apr 09 00:14:52 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:52Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: establish a secure connection to it. To learn more about this situation and"
Apr 09 00:14:52 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:52Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: how to fix it, please visit the web page mentioned above."
Apr 09 00:14:52 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:52Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: [ERROR]  000 received while testing Rancher connection. Sleeping for 5 seconds and trying again"
Apr 09 00:14:57 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:57Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: curl: (60) SSL certificate problem: unable to get local issuer certificate"
Apr 09 00:14:57 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:57Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: More details here: https://curl.se/docs/sslcerts.html"
Apr 09 00:14:57 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:57Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: "
Apr 09 00:14:57 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:57Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: curl failed to verify the legitimacy of the server and therefore could not"
Apr 09 00:14:57 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:57Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: establish a secure connection to it. To learn more about this situation and"
Apr 09 00:14:57 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:57Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: how to fix it, please visit the web page mentioned above."
Apr 09 00:14:57 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:14:57Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: [ERROR]  000 received while testing Rancher connection. Sleeping for 5 seconds and trying again"
Apr 09 00:15:02 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:15:02Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: curl: (60) SSL certificate problem: unable to get local issuer certificate"
Apr 09 00:15:02 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:15:02Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: More details here: https://curl.se/docs/sslcerts.html"
Apr 09 00:15:02 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:15:02Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: "
Apr 09 00:15:02 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:15:02Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: curl failed to verify the legitimacy of the server and therefore could not"
Apr 09 00:15:02 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:15:02Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: establish a secure connection to it. To learn more about this situation and"
Apr 09 00:15:02 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:15:02Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: how to fix it, please visit the web page mentioned above."
Apr 09 00:15:02 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:15:02Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: [ERROR]  000 received while testing Rancher connection. Sleeping for 5 seconds and trying again"
Apr 09 00:15:07 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:15:07Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: curl: (60) SSL certificate problem: unable to get local issuer certificate"
Apr 09 00:15:07 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:15:07Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: More details here: https://curl.se/docs/sslcerts.html"
Apr 09 00:15:07 m-f44165e0-fcaf-4188-a5be-9eeac0efcebd elemental-system-agent[1431]: time="2025-04-09T00:15:07Z" level=info msg="[6f7d37c82a737bac4a869ad073ee9a45480cffe324ca48659c8e6d1b9493d94e_1:stderr]: "

[zetneteork]

Just so you know, the rancher is exposed externally over Nginx with signed SSL, but it was not changed with CA in the rancher.
Nodes use the external endpoint, not a direct path to the internal rancher.
It sounds like there is a mishmash of two certificates: the external on Nginx and the Rancher internal.

[zetneteork]

I can confirm the POC when the new Rancher instance sits internally next to the nodes on the same subnetwork. The cloud-init file has internal endpoints and a Rancher CA. The cluster bootstraps without hassle.

But when I reproduce the same procedure/config on production Rancher Elemental instance behind reverse proxy. The issue with CA appears. THE internal CA had been specified in cloud-init file. But in logs elemental-system-agent.service there is a record the authority failed and removing CA data.

rancher-4846:~ # journalctl -f -u elemental-system-agent.service 
Apr 10 21:54:30 rancher-4846 systemd[1]: Started Elemental System Agent.
Apr 10 21:54:30 rancher-4846 elemental-system-agent[1502]: time="2025-04-10T21:54:30Z" level=info msg="Rancher System Agent version dev (HEAD) is starting"
Apr 10 21:54:30 rancher-4846 elemental-system-agent[1502]: time="2025-04-10T21:54:30Z" level=info msg="Using directory /var/lib/elemental/agent/work for work"
Apr 10 21:54:30 rancher-4846 elemental-system-agent[1502]: time="2025-04-10T21:54:30Z" level=info msg="Starting remote watch of plans"
Apr 10 21:54:30 rancher-4846 elemental-system-agent[1502]: time="2025-04-10T21:54:30Z" level=info msg="Initial connection to Kubernetes cluster failed with error Get \"https://rancher.turris.opentree.cz/k8s/clusters/local/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority, removing CA data and trying again"
Apr 10 21:54:31 rancher-4846 elemental-system-agent[1502]: time="2025-04-10T21:54:31Z" level=info msg="Starting /v1, Kind=Secret controller"
Apr 10 21:54:31 rancher-4846 elemental-system-agent[1502]: time="2025-04-10T21:54:31Z" level=info msg="Detected first start, force-applying one-time instruction set"
Apr 10 21:54:31 rancher-4846 elemental-system-agent[1502]: time="2025-04-10T21:54:31Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"
Apr 10 21:54:31 rancher-4846 elemental-system-agent[1502]: time="2025-04-10T21:54:31Z" level=info msg="[K8s] updated plan secret fleet-default/m-8db1d4d8-4239-4cb3-8c68-ae37eedea050 with feedback"

[csdoapaj]

I'm having this same issue. The rancher is exposed through NGINX Proxy Manager and is using Let's Encrypt signed certificate. I have also tried to put the endpoint behind a Cloudflare proxy, but both method return the same error as @zetneteork is having (curl failed to verify the legitimacy of the server).