dynamic-controller error: “failed to add resource to applyset … is not json marshallable” (unsupported type map[ref.al[]ref.Val)

[kahirokunn]

Description

Observed Behavior:
After applying the ResourceGraphDefinition and then creating an instance, the kro dynamic-controller keeps requeuing and logs this error. The Cluster resource is not reconciled successfully.

Expected Behavior:
The instance should reconcile successfully and create/update the referenced cluster.x-k8s.io/v1beta1 Cluster resource.

Reproduction Steps (ResourceGraphDefinition and Instance):

1. Apply this ResourceGraphDefinition:

apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: k0smotrondockercluster.cluster.example.com
spec:
  schema:
    group: cluster.example.com
    apiVersion: v1alpha1
    kind: K0smotronDockerCluster
    spec:
      # ClusterClass
      clusterclassName: string | required=true

      # k0s version (e.g. v1.34.2+k0s.0)
      k0sVersion: string | required=true pattern="^v[0-9]+\.[0-9]+\.[0-9]+\+k0s\.[0-9]+$"

      # Replicas
      controlPlaneReplicas: integer | required=true
      machineDeploymentReplicas: integer | required=true

      # Cluster network configuration (with defaults)
      podCIDR: string | default="192.168.0.0/16" pattern="^([0-9]{1,3}\\.){3}[0-9]{1,3}/[0-9]{1,2}$"
      serviceCIDR: string | default="10.128.0.0/12" pattern="^([0-9]{1,3}\\.){3}[0-9]{1,3}/[0-9]{1,2}$"

      # Base cluster kubeconfig secret reference
      baseCluster:
        kubeconfigRef:
          name: string | required=true
          namespace: string | required=true

      template:
        metadata:
          annotations: "map[string]string | default={}"
          labels: "map[string]string | required=true"

    status:
      clusterStatus:
        conditions: ${cluster.status.conditions}
        controlPlaneReady: ${cluster.status.controlPlaneReady}
        infrastructureReady: ${cluster.status.infrastructureReady}
        observedGeneration: ${cluster.status.observedGeneration}
        phase: ${cluster.status.phase}
        v1beta2:
          conditions: ${cluster.status.v1beta2.conditions}

  resources:
    - id: clusterClass
      externalRef:
        apiVersion: cluster.x-k8s.io/v1beta1
        kind: ClusterClass
        metadata:
          name: ${schema.spec.clusterclassName}
          namespace: ${schema.metadata.namespace}
      readyWhen:
        - ${clusterClass.status.conditions.all(x, x.status == "True")}

    - id: cluster
      template:
        apiVersion: cluster.x-k8s.io/v1beta1
        kind: Cluster
        metadata:
          name: ${schema.metadata.name}
          namespace: ${schema.metadata.namespace}
          annotations: ${schema.spec.template.metadata.annotations}
          labels: ${schema.spec.template.metadata.labels}
        spec:
          clusterNetwork:
            pods:
              cidrBlocks:
                - ${schema.spec.podCIDR}
            serviceDomain: cluster.local
            services:
              cidrBlocks:
                - ${schema.spec.serviceCIDR}
          topology:
            class: ${clusterClass.metadata.name}
            version: ${schema.spec.k0sVersion}
            controlPlane:
              replicas: ${schema.spec.controlPlaneReplicas}
            workers:
              machineDeployments:
                - class: default-worker
                  name: md-0
                  replicas: ${schema.spec.machineDeploymentReplicas}
            variables: '${has(schema.spec.baseCluster) && has(schema.spec.baseCluster.kubeconfigRef) ? [{"name":"kubeconfigRef","value": schema.spec.baseCluster.kubeconfigRef}] : []}'

2. Apply this instance:

apiVersion: cluster.appthrust.io/v1alpha1
kind: K0smotronDockerCluster
metadata:
  name: test-cluster
  namespace: test-cluster
spec:
  baseCluster:
    kubeconfigRef:
      name: base
      namespace: k0smotron-test-plane-system
  clusterclassName: k0smotron-docker-md-v1
  controlPlaneReplicas: 1
  k0sVersion: v1.34.2+k0s.0
  machineDeploymentReplicas: 1
  podCIDR: 192.168.0.0/16
  serviceCIDR: 10.128.0.0/12
  template:
    metadata:
      annotations: {}
      labels: {}

3. Check dynamic-controller logs.

Versions:

• kro version: v0.7.1
• Kubernetes Version (kubectl version): v1.34.0

Error Logs:

ERROR dynamic-controller Error syncing item, requeuing with rate limit {"item":{"name":"test-cluster","namespace":"test-cluster"},"error":"failed to add resource to applyset: object {cluster.x-k8s.io/v1beta1, Kind=Cluster test-cluster/test-cluster} is not json marshallable: json: error calling MarshalJSON for type applyset.ApplyableObject: json: unsupported type: map[ref.al[]ref.Val"}

Which option describes the most your issue?

No response

[jakobmoellerdev]

do you mind letting us know if this is a regression from 0.7.0? (would be great if you could give us a retest on 0.7.0)

[kahirokunn]

@jakobmoellerdev I retested on 0.7.0, but the exact same error occurred. 😭