Using bootstrapConfig in CK8sControlPlane results in tls errors in autoscaler pod

[hemanthnakkina]

Creating a CK8SControlplane using the following manifest resulted in following error in autoscaler pod

E0729 05:08:26.921142 2067232 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.16.2.235:6443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for 10.0.0.120, 10.152.183.1, 127.0.0.1, 10.0.0.120, ::1, fe80::f816:3eff:fe65:8d84, not 172.16.2.235"

Getting kubeconfig of deployed cluster and running kubectl commands also resulted in above error.
There is no error when bootstrapConfig section is removed.

Analysis:
When bootstrapConfig section is not there, the cloud-init writes the following file to control node

/capi/etc/config.yaml

Default:
ca-crt: |-
  <>
ca-key: |
  <>
client-ca-crt: |
  <>
client-ca-key: |
  <>
cluster-config:
  annotations:
    k8sd/v1alpha/lifecycle/skip-cleanup-kubernetes-node-on-remove: "true"
    k8sd/v1alpha/lifecycle/skip-stop-services-on-remove: "true"
  cloud-provider: external
  dns:
    cluster-domain: cluster.local
    enabled: true
  gateway:
    enabled: true
  ingress:
    enabled: true
  load-balancer:
    enabled: true
  local-storage:
    enabled: true
  metrics-server:
    enabled: true
  network:
    enabled: true
datastore-type: k8s-dqlite
extra-node-kubelet-args:
  --provider-id: openstack:///34a19eef-f39f-44c1-96eb-80e3cd0ec641
extra-sans:
- 172.16.2.205
k8s-dqlite-port: 2379
pod-cidr: 10.1.0.0/16
service-cidr: 10.152.183.0/24

where as when bootstrapConfig spec is used, the content is dumped

cluster-config:
  annotations:
    k8sd/v1alpha/lifecycle/skip-cleanup-kubernetes-node-on-remove: "true"
    k8sd/v1alpha/lifecycle/skip-stop-services-on-remove: "true"
  network:
    enabled: true
  dns:
    enabled: true
    cluster-domain: cluster.local
    upstream-nameservers: [8.8.8.8]
  local-storage:
    enabled: true
    reclaim-policy: Retain
  metrics-server:
    enabled: true
  load-balancer:
    enabled: true
    l2-mode: true
  ingress:
    enabled: false

You can see the generated content like the certificates and keys and default values of certain parameters are not part of the file written to /capi/etc/config.yaml.

Seems like the expectation from user is to provide the complete bootstrap config file.

However user cannot provide the certificates and keys as part of bootstrap config file as I believe they are generated by Cluster API.
Also it is not user firendly to write complete bootstrapConfig section just to enable/disable a feature.

Expectation is to User provided bootstrapConfig spec should be merged with the defaults or generated content.

[HomayoonAlimohammadi]

Hi @hemanthnakkina and thank you for bringing this up. As of right now, user provided bootstrap config completely replaces the generated one, instead of getting merged with it. We're discussing the best way forward with the team. The linked PR is a proposed solution for merging the two.

[HomayoonAlimohammadi]

Hi @hemanthnakkina. As this turns out to be a feature request more than a bug, and can not be solved without proper specs and discussions, I'd like to ask you to create a PF ticket so that we can plan on working on this internally. Thanks a lot.