Add AKS integration

[anson627]

What would you like to be added?

Make spiderpool work on AKS with ipvlan as underlay network, similarly to
https://spidernet-io.github.io/spiderpool/v1.1/usage/install/cloud/get-started-alibaba
https://spidernet-io.github.io/spiderpool/v1.1/usage/install/cloud/get-started-aws/

Here are the test steps:

1. Create AKS cluster with Azure CNI disabled:

RESOURCE_GROUP=cni-test
CLUSTER_NAME=cni-test

az aks create -l eastus2 \
    --resource-group "${RESOURCE_GROUP}" \
    --cluster-name "${CLUSTER_NAME}" \
    --tier standard \
    --kubernetes-version 1.34.0 \
    --network-plugin none \
    --disable-disk-driver \
    --disable-file-driver \
    --nodepool-name system \
    --vm-set-type VirtualMachines \
    --node-vm-size Standard_D8ds_v5 \
    --node-count 3

az aks get-credentials --resource-group "${RESOURCE_GROUP}" \
    --name "${CLUSTER_NAME}" \
    --overwrite-existing

2. Install spiderpool

helm repo add spiderpool https://spidernet-io.github.io/spiderpool
helm repo update spiderpool
helm install spiderpool spiderpool/spiderpool --namespace default --create-namespace --set ipam.enableStatefulSet=false --set multus.multusCNI.defaultCniCRName="ipvlan-eth0"

3. Create SpiderMultusConfig

cat <<EOF | kubectl apply -f -
apiVersion: spiderpool.spidernet.io/v2beta1
kind: SpiderMultusConfig
metadata:
  name: ipvlan-eth0
  namespace: default
spec:
  cniType: ipvlan
  enableCoordinator: true
  ipvlan:
    master:
    - eth0
EOF

4. Add secondary ip configuration on nic attached to VM:

5. Add SpiderIPPool for node/VM

cat <<EOF | kubectl apply -f -
apiVersion: spiderpool.spidernet.io/v2beta1
kind: SpiderIPPool
metadata:
  name: aks-system-27117018-vms22
spec:
  subnet: 10.224.0.32/28
  ips:
    - 10.224.0.34-10.224.0.46
  gateway: 10.224.0.33
  default: true
  nodeName: ["aks-system-27117018-vms22"]
  multusName: ["default/ipvlan-eth0"]
EOF

5. repeat step 4 and 5 for nodes aks-system-27117018-vms21 and aks-system-27117018-vms23

Why is this needed?

Currently ipam is not working as coredns and metrics-server are stuck in ContainerCreating

 k get pod -n kube-system -o wide
NAME                                  READY   STATUS              RESTARTS   AGE   IP           NODE                        NOMINATED NODE   READINESS GATES
cloud-node-manager-8vblw              1/1     Running             0          26h   10.224.0.6   aks-system-27117018-vms23   <none>           <none>
cloud-node-manager-b8wft              1/1     Running             0          26h   10.224.0.5   aks-system-27117018-vms21   <none>           <none>
cloud-node-manager-pkhhc              1/1     Running             0          26h   10.224.0.4   aks-system-27117018-vms22   <none>           <none>
coredns-6d7b684fb5-n2h4h              0/1     ContainerCreating   0          26h   <none>       aks-system-27117018-vms21   <none>           <none>
coredns-autoscaler-65bcdc4967-ckqs8   0/1     ContainerCreating   0          26h   <none>       aks-system-27117018-vms21   <none>           <none>
konnectivity-agent-67998c6866-wdwxm   1/1     Running             0          26h   10.224.0.6   aks-system-27117018-vms23   <none>           <none>
kube-proxy-6h8nz                      1/1     Running             0          26h   10.224.0.5   aks-system-27117018-vms21   <none>           <none>
kube-proxy-q8vd2                      1/1     Running             0          26h   10.224.0.6   aks-system-27117018-vms23   <none>           <none>
kube-proxy-qt52g                      1/1     Running             0          26h   10.224.0.4   aks-system-27117018-vms22   <none>           <none>
metrics-server-674b4b94b6-glmb2       0/2     ContainerCreating   0          26h   <none>       aks-system-27117018-vms21   <none>           <none>
metrics-server-674b4b94b6-gsfvd       0/2     ContainerCreating   0          26h   <none>       aks-system-27117018-vms21   <none>           <none>

k describe pod coredns-6d7b684fb5-g5n24  -n kube-system

Events:
  Type     Reason                  Age                From               Message
  ----     ------                  ----               ----               -------
  Normal   Scheduled               20s                default-scheduler  Successfully assigned kube-system/coredns-6d7b684fb5-g5n24 to aks-system-27117018-vms22
  Warning  FailedCreatePodSandBox  20s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1760b66b09bf98198e68c022168bf63b212bbdeb1cc4e5efe62caf32642c4713": plugin type="multus" name="multus-cni-network" failed (add): [kube-system/coredns-6d7b684fb5-g5n24/184105dd-26ec-4011-bc92-3900d008a1e3:ipvlan-eth0]: error adding container to network "ipvlan-eth0": plugin type="coordinator" failed (add): failed to GetCoordinatorConfig: [GET /coordinator/config][500] getCoordinatorConfigFailure  spidercoordinator: default no ready
  Warning  FailedCreatePodSandBox  19s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c3785590a8f4eda01f35f4dd41a98116fc0f547edf2e6413a7cf3f5d6d1a0735": plugin type="multus" name="multus-cni-network" failed (add): [kube-system/coredns-6d7b684fb5-g5n24/184105dd-26ec-4011-bc92-3900d008a1e3:ipvlan-eth0]: error adding container to network "ipvlan-eth0": plugin type="ipvlan" failed (add): failed to add IP addr {Interface:0xc0001b07a8 Address:{IP:10.224.0.35 Mask:fffffff0} Gateway:10.224.0.33} to "eth0": address already in use
  Warning  FailedCreatePodSandBox  18s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "26504f5226c228b32b46420acfcf104275920d458342cffdf9022b09b48f6aa3": plugin type="multus" name="multus-cni-network" failed (add): [kube-system/coredns-6d7b684fb5-g5n24/184105dd-26ec-4011-bc92-3900d008a1e3:ipvlan-eth0]: error adding container to network "ipvlan-eth0": plugin type="coordinator" failed (add): failed to GetCoordinatorConfig: [GET /coordinator/config][500] getCoordinatorConfigFailure  spidercoordinator: default no ready
  Warning  FailedCreatePodSandBox  17s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "86d1d75b052acb7f6821bb0e711ddcf9e8616363d1f9c082f6be2381c6b99015": plugin type="multus" name="multus-cni-network" failed (add): [kube-system/coredns-6d7b684fb5-g5n24/184105dd-26ec-4011-bc92-3900d008a1e3:ipvlan-eth0]: error adding container to network "ipvlan-eth0": plugin type="ipvlan" failed (add): failed to add IP addr {Interface:0xc00009a798 Address:{IP:10.224.0.35 Mask:fffffff0} Gateway:10.224.0.33} to "eth0": address already in use
  Warning  FailedCreatePodSandBox  16s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7ca0aa6f1ddab00207a153e08e781b8a4e9def0fe85bcc23a9206bc2ce592459": plugin type="multus" name="multus-cni-network" failed (add): [kube-system/coredns-6d7b684fb5-g5n24/184105dd-26ec-4011-bc92-3900d008a1e3:ipvlan-eth0]: error adding container to network "ipvlan-eth0": plugin type="coordinator" failed (add): failed to GetCoordinatorConfig: [GET /coordinator/config][500] getCoordinatorConfigFailure  spidercoordinator: default no ready
  Warning  FailedCreatePodSandBox  15s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a6d83a605ae58ae1a705438dc6ae3c7fd20cfd2c0b46f8310b83ebc47a5dd158": plugin type="multus" name="multus-cni-network" failed (add): [kube-system/coredns-6d7b684fb5-g5n24/184105dd-26ec-4011-bc92-3900d008a1e3:ipvlan-eth0]: error adding container to network "ipvlan-eth0": plugin type="coordinator" failed (add): failed to GetCoordinatorConfig: [GET /coordinator/config][500] getCoordinatorConfigFailure  spidercoordinator: default no ready
  Warning  FailedCreatePodSandBox  14s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f5ac75b25f2ac9a338ca0e0a2374540288d695b04cf6bb45576716d02befdcd4": plugin type="multus" name="multus-cni-network" failed (add): [kube-system/coredns-6d7b684fb5-g5n24/184105dd-26ec-4011-bc92-3900d008a1e3:ipvlan-eth0]: error adding container to network "ipvlan-eth0": plugin type="ipvlan" failed (add): failed to add IP addr {Interface:0xc00009b688 Address:{IP:10.224.0.35 Mask:fffffff0} Gateway:10.224.0.33} to "eth0": address already in use
  Warning  FailedCreatePodSandBox  13s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "dae197b9388da99646dc21a3dcdc4ec0cae55590285a00b728d209657874de72": plugin type="multus" name="multus-cni-network" failed (add): [kube-system/coredns-6d7b684fb5-g5n24/184105dd-26ec-4011-bc92-3900d008a1e3:ipvlan-eth0]: error adding container to network "ipvlan-eth0": plugin type="coordinator" failed (add): failed to GetCoordinatorConfig: [GET /coordinator/config][500] getCoordinatorConfigFailure  spidercoordinator: default no ready
  Warning  FailedCreatePodSandBox  12s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d6b31e2ea3d2e3478a80912b77d08d24109d463cfae7680221c32edc3622ca5f": plugin type="multus" name="multus-cni-network" failed (add): [kube-system/coredns-6d7b684fb5-g5n24/184105dd-26ec-4011-bc92-3900d008a1e3:ipvlan-eth0]: error adding container to network "ipvlan-eth0": plugin type="ipvlan" failed (add): failed to add IP addr {Interface:0xc000192798 Address:{IP:10.224.0.35 Mask:fffffff0} Gateway:10.224.0.33} to "eth0": address already in use
  Normal   SandboxChanged          0s (x20 over 19s)  kubelet            Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  0s (x12 over 11s)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "51f8efe045e24b950c9152eb783063b207f7ab123184c85d09fa03919163a17c": plugin type="multus" name="multus-cni-network" failed (add): [kube-system/coredns-6d7b684fb5-g5n24/184105dd-26ec-4011-bc92-3900d008a1e3:ipvlan-eth0]: error adding container to network "ipvlan-eth0": plugin type="coordinator" failed (add): failed to GetCoordinatorConfig: [GET /coordinator/config][500] getCoordinatorConfigFailure  spidercoordinator: default no ready

 k get spiderippool aks-system-27117018-vms22 -o yaml
apiVersion: spiderpool.spidernet.io/v2beta1
kind: SpiderIPPool
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"spiderpool.spidernet.io/v2beta1","kind":"SpiderIPPool","metadata":{"annotations":{},"name":"aks-system-27117018-vms22"},"spec":{"default":true,"gateway":"10.224.0.33","ips":["10.224.0.34-10.224.0.46"],"multusName":["kube-system/ipvlan-eth0"],"nodeName":["aks-system-27117018-vms22"],"subnet":"10.224.0.32/28"}}
  creationTimestamp: "2025-11-30T18:13:34Z"
  finalizers:
  - spiderpool.spidernet.io
  generation: 2
  labels:
    ipam.spidernet.io/ippool-cidr: 10-224-0-32-28
  name: aks-system-27117018-vms22
  resourceVersion: "455295"
  uid: a983f0d2-5a33-4b4b-ab1b-86e84d8f958b
spec:
  default: true
  disable: false
  gateway: 10.224.0.33
  ipVersion: 4
  ips:
  - 10.224.0.34-10.224.0.46
  multusName:
  - default/ipvlan-eth0
  nodeName:
  - aks-system-27117018-vms22
  subnet: 10.224.0.32/28
status:
  allocatedIPCount: 2
  allocatedIPs: '{"10.224.0.34":{"pod":"default/nginx-lb-76745b8c95-jpkkq","podUid":"84d777f1-22b1-4595-92dd-1c31d5a38c19"},"10.224.0.35":{"pod":"kube-system/coredns-6d7b684fb5-g5n24","podUid":"184105dd-26ec-4011-bc92-3900d008a1e3"}}'
  totalIPCount: 13

How to implement it (if possible)?

spidernet controller/agent/cooridnator should setup node network configs to make pod-to-pod/ingress/egress work

Additional context

No response

[anson627]

not sure why

error adding container to network "ipvlan-eth0": plugin type="ipvlan" failed (add): failed to add IP addr {Interface:0xc00009b688 Address:{IP:10.224.0.35 Mask:fffffff0} Gateway:10.224.0.33} to "eth0": address already in use

10.224.0.33 address is not in use as check

k debug node/aks-system-27117018-vms22 -it --image=ubuntu -- chroot host
# ip route show
default via 10.224.0.1 dev eth0 proto dhcp src 10.224.0.4 metric 100 
10.224.0.0/16 dev eth0 proto kernel scope link src 10.224.0.4 metric 100 
10.224.0.1 dev eth0 proto dhcp scope link src 10.224.0.4 metric 100 
168.63.129.16 via 10.224.0.1 dev eth0 proto dhcp src 10.224.0.4 metric 100 
169.254.169.254 via 10.224.0.1 dev eth0 proto dhcp src 10.224.0.4 metric 100 
# ip add show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 60:45:bd:7f:e0:42 brd ff:ff:ff:ff:ff:ff
    inet 10.224.0.4/16 metric 100 brd 10.224.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6245:bdff:fe7f:e042/64 scope link 
       valid_lft forever preferred_lft forever

[weizhoublue]

According to Spiderpool's design, the IP conflict checking with ARP probe request works during CNI is called to ensure the IP's availability. In my opinion, the VPC networking of the AKS is different from traditional networking in data center, and the arp proxy in the VPC responds the spiderpool' ARP request. So the resolution is to turn off the ARP check

there are two approaches to finish this:

1. use helm values during installation

helm install spiderpool spiderpool/spiderpool -n kube-system \
  --set ipam.enableIPConflictDetection=false \
  --set ipam.enableGatewayDetection=false

2. modify the configmap and then restart agents , if installation is finished

apiVersion: v1
kind: ConfigMap
metadata:
  name: spiderpool-conf
  namespace: kube-system
data:
  conf.yml: |
    ...
    enableIPConflictDetection: false   
    enableGatewayDetection: false      
    ...

@cyclinder do you agree with the above

[cyclinder]

I think this issue is related to spidercoordinator config. Can you run the following command ?

kubectl get spidercoordinatorconfig -o yaml
kubectl get cm -n kube-system kubeadm-config -o yaml

If you see that the kubeadm-config doesn't exist. You can create it as follows:

export POD_SUBNET=<DEFAULT_CLUSTER_POD_SUBNET> 
export SERVICE_SUBNET=<YOUR_SERVICE_SUBNET>
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: kubeadm-config
  namespace: kube-system
data:
  ClusterConfiguration: |
    networking:
      podSubnet: ${POD_SUBNET}
      serviceSubnet: ${SERVICE_SUBNET}
EOF

[anson627]

According to Spiderpool's design, the IP conflict checking with ARP probe request works during CNI is called to ensure the IP's availability. In my opinion, the VPC networking of the AKS is different from traditional networking in data center, and the arp proxy in the VPC responds the spiderpool' ARP request. So the resolution is to turn off the ARP check

there are two approaches to finish this:

1. use helm values during installation

helm install spiderpool spiderpool/spiderpool -n kube-system \
  --set ipam.enableIPConflictDetection=false \
  --set ipam.enableGatewayDetection=false

2. modify the configmap and then restart agents , if installation is finished

apiVersion: v1
kind: ConfigMap
metadata:
  name: spiderpool-conf
  namespace: kube-system
data:
  conf.yml: |
    ...
    enableIPConflictDetection: false   
    enableGatewayDetection: false      
    ...

@cyclinder do you agree with the above

thanks @weizhoublue for taking a look, I saw enableIPConflictDetection and enableGatewayDetection are false by default

k get configmap spiderpool-conf -o yaml
apiVersion: v1
data:
  clusterNetwork: ipvlan-eth0
  conf.yml: |
    ipamUnixSocketPath: /var/run/spidernet/spiderpool.sock
    enableIPv4: true
    enableIPv6: true
    enableStatefulSet: false
    enableKubevirtStaticIP: true
    enableCleanOutdatedEndpoint: false
    enableSpiderSubnet: true
    enableAutoPoolForApplication: true
    enableIPConflictDetection: false
    enableGatewayDetection: false
    enableValidatingResourcesDeletedWebhook: false
    clusterSubnetDefaultFlexibleIPNumber: 1
    tuneSysctlConfig: true
    podResourceInject:
      enabled: false
      namespacesExclude: ["kube-system","spiderpool","metallb-system","istio-system"]
      namespacesInclude: []
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: spiderpool
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2025-11-30T17:48:03Z"
  labels:
    app.kubernetes.io/component: spiderpool-controller
    app.kubernetes.io/instance: spiderpool
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: spiderpool
    app.kubernetes.io/version: 1.0.5
    helm.sh/chart: spiderpool-1.0.5
  name: spiderpool-conf
  namespace: default
  resourceVersion: "14953"
  uid: 54892977-c0ee-4b22-8772-ee4673d6a34b

[anson627]

I think this issue is related to spidercoordinator config. Can you run the following command ?

kubectl get spidercoordinatorconfig -o yaml
kubectl get cm -n kube-system kubeadm-config -o yaml

If you see that the kubeadm-config doesn't exist. You can create it as follows:

export POD_SUBNET=<DEFAULT_CLUSTER_POD_SUBNET> 
export SERVICE_SUBNET=<YOUR_SERVICE_SUBNET>
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: kubeadm-config
  namespace: kube-system
data:
  ClusterConfiguration: |
    networking:
      podSubnet: ${POD_SUBNET}
      serviceSubnet: ${SERVICE_SUBNET}
EOF

@cyclinder sorry for the confusion! the cluster is not created by kubeadm, it's created with kubelet/containerd on Azure Linux VM registered to a managed control plane (apiserver/etcd) hosted by Azure, my goal is to create a underly network not overlay network, so there is no concept of pod subnet, each node has one nic with primary ip and secondary ip range, I use primary ip as node ip, and try to assign pod ip from the secondary ip range

and there is no spidercoordinatorconfig as I checked, is it supposed to be enabled by default?

 kubectl get spidercoordinatorconfig -o yaml
error: the server doesn't have a resource type "spidercoordinatorconfig"

[anson627]

this is how I manually setup ipvlan on AKS and Azure, let me know if it is possible to do the same thing with spidernet:

https://github.com/containernetworking/plugins/pull/1218/files#diff-49c043d18a8967be0c8f30b56270791d8827d03fbc47b3ad7b35506cd35aa25e

[cyclinder]

@anson627 Looks like your environment is based on an underlay network. Spiderpool-controller obtains the cluster's pod and service CIDRs from either the kubeadm-config or the controller-manager pod, and writes them to the spidercoordinator CR. When creating Pods, Spiderpool writes these CIDRs information into the Pod as routes, enabling communication between Underlay Pods and cluster traffic.

Based on this, I suspect that spiderpool-controller failed to obtain these CIDRs. I believe you should be able to fix this by manually creating the kubeadm-config ConfigMap. It would be even better if you could provide the spiderpool-controller pod logs, which would allow me to further confirm the issue.

[cyclinder]

this is how I manually setup ipvlan on AKS and Azure, let me know if it is possible to do the same thing with spidernet:https://github.com/containernetworking/plugins/pull/1218/files#diff-49c043d18a8967be0c8f30b56270791d8827d03fbc47b3ad7b35506cd35aa25e

I think yes, it's an interesting topic.

[weizhoublue]

" ip is taken" may be is a misleading log, I notice another important error log: plugin type="coordinator" failed (add): failed to GetCoordinatorConfig: [GET /coordinator/config][500] getCoordinatorConfigFailure spidercoordinator: default no ready

@anson627 For reference, I think you could upload the logs of spiderpool-agent pod and the CNI components ( /var/log/spidernet/spiderpool.log and /var/log/spidernet/coordinator.log ) , from the node where pod failed to be created. They probably help verify @cyclinder 's idea

[anson627]

thanks for the detailed explanation! after applying kubeadm-config, ipam is working now, but egress to API server is not working

k get configmap/kubeadm-config -n kube-system -o yaml
apiVersion: v1
data:
  ClusterConfiguration: |
    networking:
      podSubnet: 10.224.0.0/16
      serviceSubnet: 10.0.0.0/16
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"ClusterConfiguration":"networking:\n  podSubnet: 10.224.0.0/16\n  serviceSubnet: 10.0.0.0/16\n"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"kubeadm-config","namespace":"kube-system"}}
  creationTimestamp: "2025-12-03T20:54:43Z"
  name: kubeadm-config
  namespace: kube-system
  resourceVersion: "1271757"
  uid: b332200a-c3fc-4a7f-9b0b-9f1f8def6b7b

k get pod -o wide -n kube-system
NAME                                  READY   STATUS             RESTARTS      AGE     IP            NODE                        NOMINATED NODE   READINESS GATES
cloud-node-manager-8vblw              1/1     Running            0             3d4h    10.224.0.6    aks-system-27117018-vms23   <none>           <none>
cloud-node-manager-b8wft              1/1     Running            0             3d4h    10.224.0.5    aks-system-27117018-vms21   <none>           <none>
cloud-node-manager-pkhhc              1/1     Running            0             3d4h    10.224.0.4    aks-system-27117018-vms22   <none>           <none>
coredns-6d7b684fb5-rs52d              0/1     Running            0             3m58s   10.224.0.51   aks-system-27117018-vms23   <none>           <none>
coredns-autoscaler-65bcdc4967-ckqs8   0/1     CrashLoopBackOff   5 (72s ago)   3d4h    10.224.0.20   aks-system-27117018-vms21   <none>           <none>
konnectivity-agent-67998c6866-wdwxm   1/1     Running            0             3d3h    10.224.0.6    aks-system-27117018-vms23   <none>           <none>
kube-proxy-6h8nz                      1/1     Running            0             3d4h    10.224.0.5    aks-system-27117018-vms21   <none>           <none>
kube-proxy-q8vd2                      1/1     Running            0             3d4h    10.224.0.6    aks-system-27117018-vms23   <none>           <none>
kube-proxy-qt52g                      1/1     Running            0             3d4h    10.224.0.4    aks-system-27117018-vms22   <none>           <none>
metrics-server-674b4b94b6-gsfvd       1/2     CrashLoopBackOff   5 (63s ago)   3d4h    10.224.0.18   aks-system-27117018-vms21   <none>           <none>
metrics-server-674b4b94b6-pc9c2       1/2     CrashLoopBackOff   4 (53s ago)   3m38s   10.224.0.52   aks-system-27117018-vms23   <none>           <none>

[cyclinder]

What's your underlay subnet for the pods? 10.224.0.0/16? Your podSubnet in kubeadm-config should not be 10.224.0.0/16, it should be a virtual subnet, and don't overlay any underlay CIDR.

[anson627]

you are right, 10.224.0.0/16 is the underlay subnet to share btw nodes and pods, and every ip is vnet routable, let me try 10.241.0.0/16, which is completely virtual and different from underlay subnet

[anson627]

virtual subnet for pods did not work, again my goal is to use underlay network, not overlay network, very similar to what's in Public Cloud Installation
https://spidernet-io.github.io/spiderpool/v1.0/usage/install/cloud/get-started-alibaba/
https://spidernet-io.github.io/spiderpool/v1.0/usage/install/cloud/get-started-aws/

in azure, there are similar constructs, one vnet (vpc equivalent) can have multiple subnet, and one vm can have multiple nic (eni equivalent), and one nic can have primary ip (for node), and secondary ip range (for pods)