Description
What happened:
kubectl apply is very slow in reading manifest files. It appears to be the case that kubectl rereads its configuration several times per manifest it is supposed to apply which gets extremely slow, several minutes when applying large amount of manifests.
How to reproduce it (as minimally and precisely as possible):
$ cat kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority: /tmp/cluster.pem
server: https://cluster.url
name: cluster
contexts:
- context:
cluster: cluster
user: user
name: cluster-user
current-context: cluster-user
kind: Config
preferences: {}
users:
- name: user
user:
token: ...
for i in {1..10}; do echo '{ "apiVersion": "v1", "kind": "Namespace", "metadata": { "name": "ns'$i'" }}' ; done > manifests
$ cat manifests
{ "apiVersion": "v1", "kind": "Namespace", "metadata": { "name": "ns1" }}
{ "apiVersion": "v1", "kind": "Namespace", "metadata": { "name": "ns2" }}
{ "apiVersion": "v1", "kind": "Namespace", "metadata": { "name": "ns3" }}
...
Now run following dry-run kubectl and observe how often it opens (and reads) the certificate-authority file cluster.pem:
$ kubectl version
Client Version: v1.31.0
Kustomize Version: v5.4.2
$ for i in {1..10}; do echo '{ "apiVersion": "v1", "kind": "Namespace", "metadata": { "name": "ns'$i'" }}' ; done > manifests
$ KUBECONFIG=config GOMAXPROCS=2 strace -f -e openat apply --dry-run=client -f manifests|& grep cluster.pem -c
27
# run in 1 second
$ for i in {1..100}; do echo '{ "apiVersion": "v1", "kind": "Namespace", "metadata": { "name": "ns'$i'" }}' ; done > manifests
$ KUBECONFIG=config GOMAXPROCS=2 strace -f -e openat apply --dry-run=client -f manifests|& grep cluster.pem -c
207
# run in 12 seconds
Increasing the number of objects in the manifests file increase the times it reads the cluster certificate from its configuration which does not seem necessary.
edit: see below, the real problem is repeated json decoding of the the openapi schema documents
When applying more manifests the time before kubectl even contacts the server increases to several minutes which can be very relevant for e.g. CI test runs.
The problem should not be the the manifest parsing time, as e.g. python yaml or the golang yq version can parse these manifests in a fraction of the time it takes kubectl.