Watch events sometimes fail to parse

[AlexandreTunstall]

Current and expected behavior

Sometimes, one of my controllers encounters one of the following errors.

QueueError(WatchFailed(SerdeError(Error("invalid type: string \"kind\", expected adjacently tagged enum WatchEvent", line: 1, column: 8))))
QueueError(WatchFailed(SerdeError(Error("invalid type: string \"code\", expected adjacently tagged enum WatchEvent", line: 1, column: 8))))
QueueError(WatchFailed(SerdeError(Error("invalid type: string \"status\", expected adjacently tagged enum WatchEvent", line: 1, column: 10))))
QueueError(WatchFailed(SerdeError(Error("invalid type: string \"reason\", expected adjacently tagged enum WatchEvent", line: 1, column: 10))))
QueueError(WatchFailed(SerdeError(Error("invalid type: string \"message\", expected adjacently tagged enum WatchEvent", line: 1, column: 11))))
QueueError(WatchFailed(SerdeError(Error("invalid type: string \"details\", expected adjacently tagged enum WatchEvent", line: 1, column: 11))))
QueueError(WatchFailed(SerdeError(Error("invalid type: string \"metadata\", expected adjacently tagged enum WatchEvent", line: 1, column: 12))))
QueueError(WatchFailed(SerdeError(Error("invalid type: string \"apiVersion\", expected adjacently tagged enum WatchEvent", line: 1, column: 14))))
QueueError(WatchFailed(SerdeError(Error("invalid type: string \"retryAfterSeconds\", expected adjacently tagged enum WatchEvent", line: 1, column: 23))))

As a result, the controller is unable to respond immediately to events and has to retry until the error disappears (which it usually does after a few retries).

It seems to me like kube is not correctly parsing some WatchEvent entries, but I can't understand why, nor am I able to see from the debug logs the raw JSON of the events that fail to parse (tcpdump would not help because of TLS).

Possible solution

No response

Additional context

The error is frequent, unpredictable, and intermittent.

Could it be something wrong with my custom resources?

Environment

The Kubernetes cluster was manually set up.

Server Version: v1.36.1
kube version: 3.1.0

What is OS in your Dockerfile?

N/A. The image in which kube is running was generated via nix (i.e. effectively FROM scratch). The image consists solely of the statically linked binary.

Configuration and features

k8s-openapi = { version = "0.27.0", features = ["schemars", "v1_35"] }
kube = { version = "3.0.1", features = ["derive", "runtime", "unstable-runtime", "ws"] }

YAML

No response

Affected crates

kube-core

Would you like to work on fixing this bug?

None

[nightkr]

but I can't understand why, nor am I able to see from the debug logs the raw JSON of the events that fail to parse (tcpdump would not help because of TLS).

You can use kubectl proxy to expose a plaintext proxy on localhost. (It's not quite 1:1 since it also delegates authn to kubelet, but that should be fine here.)

[daemonfire300]

Any indication as to how this can be (somewhat) reliably reproduced, e.g., setup of the watcher etc.? E.g., if you think it might be related to your CR(D) you might want to share it?

[daemonfire300]

I just checked the watch api, e.g., from this example: https://kubernetes.io/docs/reference/using-api/api-concepts/#streaming-lists / https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.35/#watchevent-v1-meta

{
  "type": "ADDED",
  "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "8467", "name": "foo"}, ...}
}
{
  "type": "ADDED",
  "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "5726", "name": "bar"}, ...}
}

I have not looked at the internals of kube-rs for this, but as far as I understand the serde error. This would happen if "type", is missing and object as well, or put more correctly: the inner value of object is being provided. (Assumption, I have not tested this with serde, but the minimal setup would be "easy" I think, use WatchEvent and try to deserialize:

Just this, and see whether it produces this error (note, ... is a placeholder, so dont take literally that json)

{"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "5726", "name": "bar"}, ...}

So either the wrong "endpoint" at the api server is hit or the stream "reader" does not correctly chunk the data so that it is correctly deserializable.

[daemonfire300]

My output when running this test

running 1 test
missing field `type` at line 1 column 14
invalid type: string "kind", expected adjacently tagged enum WatchEvent at line 1 column 8
test test::watch_serde_test ... ok

Minimal test setup

#[cfg(test)]
mod test {
    #[test]
    fn watch_serde_test() {
        use serde::Deserialize;

        #[derive(Debug, Deserialize)]
        #[serde(tag = "type", content = "object", rename_all = "UPPERCASE")]
        enum WatchEvent<K> {
            Added(K),
            Modified(K),
            Deleted(K),
        }

        #[derive(Debug, Deserialize)]
        struct Pod {
            kind: String,
        }

        // Valid:
        let ok = r#"{"type":"ADDED","object":{"kind":"Pod"}}"#;
        let _: WatchEvent<Pod> = serde_json::from_str(ok).unwrap();

        // Wrong shape: bare Kubernetes object/list/status, not a watch event.
        let wrong = r#"{"kind":"Pod"}"#;
        let err = serde_json::from_str::<WatchEvent<Pod>>(wrong).unwrap_err();
        eprintln!("{err}");

        let line = r#"  "kind": "PodList","#;
        let err = serde_json::from_str::<WatchEvent<Pod>>(line).unwrap_err();
        eprintln!("{err}");
    }
}

This is a dummy with which I can trigger the error. But it does not actually tell us how your code hits it and what inside kube-rs triggers it.

[AlexandreTunstall]

I can't share the full code nor have I found a minimal reproducer yet, but one of the affected controllers is doing something like this:

let api: Api<MyCustomResource> = Api::all(client.clone());
let other_api: Api<MyCustomResource> = Api::all(client.clone());
Controller::new(api, watcher::Config::default())
    .owns(other_api, watcher::Config::default())
    .run(...)

Both MyCustomResource and MyOtherCustomResource (names redacted) are for types of this shape:

#[derive(CustomResource, Serialize, Deserialize, Debug, PartialEq, Clone, KubeSchema)]
#[kube(
    group = "...",
    version = "v1alpha1",
    kind = "MyCustomResource",
    plural = "mycustomresources",
    derive = "PartialEq",
    status = "MyCustomResourceStatus",
    namespaced
)]
#[serde(rename_all = "camelCase")]
pub struct MyCustomResourceSpec {
    #[x_kube(
        validation = ...
    )]
    pub template: PodTemplateSpec,
}

#[derive(Serialize, Deserialize, Debug, PartialEq, Clone, KubeSchema, Default)]
#[serde(rename_all = "camelCase")]
pub struct MyCustomResourceStatus {
    pub some_number: i32
}

As far as I understand, this is the intended way to use kube-rs for controllers?

I'm currently trying to reproduce it under kubectl proxy with tcpdump running, but I'll see if I can find a minimal reproducer too.

[AlexandreTunstall]

I have successfully gotten a dump and correlated it with the logs from my controllers.

Here are two of the problematic responses (JSON pretty-printed for readability).

HTTP/1.1 503 Service Unavailable 
Audit-Id: 60773a90-008b-472a-88d6-08442cdf0c4b 
Cache-Control: no-cache, private 
Content-Length: 248 
Content-Type: application/json 
Date: Sat, 23 May 2026 19:10:21 GMT 
Retry-After: 5 
X-Kubernetes-Pf-Flowschema-Uid:  
X-Kubernetes-Pf-Prioritylevel-Uid:  
 
{
    "apiVersion": "v1",
    "code": 503,
    "details": {
        "retryAfterSeconds": 5
    },
    "kind": "Status",
    "message": "the request has been made before all known HTTP paths have been installed, please try again",
    "metadata": {
    },
    "reason": "ServiceUnavailable",
    "status": "Failure"
}

HTTP/1.1 429 Too Many Requests 
Audit-Id: 7ff721fe-9075-4d59-8bb2-c8dc8339d1f3 
Cache-Control: no-cache, private 
Content-Length: 222 
Content-Type: application/json 
Date: Sat, 23 May 2026 19:10:23 GMT 
Retry-After: 1 
X-Kubernetes-Pf-Flowschema-Uid: 7de3483b-449d-45e8-80ab-32e356aad7af 
X-Kubernetes-Pf-Prioritylevel-Uid: 08e2ca5c-9356-4632-aea1-e033d865c9c8 
 
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "storage is (re)initializing",
  "reason": "TooManyRequests",
  "details": {
    "retryAfterSeconds": 1
  },
  "code": 429
}

(The above errors were probably the result of querying the apiserver right after it starts.)

Is kube-rs trying to decode error responses as WatchEvent and failing because they don't have the usual type field?

[daemonfire300]

I could find this error only once in k8s (https://github.com/kubernetes/kubernetes/blob/release-1.36/staging/src/k8s.io/apiserver/pkg/util/notfoundhandler/not_found_handler.go)

My best guess, because I think I never ever over half a decade seen this message, is that something with your cluster is not ready yet, i.e., have you check listing api-resources with kubectl and using kubectl watch/get on your CRs in parallel?

The 429 looks like it might be related to kube-rs trying it too often, but I am not a kube-rs maintainer so I would need to dig into the internals there, which I try to delay until someone with more knowledge looks at this or we can nail the problem before that, e.g., maybe it's on your kubernetes setup's end.

[nightkr]

I mean yeah it seems to be basically both. The root issue is that the cluster isn't ready, but we should also do a much better job handling that failure.

[daemonfire300]

I mean yeah it seems to be basically both. The root issue is that the cluster isn't ready, but we should also do a much better job handling that failure.

If you want I could look into this, because I would like to help out with kube-rs if possible. Happy to also collaborate on this as well. :)

[AlexandreTunstall]

My best guess [...] is that something with your cluster is not ready yet

Yes, I agree. Namely, the kube-apiserver process seems to still be spinning up. This hypothesis matches up with the observed temporary nature of the issue. The strange part is that Cilium is still routing traffic to such freshly spawned processes even though I have readiness checks configured on the static pods.

I have also observed the issue while scaling up the control plane and while moving it between nodes, so the problem seems to occur whenever a brand new kube-apiserver process is serving requests. Perhaps the rarity of these short-lived causative events would also explain why you've never seen these errors before?

But I do believe that kube-rs should handle the responses better to return error messages that are more descriptive of the issue. As is, the error message contains none of the helpful information the server provided and it may mislead developers to go looking for deserializer problems in their own code.

The 429 looks like it might be related to kube-rs trying it too often,

I doubt it, because my logs do not suggest that there are terribly many attempts.
Note that I have not configured any rate limiting in kube-apiserver; the default settings, which I suspect to be no limiting, apply.

If you want I could look into this, because I would like to help out with kube-rs if possible.

If you need any help reproducing the issue or testing a fix, please feel free to ask.