listing resources often gets 429 "storage is (re)initializing" errors in envtest

[orangecms]

Just for tracking/understading, here are the rough details now; I'll add a reproducer later.

I have a controller with a reconciler operating on CRDs from Envoy Gateway and Kubernetes Gateway.
Rust code is generated via Kopium.

CRDs stem from https://github.com/envoyproxy/gateway/releases/download/v1.6.4/envoy-gateway-crds.yaml and https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.4.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml (also others from k8s-gateway, just one example).

My reconciler fetches a list of some of the resources, and when I load the CRDs, then the list API calls would get me 429 errors.

Here is a helper that I use for the env setup; may make sense to get this here upstream, as I think it's a very common use case:

    /// Helper for test setup boilerplate.
    pub async fn setup_env_with_crds_and_namespaces(
        crds: &[CustomResourceDefinition],
        namespaces: Vec<String>,
    ) -> (Client, Server) {
        let env = Environment::default().with_crds(crds).unwrap();
        let server = env.create().await.unwrap();
        let client = server.client().unwrap();
        for ns in namespaces {
            create_ns(&ns, &client).await;
        }
        (client, server)
 }

Now I am doing this in my test case:

        let (client, _server) = setup_env_with_crds_and_namespaces(crds, namespaces).await;
        /* ... creating `settings` here, not important for repro, just for completeness ... */
        let ctx = Context { client, settings };
        reconcile(Arc::new(resource), Arc::new(ctx)).await.unwrap();

and reconcile() does this (regardless of listing full or metadata only):

    let list_params = ListParams::default();
    let api = kube::Api::<K>::all(client.clone());
    let res = api.list_metadata(&list_params).await?;

I get the following (formatted and unescaped for readability):

ApiError: storage is (re)initializing: TooManyRequests (Status {
  status: Some(Failure),
  code: 429,
  message: "storage is (re)initializing",
  metadata: Some(ListMeta {
    continue_: None,
    remaining_item_count: None,
    resource_version: None,
    self_link: None
  }),
  reason: "TooManyRequests",
  details: Some(StatusDetails {
    name: "",
    group: "",
    kind: "",
    uid: "",
    causes: [],
    retry_after_seconds: 1
  })
})

For the sake of the experiments, I added a timeout between the setup and running reconcile(), even a whole minute, to no avail.
A colleague of mine hinted on an event signalling readiness after CRD loading on the Go side, is that possibly something envtest will need to wait on / acknowledge somehow?

Note that I get this only for list API calls. Doing a patch() seems to work just fine, even when loading the same CRDs that I get the errors with when doing the list.

[orangecms]

alright, here is the reproducer: https://github.com/orangecms/kube_list_test
just cargo run and besides the warnings due to unused generated code, I get:

Set up env
Set up server (may take a minute for initial download...)
Set up namespaces
Setup done
Err(
    Api(
        Status {
            status: Some(
                Failure,
            ),
            code: 429,
            message: "storage is (re)initializing",
            metadata: Some(
                ListMeta {
                    continue_: None,
                    remaining_item_count: None,
                    resource_version: None,
                    self_link: None,
                },
            ),
            reason: "TooManyRequests",
            details: Some(
                StatusDetails {
                    name: "",
                    group: "",
                    kind: "",
                    uid: "",
                    causes: [],
                    retry_after_seconds: 1,
                },
            ),
        },
    ),
)

[orangecms]

My current workaround is to repeat the request within the test case, but it feels like that's the wrong place to do it.

I think it should be handled via the retry strategy, such as in https://github.com/kube-rs/kube/blob/main/examples/custom_client_retry.rs

Would you be okay with me adding this here in envtest?

[Danil-Grigorev]

Hey, that would be great to try, sounds like a good feature to add. Probably would also need to make kube version pinned with that addition, as this change is a recent one.