DiskPool operator seems to lose its DiskPool objects for a while

[lahwaacz]

After installing OpenEBS 4.4.0 with single-node Mayastor, creating 2 diskpools on the node, and starting to use it, I'm getting frequent errors from the diskpool operator pod. It seems to be recovering after a while, but it delays binding new PVCs for several minutes, for example...

$ kubectl logs --namespace openebs pods/openebs-operator-diskpool-64b7488945-fvl4c
...
  2026-05-02T16:30:18.871564Z  WARN operator_diskpool: MayastorPool CRD was not found in the cluster, skipping migration
    at k8s/operators/src/pool/main.rs:397

  2026-05-02T16:30:18.920209Z  INFO operator_diskpool: Starting DiskPool Operator (dsp), namespace: "openebs"
    at k8s/operators/src/pool/main.rs:202

  2026-05-02T16:30:18.934078Z  INFO operator_diskpool::context: new resource_version inserted, name: "pool-2"
    at k8s/operators/src/pool/context.rs:134
    in operator_diskpool::reconcile with name: k8s.hostname.tld, status: Some(DiskPoolStatus { cr_state: Created, pool_status: Some(Online), capacity: 1918504009728, used: 1010827264, available: 1917493182464, capacity_q: Some(Quantity(1869169767219)), used_q: Some(Quantity(1010827264)), available_q: Some(Quantity(1869169767219)), encrypted: Some(false), cluster_size: Some(Quantity(4194304)), disk_capacity: Some(Quantity(1869169767219)), max_expandable_size: Some(Quantity(1869169767219)) })
    in kube_runtime::controller::reconciling object with object.ref: DiskPool.v1beta3.openebs.io/pool-2.openebs, object.reason: object updated

  2026-05-02T16:30:18.938114Z  INFO operator_diskpool::context: new resource_version inserted, name: "pool-1"
    at k8s/operators/src/pool/context.rs:134
    in operator_diskpool::reconcile with name: k8s.hostname.tld, status: Some(DiskPoolStatus { cr_state: Created, pool_status: None, capacity: 0, used: 0, available: 0, capacity_q: None, used_q: None, available_q: None, encrypted: None, cluster_size: None, disk_capacity: None, max_expandable_size: None })
    in kube_runtime::controller::reconciling object with object.ref: DiskPool.v1beta3.openebs.io/pool-1.openebs, object.reason: object updated

  2026-05-02T16:30:18.942808Z  INFO operator_diskpool::context: new resource_version inserted, name: "pool-1"
    at k8s/operators/src/pool/context.rs:134
    in operator_diskpool::reconcile with name: k8s.hostname.tld, status: Some(DiskPoolStatus { cr_state: Created, pool_status: Some(Online), capacity: 1918504009728, used: 1681915904, available: 1916822093824, capacity_q: Some(Quantity(1869169767219)), used_q: Some(Quantity(1717986918)), available_q: Some(Quantity(1869169767219)), encrypted: Some(false), cluster_size: Some(Quantity(4194304)), disk_capacity: Some(Quantity(1869169767219)), max_expandable_size: Some(Quantity(1869169767219)) })
    in kube_runtime::controller::reconciling object with object.ref: DiskPool.v1beta3.openebs.io/pool-1.openebs, object.reason: object updated

  2026-05-02T16:31:57.313880Z ERROR operator_diskpool::context: Pool not found, clearing status, name: "pool-1"
    at k8s/operators/src/pool/context.rs:238
    in operator_diskpool::context::pool_check with name: "pool-1", status: Some(DiskPoolStatus { cr_state: Created, pool_status: Some(Online), capacity: 1918504009728, used: 1681915904, available: 1916822093824, capacity_q: Some(Quantity(1869169767219)), used_q: Some(Quantity(1717986918)), available_q: Some(Quantity(1869169767219)), encrypted: Some(false), cluster_size: Some(Quantity(4194304)), disk_capacity: Some(Quantity(1869169767219)), max_expandable_size: Some(Quantity(1869169767219)) })
    in operator_diskpool::reconcile with name: k8s.hostname.tld, status: Some(DiskPoolStatus { cr_state: Created, pool_status: Some(Online), capacity: 1918504009728, used: 1681915904, available: 1916822093824, capacity_q: Some(Quantity(1869169767219)), used_q: Some(Quantity(1717986918)), available_q: Some(Quantity(1869169767219)), encrypted: Some(false), cluster_size: Some(Quantity(4194304)), disk_capacity: Some(Quantity(1869169767219)), max_expandable_size: Some(Quantity(1869169767219)) })
    in kube_runtime::controller::reconciling object with object.ref: DiskPool.v1beta3.openebs.io/pool-1.openebs, object.reason: reconciler requested retry

  2026-05-02T16:31:57.315078Z  INFO operator_diskpool::context: new resource_version inserted, name: "pool-1"
    at k8s/operators/src/pool/context.rs:134
    in operator_diskpool::reconcile with name: k8s.hostname.tld, status: Some(DiskPoolStatus { cr_state: Created, pool_status: None, capacity: 0, used: 0, available: 0, capacity_q: None, used_q: None, available_q: None, encrypted: None, cluster_size: None, disk_capacity: None, max_expandable_size: None })
    in kube_runtime::controller::reconciling object with object.ref: DiskPool.v1beta3.openebs.io/pool-1.openebs, object.reason: object updated

  2026-05-02T16:31:57.397490Z ERROR operator_diskpool::context: Pool not found, clearing status, name: "pool-2"
    at k8s/operators/src/pool/context.rs:238
    in operator_diskpool::context::pool_check with name: "pool-2", status: Some(DiskPoolStatus { cr_state: Created, pool_status: Some(Online), capacity: 1918504009728, used: 1010827264, available: 1917493182464, capacity_q: Some(Quantity(1869169767219)), used_q: Some(Quantity(1010827264)), available_q: Some(Quantity(1869169767219)), encrypted: Some(false), cluster_size: Some(Quantity(4194304)), disk_capacity: Some(Quantity(1869169767219)), max_expandable_size: Some(Quantity(1869169767219)) })
    in operator_diskpool::reconcile with name: k8s.hostname.tld, status: Some(DiskPoolStatus { cr_state: Created, pool_status: Some(Online), capacity: 1918504009728, used: 1010827264, available: 1917493182464, capacity_q: Some(Quantity(1869169767219)), used_q: Some(Quantity(1010827264)), available_q: Some(Quantity(1869169767219)), encrypted: Some(false), cluster_size: Some(Quantity(4194304)), disk_capacity: Some(Quantity(1869169767219)), max_expandable_size: Some(Quantity(1869169767219)) })
    in kube_runtime::controller::reconciling object with object.ref: DiskPool.v1beta3.openebs.io/pool-2.openebs, object.reason: reconciler requested retry

  2026-05-02T16:31:57.398726Z  INFO operator_diskpool::context: new resource_version inserted, name: "pool-2"
    at k8s/operators/src/pool/context.rs:134
    in operator_diskpool::reconcile with name: k8s.hostname.tld, status: Some(DiskPoolStatus { cr_state: Created, pool_status: None, capacity: 0, used: 0, available: 0, capacity_q: None, used_q: None, available_q: None, encrypted: None, cluster_size: None, disk_capacity: None, max_expandable_size: None })
    in kube_runtime::controller::reconciling object with object.ref: DiskPool.v1beta3.openebs.io/pool-2.openebs, object.reason: object updated

[lahwaacz]

It seems that the api-rest pod is being restarted a lot:

Warning  BackOff    36m (x28 over 27h)     kubelet  spec.containers{api-rest}: Back-off restarting failed container api-rest in pod openebs-api-rest-7d586fcd5d-gd8dz_openebs(f6b33b23-0f37-44b4-8ce5-d19f6fc634e2)
  Normal   Pulled     12m (x55 over 28h)     kubelet  spec.containers{api-rest}: Container image "docker.io/openebs/mayastor-api-rest:v2.10.0" already present on machine and can be accessed by the pod
  Normal   Created    12m (x55 over 28h)     kubelet  spec.containers{api-rest}: Container created
  Warning  Unhealthy  12m (x44 over 28h)     kubelet  spec.containers{api-rest}: Readiness probe failed: Get "http://10.240.10.90:8081/ready": dial tcp 10.240.10.90:8081: connect: connection refused
  Warning  Unhealthy  3m44s (x134 over 28h)  kubelet  spec.containers{api-rest}: Readiness probe failed: Get "http://10.240.10.90:8081/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  2m (x57 over 28h)      kubelet  spec.containers{api-rest}: Liveness probe failed: Get "http://10.240.10.90:8081/live": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Normal   Killing    2m (x57 over 28h)      kubelet  spec.containers{api-rest}: Container api-rest failed liveness probe, will be restarted 

[lahwaacz]

So I just reconfigured the health probes and set replicaCount: 2 for the rest-api component.

[tiagolobocastro]

Would you be able to try the current develop version in a non-prod cluster? We've fixed a couple of issues, though I think there's still some problems when adding and removing a diskpool that may show up as temporary blips.

As for the rest health probe, could you share a describe of the deployment to see if there's any more information?

[lahwaacz]

I can't easily test it on a second cluster, but the deployment looks like this (after I've increased the replicaCount):

$ kubectl --namespace openebs describe deployments.apps openebs-api-rest
Name:                   openebs-api-rest
Namespace:              openebs
CreationTimestamp:      Sat, 02 May 2026 00:11:56 +0200
Labels:                 app=api-rest
                        app.kubernetes.io/managed-by=Helm
                        openebs.io/release=openebs
                        openebs.io/version=2.10.0
Annotations:            deployment.kubernetes.io/revision: 6
                        meta.helm.sh/release-name: openebs
                        meta.helm.sh/release-namespace: openebs
Selector:               app=api-rest,openebs.io/release=openebs
Replicas:               2 desired | 2 updated | 2 total | 1 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=api-rest
           openebs.io/logging=true
           openebs.io/release=openebs
           openebs.io/version=2.10.0
  Init Containers:
   agent-core-grpc-probe:
    Image:      docker.io/openebs/alpine-sh:4.3.0
    Port:       <none>
    Host Port:  <none>
    Command:
      sh
      -c
      trap "exit 1" TERM; until nc -vzw 5 openebs-agent-core 50051; do date; echo "Waiting for agent-core-grpc services..."; sleep 1; done;
    Environment:  <none>
    Mounts:       <none>
   etcd-probe:
    Image:      docker.io/openebs/alpine-sh:4.3.0
    Port:       <none>
    Host Port:  <none>
    Command:
      sh
      -c
      trap "exit 1" TERM; until nc -vzw 5 openebs-etcd:2379 2379; do date; echo "Waiting for etcd..."; sleep 1; done;
    Environment:  <none>
    Mounts:       <none>
  Containers:
   api-rest:
    Image:       docker.io/openebs/mayastor-api-rest:v2.10.0
    Ports:       8080/TCP, 8081/TCP
    Host Ports:  0/TCP, 0/TCP
    Args:
      --dummy-certificates
      --no-auth
      --http=[::]:8081
      --request-timeout=5s
      --core-grpc=https://openebs-agent-core:50051
      --ansi-colors=true
      --fmt-style=pretty
      --core-health-freq=20s
    Limits:
      cpu:     100m
      memory:  64Mi
    Requests:
      cpu:      50m
      memory:   32Mi
    Liveness:   http-get http://:8081/live delay=0s timeout=10s period=20s #success=1 #failure=3
    Readiness:  http-get http://:8081/ready delay=0s timeout=10s period=20s #success=1 #failure=2
    Environment:
      RUST_LOG:         info
    Mounts:             <none>
  Volumes:              <none>
  Priority Class Name:  openebs-cluster-critical
  Node-Selectors:       kubernetes.io/arch=amd64
  Tolerations:          node.kubernetes.io/not-ready:NoExecute op=Exists for 5s
                        node.kubernetes.io/unreachable:NoExecute op=Exists for 5s
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      False   MinimumReplicasUnavailable
OldReplicaSets:  openebs-api-rest-5f75dd7659 (0/0 replicas created), openebs-api-rest-7d586fcd5d (0/0 replicas created), openebs-api-rest-6747bc5ff6 (0/0 replicas created), openebs-api-rest-7dc6d4dbbd (0/0 replicas created)
NewReplicaSet:   openebs-api-rest-7f896c94bc (2/2 replicas created)
Events:          <none>

In the logs I see timeout-related errors for api-rest, csi-controller, csi-node, and operator-diskpool. I can't tell what is the root cause as the logs don't say where the pod is connecting to. When it happens, some DiskPools go temporarily missing in kubectl get diskpools.openebs.io and PVC operations get stalled.

Increasing the replicaCount seems to have helped, but not 100%. Maybe I need to switch to debug log level to see more details...

[lahwaacz]

I found this with kubectl --namespace openebs logs --all-containers deployments/openebs-agent-core (note that --all-containers is critical):

  2026-05-05T19:23:46.386718Z  INFO agent_ha_cluster::switchover: Sending failed Switchover request back to the work queue, volume.uuid: eaf4217e-5ebe-4106-989d-ecb40aed1c45, error: Nvme path replacement failed: Unavailable: status: Unavailable, message: "tcp connect error: No route to host (os error 113)", details: [], metadata: MetadataMap { headers: {} }
    at control-plane/agents/src/bin/ha/cluster/switchover.rs:573

  2026-05-05T19:23:46.387634Z  INFO agent_ha_cluster::switchover: Sending new volume target to node agent, volume.uuid: afb91cb0-8439-4ef7-a331-2ad8d8eb5f71
    at control-plane/agents/src/bin/ha/cluster/switchover.rs:408

  2026-05-05T19:23:46.387655Z  INFO agent_ha_cluster::switchover: Creating node agent client using callback uri, uri: http://x.x.x.x:50053/
    at control-plane/agents/src/bin/ha/cluster/switchover.rs:410

  2026-05-05T19:23:46.387914Z ERROR grpc::operations::ha_node::client: error: Unavailable: status: Unavailable, message: "tcp connect error: No route to host (os error 113)", details: [], metadata: MetadataMap { headers: {} }
    at control-plane/grpc/src/operations/ha_node/client.rs:99
    in grpc::operations::ha_node::client::NodeAgentClient::replace_path with request: ReplacePath { target_nqn: "nqn.2019-05.io.openebs:afb91cb0-8439-4ef7-a331-2ad8d8eb5f71", new_path: "nvmf://x.x.x.x:8420/nqn.2019-05.io.openebs:afb91cb0-8439-4ef7-a331-2ad8d8eb5f71?hostnqn=nqn.2019-05.io.openebs:node-name:k8s.fjfi.cvut.cz", publish_context: Some({}) }, context: None

So the agent-core component is connecting to the host at http://x.x.x.x:50053 which is not documented in https://openebs.io/docs/quickstart-guide/prerequisites#network-requirements

[lahwaacz]

After enabling port 50053 in the firewall on the host, it seems to work well 🎉 I will continue monitoring it for further issues.

[lahwaacz]

That was not all, I also had to increase resources for the api-rest component: with only 100m CPU / 64Mi RAM it was being restarted frequently and this was likely the root issue. https://github.com/openebs/mayastor-extensions/blob/v2.10.0/chart/values.yaml#L294-L304