requesting a new Mayastor PVC, error related to a comparison function.

[Benjamin-Ast]

Hello,

Describe the bug

When requesting a new Mayastor PVC, the PVC remains in Pending state.
We encounter an error related to a comparison function in the Rust Code .

Logs :

failed to provision volume with StorageClass "mayastor-repl2": rpc error: code = Unavailable desc = error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: "Failed to
wait for thread", message: "task 36179 panicked with message "user-provided comparison function does not correctly implement a total order"", kind: Aborted }'

To Reproduce
Create (or apply) a new PVC.

Expected behavior
The PVC should be successfully bound and reach the Bound state without any errors.

Mayastor setup

Events:
  Type     Reason                Age              From                                                                            Message
 
  Warning  ProvisioningFailed    6s               io.openebs.csi-mayastor_prd-app-kub-wsl12_160dff92-82a9-4ad0-bea0-b3f2cf2a2f91  failed to provision volume with StorageClass "mayastor-repl1": rpc error: code = Unavailable desc = error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: "Failed to wait for thread", message: "task 40900 panicked with message \"user-provided comparison function does not correctly implement a total order\"", kind: Aborted }'
  Normal   Provisioning          3s (x3 over 6s)  io.openebs.csi-mayastor_prd-app-kub-wsl12_160dff92-82a9-4ad0-bea0-b3f2cf2a2f91  External provisioner is provisioning volume for claim "mayastor/pvc-test-4"
  Normal   ExternalProvisioning  3s (x3 over 6s)  persistentvolume-controller                                                     Waiting for a volume to be created either by the external provisioner 'io.openebs.csi-mayastor' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Warning  ProvisioningFailed    3s (x2 over 5s)  io.openebs.csi-mayastor_prd-app-kub-wsl12_160dff92-82a9-4ad0-bea0-b3f2cf2a2f91  failed to provision volume with StorageClass "mayastor-repl1": rpc error: code = Aborted desc = Existing volume 9f6e0192-3d61-4694-be59-d7535666189c is in insufficient state: Creating

pool mayastor :

NAMESPACE   NAME                     NODE                STATE     POOL_STATUS   ENCRYPTED   CAPACITY   USED        AVAILABLE
mayastor    pool-prd-app-kub-wsf11   prd-app-kub-wsf11   Created   Online        false       1.7 TiB    59.1 GiB    1.7 TiB
mayastor    pool-prd-app-kub-wsf12   prd-app-kub-wsf12   Created   Online        false       1.7 TiB    120 GiB     1.6 TiB
mayastor    pool-prd-app-kub-wsf13   prd-app-kub-wsf13   Created   Online        false       1.7 TiB    68 GiB      1.7 TiB
mayastor    pool-prd-app-kub-wsf14   prd-app-kub-wsf14   Created   Online        false       1.7 TiB    75.9 GiB    1.7 TiB
mayastor    pool-prd-app-kub-wsf15   prd-app-kub-wsf15   Created   Online        false       1.7 TiB    83 GiB      1.7 TiB
mayastor    pool-prd-app-kub-wsf16   prd-app-kub-wsf16   Created   Online        false       1.7 TiB    53.3 GiB    1.7 TiB
mayastor    pool-prd-app-kub-wsf17   prd-app-kub-wsf17   Created   Online        false       1.7 TiB    229 GiB     1.5 TiB
mayastor    pool-prd-app-kub-wsf18   prd-app-kub-wsf18   Created   Online        false       1.7 TiB    120.7 GiB   1.6 TiB
mayastor    pool-prd-app-kub-wsf19   prd-app-kub-wsf19   Created   Online        false       1.7 TiB    61.8 GiB    1.7 TiB
mayastor    pool-prd-app-kub-wsf20   prd-app-kub-wsf20   Created   Online        false       1.7 TiB    116.1 GiB   1.6 TiB
mayastor    pool-prd-app-kub-wsf21   prd-app-kub-wsf21   Created   Online        false       1.7 TiB    54.8 GiB    1.7 TiB
mayastor    pool-prd-app-kub-wsf22   prd-app-kub-wsf22   Created   Online        false       1.7 TiB    125.8 GiB   1.6 TiB
mayastor    pool-prd-app-kub-wsf23   prd-app-kub-wsf23   Created   Online        false       1.7 TiB    133.8 GiB   1.6 TiB
mayastor    pool-prd-app-kub-wsf24   prd-app-kub-wsf24   Created   Online        false       1.7 TiB    120.9 GiB   1.6 TiB
mayastor    pool-prd-app-kub-wsf25   prd-app-kub-wsf25   Created   Online        false       1.7 TiB    103.2 GiB   1.6 TiB
mayastor    pool-prd-app-kub-wsf26   prd-app-kub-wsf26   Created   Online        false       1.7 TiB    57.4 GiB    1.7 TiB
mayastor    pool-prd-app-kub-wsf27   prd-app-kub-wsf27   Created   Online        false       1.7 TiB    168.8 GiB   1.6 TiB
mayastor    pool-prd-app-kub-wsf28   prd-app-kub-wsf28   Created   Online        false       1.7 TiB    55.6 GiB    1.7 TiB
mayastor    pool-prd-app-kub-wsf29   prd-app-kub-wsf29   Created   Online        false       1.7 TiB    103.6 GiB   1.6 TiB
mayastor    pool-prd-app-kub-wsf30   prd-app-kub-wsf30   Created   Online        false       1.7 TiB    110.7 GiB   1.6 TiB
mayastor    pool-prd-app-kub-wsf31   prd-app-kub-wsf31   Created   Online        false       1.7 TiB    50 GiB      1.7 TiB
mayastor    pool-prd-app-kub-wsf32   prd-app-kub-wsf32   Created   Online        false       1.7 TiB    132.2 GiB   1.6 TiB
mayastor    pool-prd-app-kub-wsf33   prd-app-kub-wsf33   Created   Online        false       1.7 TiB    126.5 GiB   1.6 TiB
mayastor    pool-prd-app-kub-wsf34   prd-app-kub-wsf34   Created   Online        false       1.7 TiB    61.9 GiB    1.7 TiB

stern 'ac-chart-mayastor-agent-(core|ha-node)' -n mayastor --max-log-requests 100 | grep -Ei 'panic|fail|error|warn|backtrace'

ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core stack backtrace:
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core    0:     0x561aefc3b0ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core    3:     0x561aefc3cbd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core    4:     0x561aefc3c83e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core    5:     0x561aefc3d4cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core    6:     0x561aefc3d183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core    7:     0x561aefc3b5a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core    9:     0x561aefc67173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core   10:     0x561aefc6bbdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-5c4df49757-k22jd agent-core   31:     0x561aefbda94e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f

kmay get pools --show-labels

ID                      DISKS                                                                                  MANAGED  NODE               STATUS  CAPACITY  ALLOCATED  AVAILABLE  COMMITTED  ENCRYPTED  LABELS
pool-prd-app-kub-wsf27  aio:///dev/disk/by-id/virtio-mayastor_wsf27?uuid=9893fb1d-aaa2-49c0-8a24-d6f94f1c1210  true     prd-app-kub-wsf27  Online  1.7 TiB   168.8 GiB  1.6 TiB    168.8 GiB  false      node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf20  aio:///dev/disk/by-id/virtio-mayastor_wsf20?uuid=5f319944-feb3-4761-b80c-693149e6f59a  true     prd-app-kub-wsf20  Online  1.7 TiB   116.1 GiB  1.6 TiB    116.1 GiB  false      node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf26  aio:///dev/disk/by-id/virtio-mayastor_wsf26?uuid=d600f886-340a-45a1-9076-6c61f3057af5  true     prd-app-kub-wsf26  Online  1.7 TiB   57.4 GiB   1.7 TiB    57.4 GiB   false      node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf34  aio:///dev/disk/by-id/virtio-mayastor_wsf34?uuid=8b9ff5ae-8a5c-4f23-99c8-99de71d85a11  true     prd-app-kub-wsf34  Online  1.7 TiB   61.9 GiB   1.7 TiB    61.9 GiB   false      node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf25  aio:///dev/disk/by-id/virtio-mayastor_wsf25?uuid=943b4ee3-6992-4d69-921d-f39711b33c6c  true     prd-app-kub-wsf25  Online  1.7 TiB   103.2 GiB  1.6 TiB    103.2 GiB  false      node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf30  aio:///dev/disk/by-id/virtio-mayastor_wsf30?uuid=73ee0168-73a4-4414-9dee-d66328e4446a  true     prd-app-kub-wsf30  Online  1.7 TiB   110.7 GiB  1.6 TiB    110.7 GiB  false      node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf11  aio:///dev/disk/by-id/virtio-mayastor_wsf11?uuid=e86ee600-6a39-48bc-9fd6-68da14ac17a0  true     prd-app-kub-wsf11  Online  1.7 TiB   59.1 GiB   1.7 TiB    59.1 GiB   false      node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf24  aio:///dev/disk/by-id/virtio-mayastor_wsf24?uuid=6100f580-a7fe-4dc2-ad78-50dab9cb9d91  true     prd-app-kub-wsf24  Online  1.7 TiB   120.9 GiB  1.6 TiB    120.9 GiB  false      node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf28  aio:///dev/disk/by-id/virtio-mayastor_wsf28?uuid=d2ad2f3d-5b6a-49ad-9ab8-5d7f3526d547  true     prd-app-kub-wsf28  Online  1.7 TiB   55.6 GiB   1.7 TiB    55.6 GiB   false      node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf17  aio:///dev/disk/by-id/virtio-mayastor_wsf17?uuid=1aee9514-ac83-4b2a-bde4-5bc599dba07e  true     prd-app-kub-wsf17  Online  1.7 TiB   229 GiB    1.5 TiB    229 GiB    false      node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf12  aio:///dev/disk/by-id/virtio-mayastor_wsf12?uuid=85c2d5f5-9fde-4ca2-8e2f-05bf743d0537  true     prd-app-kub-wsf12  Online  1.7 TiB   120 GiB    1.6 TiB    120 GiB    false      node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf31  aio:///dev/disk/by-id/virtio-mayastor_wsf31?uuid=e4d9e618-2347-40be-96d8-b891118cb12e  true     prd-app-kub-wsf31  Online  1.7 TiB   50 GiB     1.7 TiB    50 GiB     false      node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf14  aio:///dev/disk/by-id/virtio-mayastor_wsf14?uuid=c2ccb31e-b538-4dc2-89ef-dd990d4fe153  true     prd-app-kub-wsf14  Online  1.7 TiB   75.9 GiB   1.7 TiB    75.9 GiB   false      node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf16  aio:///dev/disk/by-id/virtio-mayastor_wsf16?uuid=3ace06e5-3c90-4bf2-a0d0-92fce9594932  true     prd-app-kub-wsf16  Online  1.7 TiB   53.3 GiB   1.7 TiB    53.3 GiB   false      node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf32  aio:///dev/disk/by-id/virtio-mayastor_wsf32?uuid=16053a58-57be-40d1-8f71-c93bd3e364a5  true     prd-app-kub-wsf32  Online  1.7 TiB   132.2 GiB  1.6 TiB    132.2 GiB  false      node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf33  aio:///dev/disk/by-id/virtio-mayastor_wsf33?uuid=8e43a15b-8c4c-4335-be96-eba1dc25980f  true     prd-app-kub-wsf33  Online  1.7 TiB   126.5 GiB  1.6 TiB    126.5 GiB  false      node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf13  aio:///dev/disk/by-id/virtio-mayastor_wsf13?uuid=38a11fd4-f589-4018-889e-36731bccd568  true     prd-app-kub-wsf13  Online  1.7 TiB   68 GiB     1.7 TiB    68 GiB     false      node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf18  aio:///dev/disk/by-id/virtio-mayastor_wsf18?uuid=37d34532-48e8-47ae-997f-dc93d5ab2741  true     prd-app-kub-wsf18  Online  1.7 TiB   120.7 GiB  1.6 TiB    120.7 GiB  false      node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf21  aio:///dev/disk/by-id/virtio-mayastor_wsf21?uuid=8f44cb99-b74e-4be6-9361-f7bccc9855c4  true     prd-app-kub-wsf21  Online  1.7 TiB   54.8 GiB   1.7 TiB    54.8 GiB   false      node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf29  aio:///dev/disk/by-id/virtio-mayastor_wsf29?uuid=6261ea27-0bd2-44e8-93ad-df0655ae4b20  true     prd-app-kub-wsf29  Online  1.7 TiB   103.6 GiB  1.6 TiB    103.6 GiB  false      node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf22  aio:///dev/disk/by-id/virtio-mayastor_wsf22?uuid=9ff65f3d-96e5-43fb-9fb9-5127c9dafa43  true     prd-app-kub-wsf22  Online  1.7 TiB   125.8 GiB  1.6 TiB    125.8 GiB  false      node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf23  aio:///dev/disk/by-id/virtio-mayastor_wsf23?uuid=3c7d7bec-4ed1-4590-8ede-39ea31bbc343  true     prd-app-kub-wsf23  Online  1.7 TiB   133.8 GiB  1.6 TiB    133.8 GiB  false      node.k8s.toto/datacenter=datacenter1
pool-prd-app-kub-wsf15  aio:///dev/disk/by-id/virtio-mayastor_wsf15?uuid=d941064d-efeb-49a7-86bd-b12798265805  true     prd-app-kub-wsf15  Online  1.7 TiB   83 GiB     1.7 TiB    83 GiB     false      node.k8s.toto/datacenter=datacenter2
pool-prd-app-kub-wsf19  aio:///dev/disk/by-id/virtio-mayastor_wsf19?uuid=c3508212-b080-48c8-a0ad-2d4ba26cd12e  true     prd-app-kub-wsf19  Online  1.7 TiB   61.8 GiB   1.7 TiB    61.8 GiB   false      node.k8s.toto/datacenter=datacenter2

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
  labels:
    argocd.argoproj.io/instance: ac-chart-mayastor
  name: mayastor-repl1
parameters:
  repl: "1"
provisioner: io.openebs.csi-mayastor
reclaimPolicy: Delete
volumeBindingMode: Immediate


allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
  labels:
    argocd.argoproj.io/instance: ac-chart-mayastor
  name: mayastor-repl2
parameters:
  poolHasTopologyKey: |
    node.k8s.toto/datacenter
  repl: "2"
provisioner: io.openebs.csi-mayastor
reclaimPolicy: Delete
volumeBindingMode: Immediate

Thank you for your help.

[TheoCouss]

@tiagolobocastro is it related to ./io-engine/src/core/nic.rs:227 ?

[tiagolobocastro]

No, this is on the control-plane.

Is there any more backtrace information which would point to what type is at fault?

[kiliandsc]

Thanks for your reply. Here’s some clarification and the current status:

We were able to successfully create almost 220 PVCs without any issues, but the problem suddenly appeared and we've been unable to find a solution since.

Our etcd instances are running on data-plane nodes, not on control-plane nodes.
→ Our control-plane nodes are hosted on PPCe-based machines (PowerPC), which are not compatible with Mayastor’s requirements, so we run etcd on the kvm nodes (lvm) (x86_64 with large CPU and RAM capacity).

In another environment where we also run Mayastor (same version), we don’t use any topology or node/pool labels at all — no affinity filters, no inclusion, no exclusion.

In this environment, everything is working perfectly, including:
Provisioning with replica count = 2
Fast and stable volume creation
No panics, no sorting-related errors, no delays

In production env

We are using label and topology , we can see on the etcd pod the datacenter label is empty

"Created":"Online"},"policy":{"self_heal":true},"topology":{"node":{"Labelled":{"exclusion":{},"inclusion":{},"creation_affinity_keys":[]}},"pool":{"Labelled":{"exclusion":{},"inclusion":{"openebs.io/created-by":"operator-diskpool","node.k8s.toto/datacenter":""}

"node.k8s.toto/datacenter":""

k get pod -o wide
NAME                                                     READY   STATUS    RESTARTS       AGE     IP               NODE                NOMINATED NODE   READINESS GATES
agent-core-7cd7969d95-4vkrf            2/2     Running   0              114m    10.233.71.149    prd-app-kub-wsl11   <none>           <none>
agent-ha-node-48qhl                    1/1     Running   0              109m    xxx.xx.xxx.225   prd-app-kub-wsf25   <none>           <none>
agent-ha-node-89ft4                    1/1     Running   0              114m    xxx.xx.xxx.231   prd-app-kub-wsf31   <none>           <none>
agent-ha-node-8scxk                    1/1     Running   0              108m    xxx.xx.xxx.222   prd-app-kub-wsf22   <none>           <none>
agent-ha-node-c9mr5                    1/1     Running   0              107m    xxx.xx.xxx.234   prd-app-kub-wsf34   <none>           <none>
agent-ha-node-cqdcb                    1/1     Running   0              111m    xxx.xx.xxx.224   prd-app-kub-wsf24   <none>           <none>
agent-ha-node-d2r7n                    1/1     Running   0              108m    xxx.xx.xxx.215   prd-app-kub-wsf15   <none>           <none>
agent-ha-node-d9khw                    1/1     Running   0              113m    xxx.xx.xxx.226   prd-app-kub-wsf26   <none>           <none>
agent-ha-node-ffm7w                    1/1     Running   0              114m    xxx.xx.xxx.223   prd-app-kub-wsf23   <none>           <none>
agent-ha-node-fmshc                    1/1     Running   0              106m    xxx.xx.xxx.217   prd-app-kub-wsf17   <none>           <none>
agent-ha-node-fphb8                    1/1     Running   0              112m    xxx.xx.xxx.211   prd-app-kub-wsf11   <none>           <none>
agent-ha-node-g74l4                    1/1     Running   0              106m    xxx.xx.xxx.229   prd-app-kub-wsf29   <none>           <none>
agent-ha-node-hth6c                    1/1     Running   0              107m    xxx.xx.xxx.230   prd-app-kub-wsf30   <none>           <none>
agent-ha-node-klpnw                    1/1     Running   0              110m    xxx.xx.xxx.216   prd-app-kub-wsf16   <none>           <none>
agent-ha-node-kswlb                    1/1     Running   0              110m    xxx.xx.xxx.228   prd-app-kub-wsf28   <none>           <none>
agent-ha-node-nk7dg                    1/1     Running   0              105m    xxx.xx.xxx.219   prd-app-kub-wsf19   <none>           <none>
agent-ha-node-nl5n7                    1/1     Running   0              110m    xxx.xx.xxx.232   prd-app-kub-wsf32   <none>           <none>
agent-ha-node-pgm94                    1/1     Running   0              106m    xxx.xx.xxx.221   prd-app-kub-wsf21   <none>           <none>
agent-ha-node-ppmm6                    1/1     Running   0              108m    xxx.xx.xxx.213   prd-app-kub-wsf13   <none>           <none>
agent-ha-node-qwlnp                    1/1     Running   0              113m    xxx.xx.xxx.227   prd-app-kub-wsf27   <none>           <none>
agent-ha-node-s8zl9                    1/1     Running   0              108m    xxx.xx.xxx.233   prd-app-kub-wsf33   <none>           <none>
agent-ha-node-snnlt                    1/1     Running   0              113m    xxx.xx.xxx.214   prd-app-kub-wsf14   <none>           <none>
agent-ha-node-vrbzg                    1/1     Running   0              111m    xxx.xx.xxx.212   prd-app-kub-wsf12   <none>           <none>
agent-ha-node-w69xj                    1/1     Running   0              108m    xxx.xx.xxx.218   prd-app-kub-wsf18   <none>           <none>
agent-ha-node-wqw4n                    1/1     Running   0              113m    xxx.xx.xxx.220   prd-app-kub-wsf20   <none>           <none>
api-rest-6654667675-ntl69              1/1     Running   0              114m    10.233.97.0      prd-app-kub-wsl30   <none>           <none>
csi-controller-7cd9c66c69-frw7f        6/6     Running   0              114m    xxx.xx.xxx.125   prd-app-kub-wsl25   <none>           <none>
csi-node-2vb65                         2/2     Running   0              6h14m   xxx.xx.xxx.228   prd-app-kub-wsf28   <none>           <none>
csi-node-42xgc                         2/2     Running   0              6h13m   xxx.xx.xxx.224   prd-app-kub-wsf24   <none>           <none>
csi-node-479wn                         2/2     Running   0              6h19m   xxx.xx.xxx.211   prd-app-kub-wsf11   <none>           <none>
csi-node-55hg4                         2/2     Running   0              6h12m   xxx.xx.xxx.212   prd-app-kub-wsf12   <none>           <none>
csi-node-6j45r                         2/2     Running   0              6h21m   xxx.xx.xxx.218   prd-app-kub-wsf18   <none>           <none>
csi-node-9srbx                         2/2     Running   0              4h58m   xxx.xx.xxx.216   prd-app-kub-wsf16   <none>           <none>
csi-node-c57sj                         2/2     Running   0              112m    xxx.xx.xxx.231   prd-app-kub-wsf31   <none>           <none>
csi-node-djbb8                         2/2     Running   0              6h20m   xxx.xx.xxx.227   prd-app-kub-wsf27   <none>           <none>
csi-node-dwknp                         2/2     Running   0              6h12m   xxx.xx.xxx.226   prd-app-kub-wsf26   <none>           <none>
csi-node-f7hqr                         2/2     Running   0              6h15m   xxx.xx.xxx.217   prd-app-kub-wsf17   <none>           <none>
csi-node-gjtbz                         2/2     Running   0              6h12m   xxx.xx.xxx.232   prd-app-kub-wsf32   <none>           <none>
csi-node-gw2tt                         2/2     Running   0              6h18m   xxx.xx.xxx.230   prd-app-kub-wsf30   <none>           <none>
csi-node-gwq6b                         2/2     Running   0              6h18m   xxx.xx.xxx.233   prd-app-kub-wsf33   <none>           <none>
csi-node-jz87z                         2/2     Running   0              6h15m   xxx.xx.xxx.214   prd-app-kub-wsf14   <none>           <none>
csi-node-kdg47                         2/2     Running   0              4h31m   xxx.xx.xxx.215   prd-app-kub-wsf15   <none>           <none>
csi-node-kq7rb                         2/2     Running   0              6h12m   xxx.xx.xxx.219   prd-app-kub-wsf19   <none>           <none>
csi-node-lmp5w                         2/2     Running   0              6h15m   xxx.xx.xxx.213   prd-app-kub-wsf13   <none>           <none>
csi-node-msrw5                         2/2     Running   0              6h21m   xxx.xx.xxx.225   prd-app-kub-wsf25   <none>           <none>
csi-node-pkckj                         2/2     Running   0              6h16m   xxx.xx.xxx.220   prd-app-kub-wsf20   <none>           <none>
csi-node-psqwx                         2/2     Running   0              6h14m   xxx.xx.xxx.234   prd-app-kub-wsf34   <none>           <none>
csi-node-r4tlh                         2/2     Running   0              4h15m   xxx.xx.xxx.229   prd-app-kub-wsf29   <none>           <none>
csi-node-tbtdf                         2/2     Running   0              6h14m   xxx.xx.xxx.221   prd-app-kub-wsf21   <none>           <none>
csi-node-tl692                         2/2     Running   0              6h16m   xxx.xx.xxx.223   prd-app-kub-wsf23   <none>           <none>
csi-node-tzhdv                         2/2     Running   0              6h18m   xxx.xx.xxx.222   prd-app-kub-wsf22   <none>           <none>
etcd-0                                 1/1     Running   0              110m    10.233.114.148   prd-app-kub-wsl20   <none>           <none>
etcd-1                                 1/1     Running   0              111m    10.233.89.239    prd-app-kub-wsl22   <none>           <none>
etcd-2                                 1/1     Running   0              112m    10.233.116.166   prd-app-kub-wsl34   <none>           <none>
io-engine-42c4x                        2/2     Running   0              7h55m   xxx.xx.xxx.234   prd-app-kub-wsf34   <none>           <none>
io-engine-4t7h7                        2/2     Running   0              8h      xxx.xx.xxx.218   prd-app-kub-wsf18   <none>           <none>
io-engine-4x8r2                        2/2     Running   0              8h      xxx.xx.xxx.227   prd-app-kub-wsf27   <none>           <none>
io-engine-5lvm7                        2/2     Running   0              7h57m   xxx.xx.xxx.230   prd-app-kub-wsf30   <none>           <none>
io-engine-5qxgw                        2/2     Running   0              8h      xxx.xx.xxx.221   prd-app-kub-wsf21   <none>           <none>
io-engine-75c7j                        2/2     Running   0              7h56m   xxx.xx.xxx.232   prd-app-kub-wsf32   <none>           <none>
io-engine-7jskm                        2/2     Running   0              4h31m   xxx.xx.xxx.215   prd-app-kub-wsf15   <none>           <none>
io-engine-8brfn                        2/2     Running   0              8h      xxx.xx.xxx.223   prd-app-kub-wsf23   <none>           <none>
io-engine-9sh6t                        2/2     Running   0              8h      xxx.xx.xxx.217   prd-app-kub-wsf17   <none>           <none>
io-engine-bgzxv                        2/2     Running   0              5h14m   xxx.xx.xxx.212   prd-app-kub-wsf12   <none>           <none>
io-engine-c58gb                        2/2     Running   0              8h      xxx.xx.xxx.219   prd-app-kub-wsf19   <none>           <none>
io-engine-cbfjg                        2/2     Running   0              4h16m   xxx.xx.xxx.229   prd-app-kub-wsf29   <none>           <none>
io-engine-czprx                        2/2     Running   0              8h      xxx.xx.xxx.225   prd-app-kub-wsf25   <none>           <none>
io-engine-f45zh                        2/2     Running   0              7h55m   xxx.xx.xxx.233   prd-app-kub-wsf33   <none>           <none>
io-engine-fh28x                        2/2     Running   0              8h      xxx.xx.xxx.222   prd-app-kub-wsf22   <none>           <none>
io-engine-jgx4t                        2/2     Running   0              8h      xxx.xx.xxx.224   prd-app-kub-wsf24   <none>           <none>
io-engine-jp999                        2/2     Running   0              8h      xxx.xx.xxx.228   prd-app-kub-wsf28   <none>           <none>
io-engine-pk59x                        2/2     Running   0              5h12m   xxx.xx.xxx.214   prd-app-kub-wsf14   <none>           <none>
io-engine-pnf4h                        2/2     Running   0              5h13m   xxx.xx.xxx.213   prd-app-kub-wsf13   <none>           <none>
io-engine-pstnd                        2/2     Running   0            7h56m   xxx.xx.xxx.231   prd-app-kub-wsf31   <none>           <none>
io-engine-t4s9b                        2/2     Running   0              5h      xxx.xx.xxx.216   prd-app-kub-wsf16   <none>           <none>
io-engine-vb92z                        2/2     Running   0              5h27m   xxx.xx.xxx.211   prd-app-kub-wsf11   <none>           <none>
io-engine-wlh9m                        2/2     Running   0              8h      xxx.xx.xxx.226   prd-app-kub-wsf26   <none>           <none>
io-engine-znlvj                        2/2     Running   0              8h      xxx.xx.xxx.220   prd-app-kub-wsf20   <none>           <none>
localpv-provisioner-55666776f8-pf94d   1/1     Running   0              4d7h    10.233.98.251    prd-app-kub-wsl32   <none>           <none>
obs-callhome-7f95bd4d44-nvvjx          1/1     Running   0              114m    10.233.77.54     prd-app-kub-wsl27   <none>           <none>
operator-diskpool-854f766ccd-6zdv2     1/1     Running   0              6h21m   10.233.107.51    prd-app-kub-wsf16   <none>           <none>

agent-core

ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    0:     0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    3:     0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    4:     0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    5:     0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    6:     0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    7:     0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    9:     0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   10:     0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   31:     0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f

csi-controller

ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller   2025-07-22T16:55:18.096743Z  INFO csi_driver::trace: [ CSI ] Create Volume Request started
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller     at control-plane/csi-driver/src/trace.rs:10
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller     in csi_controller::controller::create_volume with request: Request { metadata: MetadataMap { headers: {"content-type": "application/grpc", "user-agent": "grpc-go/1.69.2", "te": "trailers", "grpc-timeout": "35999957u"} }, message: CreateVolumeRequest { name: "pvc-dbafed96-a78f-4d48-9a0a-9e190e081353", capacity_range: Some(CapacityRange { required_bytes: 2000000000, limit_bytes: 0 }), volume_capabilities: [VolumeCapability { access_mode: Some(AccessMode { mode: SingleNodeWriter }), access_type: Some(Mount(MountVolume { fs_type: "ext4", mount_flags: [], volume_mount_group: "" })) }], parameters: {"protocol": "nvmf", "ioTimeout": "30", "csi.storage.k8s.io/pvc/namespace": "toto-cloud--interne--prd--glpi-nginx--arch", "csi.storage.k8s.io/pv/name": "pvc-dbafed96-a78f-4d48-9a0a-9e190e081353", "poolHasTopologyKey": "node.k8s.toto/datacenter", "csi.storage.k8s.io/pvc/name": "data-mariadb-0", "repl": "2"}, secrets: {}, volume_content_source: None, accessibility_requirements: Some(TopologyRequirement { requisite: [Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf11", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf12"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf13", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf14"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf15"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf16", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf17"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf18"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf19"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf20", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf21"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf22", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf23", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf24", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf25", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf26"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf27"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf28", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf29"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf30", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf31"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf32", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf33"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf34"} }], preferred: [Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf22", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf23"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf24"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf25"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf26", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf27"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf28"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf29"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf30", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf31", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf32", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf33"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf34", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf11", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf12"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf13", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf14", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf15", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf16", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf17"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf18", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf19"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf20"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf21"} }] }), mutable_parameters: {} }, extensions: Extensions }, volume.uuid: "dbafed96-a78f-4d48-9a0a-9e190e081353"
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller   2025-07-22T16:55:18.100878Z ERROR csi_controller::controller: error: status: AlreadyExists, message: "Existing volume dbafed96-a78f-4d48-9a0a-9e190e081353 is in insufficient state: Deleting", details: [], metadata: MetadataMap { headers: {} }
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller     at control-plane/csi-driver/src/bin/controller/controller.rs:299
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller     in csi_controller::controller::create_volume with request: Request { metadata: MetadataMap { headers: {"content-type": "application/grpc", "user-agent": "grpc-go/1.69.2", "te": "trailers", "grpc-timeout": "35999957u"} }, message: CreateVolumeRequest { name: "pvc-dbafed96-a78f-4d48-9a0a-9e190e081353", capacity_range: Some(CapacityRange { required_bytes: 2000000000, limit_bytes: 0 }), volume_capabilities: [VolumeCapability { access_mode: Some(AccessMode { mode: SingleNodeWriter }), access_type: Some(Mount(MountVolume { fs_type: "ext4", mount_flags: [], volume_mount_group: "" })) }], parameters: {"protocol": "nvmf", "ioTimeout": "30", "csi.storage.k8s.io/pvc/namespace": "toto-cloud--interne--prd--glpi-nginx--arch", "csi.storage.k8s.io/pv/name": "pvc-dbafed96-a78f-4d48-9a0a-9e190e081353", "poolHasTopologyKey": "node.k8s.toto/datacenter", "csi.storage.k8s.io/pvc/name": "data-mariadb-0", "repl": "2"}, secrets: {}, volume_content_source: None, accessibility_requirements: Some(TopologyRequirement { requisite: [Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf11", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf12"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf13", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf14"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf15"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf16", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf17"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf18"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf19"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf20", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf21"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf22", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf23", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf24", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf25", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf26"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf27"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf28", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf29"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf30", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf31"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf32", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf33"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf34"} }], preferred: [Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf22", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf23"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf24"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf25"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf26", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf27"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf28"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf29"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf30", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf31", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf32", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf33"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf34", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf11", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf12"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf13", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf14", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf15", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf16", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf17"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf18", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf19"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf20"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf21"} }] }), mutable_parameters: {} }, extensions: Extensions }, volume.uuid: "dbafed96-a78f-4d48-9a0a-9e190e081353"
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller

csi-controller

ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller   2025-07-22T16:56:49.798146Z ERROR csi_controller::controller: error: status: Unavailable, message: "error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: \"Failed to wait for thread\", message: \"task 124905 panicked with message \\\"user-provided comparison function does not correctly implement a total order\\\"\", kind: Aborted }'", details: [], metadata: MetadataMap { headers: {} }
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller     at control-plane/csi-driver/src/bin/controller/controller.rs:299
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner I0722 16:56:49.798508       1 controller.go:1115] "Temporary error received, adding PVC to claims in progress" claimUID="6b82173a-5d24-4358-8b0e-303cd0f58f08"
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner I0722 16:56:49.798529       1 controller.go:965] "Retrying syncing claim" key="6b82173a-5d24-4358-8b0e-303cd0f58f08" failures=28
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner E0722 16:56:49.798558       1 controller.go:988] "Unhandled Error" err="error syncing claim \"6b82173a-5d24-4358-8b0e-303cd0f58f08\": failed to provision volume with StorageClass \"mayastor-repl2\": rpc error: code = Unavailable desc = error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: \"Failed to wait for thread\", message: \"task 124905 panicked with message \\\"user-provided comparison function does not correctly implement a total order\\\"\", kind: Aborted }'" logger="UnhandledError"

csi-controller

ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller   2025-07-22T16:55:18.100878Z ERROR csi_controller::controller: error: status: AlreadyExists, message: "Existing volume dbafed96-a78f-4d48-9a0a-9e190e081353 is in insufficient state: Deleting", details: [], metadata: MetadataMap { headers: {} }
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller     at control-plane/csi-driver/src/bin/controller/controller.rs:299

describe pvc

  Type     Reason                Age                            From                                                                            Message
  ----     ------                ----                           ----                                                                            -------
  Normal   ExternalProvisioning  3s (x3 over 9s)                persistentvolume-controller                                                     Waiting for a volume to be created either by the external provisioner 'io.openebs.csi-mayastor' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Warning  ProvisioningFailed    <invalid>                      io.openebs.csi-mayastor_prd-app-kub-wsl25_7e569ec3-7c63-4d71-aeb9-5d056cd0c01f  failed to provision volume with StorageClass "mayastor-repl1": rpc error: code = Unavailable desc = error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: "Failed to wait for thread", message: "task 129518 panicked with message \"user-provided comparison function does not correctly implement a total order\"", kind: Aborted }'
  Normal   Provisioning          <invalid> (x4 over <invalid>)  io.openebs.csi-mayastor_prd-app-kub-wsl25_7e569ec3-7c63-4d71-aeb9-5d056cd0c01f  External provisioner is provisioning volume for claim "mayastor/pvc-test-1"
  Warning  ProvisioningFailed    <invalid> (x3 over <invalid>)  io.openebs.csi-mayastor_prd-app-kub-wsl25_7e569ec3-7c63-4d71-aeb9-5d056cd0c01f  failed to provision volume with StorageClass "mayastor-repl1": rpc error: code = Aborted desc = Existing volume 5bba40cd-d8b7-41bf-8e98-b80279be976f is in insufficient state: Creating

debian:~ (admin_apps:mayastor) % kmay delete volume 6b82173a-5d24-4358-8b0e-303cd0f58f08
> Are you sure you want to delete the resource? Yes
Failed to get volume 6b82173a-5d24-4358-8b0e-303cd0f58f08. Error error in response: status code '507 Insufficient Storage', content: 'RestJsonError { details: "", message: "SvcError :: StoreDirty: Storage Error: Volume Config for Resource id 6b82173a-5d24-4358-8b0e-303cd0f58f08 not committed to the store", kind: FailedPersist }'

on etcd pod

Replicas 1

kubectl exec -n mayastor -it ac-chart-mayastor-etcd-0 -- sh

I have no name!@etcd-0:/opt/bitnami/etcd$ etcdctl get --prefix /openebs.io/mayastor/apis/v0/clusters/0c01be95-e134-4770-b7a1-7c1f13400acc/namespaces/mayastor/ | grep 072bff26
{"uuid":"5bad0bc9-9ca4-466b-8dad-12ee0798cc7b","name":"072bff26-78f5-414a-ba2d-ba044d677dc9","node":"prd-app-kub-wsf26","children":[{"Replica":{"uuid":"e04d79cb-af13-4473-a706-b741260ed09c","share_uri":"bdev:///e04d79cb-af13-4473-a706-b741260ed09c?uuid=e04d79cb-af13-4473-a706-b741260ed09c"}}],"size":100000000,"spec_status":{"Created":"Online"},"share":"nvmf","managed":true,"owner":"072bff26-78f5-414a-ba2d-ba044d677dc9","operation":null,"nvmf_config":{"controllerIdRange":{"start":1,"end":2},"reservationKey":10208836743857753211,"reservationType":"ExclusiveAccess","preemptPolicy":"Holder"},"status_info":{"shutdown_failed":false},"allowed_hosts":["nqn.2019-05.io.openebs:node-name:prd-app-kub-wsf24"]}
{"name":"e04d79cb-af13-4473-a706-b741260ed09c","uuid":"e04d79cb-af13-4473-a706-b741260ed09c","size":100000000,"pool":"pool-prd-app-kub-wsf26","share":"none","thin":false,"status":{"Created":"online"},"managed":true,"owners":{"volume":"072bff26-78f5-414a-ba2d-ba044d677dc9","disown_all":false},"operation":null,"allowed_hosts":[],"encrypted":false}
/openebs.io/mayastor/apis/v0/clusters/0c01be95-e134-4770-b7a1-7c1f13400acc/namespaces/mayastor/VolumeSpec/072bff26-78f5-414a-ba2d-ba044d677dc9
{"uuid":"072bff26-78f5-414a-ba2d-ba044d677dc9","size":100000000,"labels":null,"num_replicas":1,"status":{"Created":"Online"},"policy":{"self_heal":true},"topology":{"node":{"Labelled":{"exclusion":{},"inclusion":{},"creation_affinity_keys":[]}},"pool":{"Labelled":{"exclusion":{},"inclusion":{"openebs.io/created-by":"operator-diskpool"},"creation_affinity_keys":[]}}},"last_nexus_id":null,"operation":null,"thin":false,"target":{"node":"prd-app-kub-wsf26","nexus":"5bad0bc9-9ca4-466b-8dad-12ee0798cc7b","protocol":"nvmf","active":true,"config":{"controllerIdRange":{"start":1,"end":2},"reservationKey":10208836743857753211,"reservationType":"ExclusiveAccess","preemptPolicy":"Holder"},"frontend":{"host_acl":[{"node_name":"prd-app-kub-wsf24","node_nqn":"nqn.2019-05.io.openebs:node-name:prd-app-kub-wsf24"}]}},"publish_context":{},"affinity_group":null,"encrypted":false}
/openebs.io/mayastor/apis/v0/clusters/0c01be95-e134-4770-b7a1-7c1f13400acc/namespaces/mayastor/volume/072bff26-78f5-414a-ba2d-ba044d677dc9/nexus/5bad0bc9-9ca4-466b-8dad-12ee0798cc7b/info

Replicas 2

kubectl exec -n mayastor -it ac-chart-mayastor-etcd-0 -- sh

etcdctl get --prefix /openebs.io/mayastor/apis/v0/clusters/0c01be95-e134-4770-b7a1-7c1f13400acc/namespaces/mayastor/ | grep f3a996b6

I have no name!@etcd-0:/opt/bitnami/etcd$ etcdctl get --prefix /openebs.io/mayastor/apis/v0/clusters/0c01be95-e134-4770-b7a1-7c1f13400acc/namespaces/mayastor/ | grep f3a996b6
{"uuid":"33f6939c-1ac6-41f7-aa0c-19a388babf00","name":"f3a996b6-af76-4bbd-9bcd-bdb0648bd313","node":"prd-app-kub-wsf23","children":[{"Replica":{"uuid":"b966bc4c-b8cb-4dc3-8cfd-3abed1d76421","share_uri":"nvmf://172.17.162.218:8420/nqn.2019-05.io.openebs:b966bc4c-b8cb-4dc3-8cfd-3abed1d76421?uuid=b966bc4c-b8cb-4dc3-8cfd-3abed1d76421"}},{"Replica":{"uuid":"f93de5d4-6b94-41ad-8591-865aaf65a44b","share_uri":"bdev:///f93de5d4-6b94-41ad-8591-865aaf65a44b?uuid=f93de5d4-6b94-41ad-8591-865aaf65a44b"}}],"size":8589934592,"spec_status":{"Created":"Online"},"share":"nvmf","managed":true,"owner":"f3a996b6-af76-4bbd-9bcd-bdb0648bd313","operation":null,"nvmf_config":{"controllerIdRange":{"start":5,"end":6},"reservationKey":12253196876332580608,"reservationType":"ExclusiveAccess","preemptPolicy":"Holder"},"status_info":{"shutdown_failed":false},"allowed_hosts":["nqn.2019-05.io.openebs:node-name:prd-app-kub-wsf32"]}
{"name":"b966bc4c-b8cb-4dc3-8cfd-3abed1d76421","uuid":"b966bc4c-b8cb-4dc3-8cfd-3abed1d76421","size":8589934592,"pool":"pool-prd-app-kub-wsf18","share":"nvmf","thin":false,"status":{"Created":"online"},"managed":true,"owners":{"volume":"f3a996b6-af76-4bbd-9bcd-bdb0648bd313","disown_all":false},"operation":null,"allowed_hosts":["nqn.2019-05.io.openebs:node-name:prd-app-kub-wsf23"],"encrypted":false}
{"name":"f93de5d4-6b94-41ad-8591-865aaf65a44b","uuid":"f93de5d4-6b94-41ad-8591-865aaf65a44b","size":8589934592,"pool":"pool-prd-app-kub-wsf23","share":"none","thin":false,"status":{"Created":"online"},"managed":true,"owners":{"volume":"f3a996b6-af76-4bbd-9bcd-bdb0648bd313","disown_all":false},"operation":null,"allowed_hosts":["nqn.2019-05.io.openebs:node-name:prd-app-kub-wsf32"],"encrypted":false}
/openebs.io/mayastor/apis/v0/clusters/0c01be95-e134-4770-b7a1-7c1f13400acc/namespaces/mayastor/VolumeSpec/f3a996b6-af76-4bbd-9bcd-bdb0648bd313
{"uuid":"f3a996b6-af76-4bbd-9bcd-bdb0648bd313","size":8589934592,"labels":null,"num_replicas":2,"status":{"Created":"Online"},"policy":{"self_heal":true},"topology":{"node":{"Labelled":{"exclusion":{},"inclusion":{},"creation_affinity_keys":[]}},"pool":{"Labelled":{"exclusion":{},"inclusion":{"openebs.io/created-by":"operator-diskpool","node.k8s.toto/datacenter":""},"creation_affinity_keys":[]}}},"last_nexus_id":null,"operation":null,"thin":false,"target":{"node":"prd-app-kub-wsf23","nexus":"33f6939c-1ac6-41f7-aa0c-19a388babf00","protocol":"nvmf","active":true,"config":{"controllerIdRange":{"start":5,"end":6},"reservationKey":12253196876332580608,"reservationType":"ExclusiveAccess","preemptPolicy":"Holder"},"frontend":{"host_acl":[{"node_name":"prd-app-kub-wsf32","node_nqn":"nqn.2019-05.io.openebs:node-name:prd-app-kub-wsf32"}]}},"publish_context":{},"affinity_group":null,"encrypted":false}
/openebs.io/mayastor/apis/v0/clusters/0c01be95-e134-4770-b7a1-7c1f13400acc/namespaces/mayastor/volume/f3a996b6-af76-4bbd-9bcd-bdb0648bd313/nexus/33f6939c-1ac6-41f7-aa0c-19a388babf00/info

[tiagolobocastro]

Thank you for detailed information!

With that I would like to point the finger to the topology, but the topology would only be used here for filtering pools, not for sorting.

On the agent-core logs, is that the entire backtrace? Seems to be missing the entire call stack, which would help point to where in our code we're hitting it

[kiliandsc]

Agent core: this is the only the same stack output I get when I try to kubectl apply -f pvc.yaml. I really don’t see anything else — just that small snippet. Same thing happens when I try to delete the PVC.

ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    0:     0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    3:     0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    4:     0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    5:     0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    6:     0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    7:     0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    9:     0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   10:     0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   31:     0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    0:     0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    3:     0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    4:     0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    5:     0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    6:     0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    7:     0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    9:     0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   10:     0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   31:     0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   2025-07-22T18:32:55.545216Z ERROR core::controller::reconciler::volume::garbage_collector: Failed to destroy volume, error: Storage Error: Volume Config for Resource id dbe26fcf-d022-455e-9a80-0478ef83c5c6 not committed to the store
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    0:     0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    3:     0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    4:     0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    5:     0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    6:     0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    7:     0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    9:     0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   10:     0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   31:     0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    0:     0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    3:     0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    4:     0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    5:     0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    6:     0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    7:     0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    9:     0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   10:     0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   31:     0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    0:     0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    3:     0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    4:     0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    5:     0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    6:     0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    7:     0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    9:     0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   10:     0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   31:     0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f

It always follows the same pattern: it logs INFO csi_driver::trace: [CSI] Create Volume Request started, then it fails, retries, says the volume already exists, and retries again — as you can see in the logs.

ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner I0722 18:33:11.205838       1 event.go:389] "Event occurred" object="mayastor/pvc-test-1" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"mayastor/pvc-test-1\""
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller   2025-07-22T18:33:11.208110Z  INFO csi_driver::trace: [ CSI ] Create Volume Request started
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller     at control-plane/csi-driver/src/trace.rs:10
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller     in csi_controller::controller::create_volume with request: Request { metadata: MetadataMap { headers: {"content-type": "application/grpc", "user-agent": "grpc-go/1.69.2", "te": "trailers", "grpc-timeout": "35999962u"} }, message: CreateVolumeRequest { name: "pvc-dbe26fcf-d022-455e-9a80-0478ef83c5c6", capacity_range: Some(CapacityRange { required_bytes: 7516192768, limit_bytes: 0 }), volume_capabilities: [VolumeCapability { access_mode: Some(AccessMode { mode: SingleNodeWriter }), access_type: Some(Mount(MountVolume { fs_type: "ext4", mount_flags: [], volume_mount_group: "" })) }], parameters: {"protocol": "nvmf", "ioTimeout": "30", "csi.storage.k8s.io/pvc/namespace": "mayastor", "csi.storage.k8s.io/pvc/name": "pvc-test-1", "repl": "1", "csi.storage.k8s.io/pv/name": "pvc-dbe26fcf-d022-455e-9a80-0478ef83c5c6"}, secrets: {}, volume_content_source: None, accessibility_requirements: Some(TopologyRequirement { requisite: [Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf11", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf12", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf13"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf14"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf15"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf16"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf17", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf18"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf19", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf20"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf21", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf22"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf23", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf24", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf25", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf26"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf27", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf28"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf29"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf30", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf31"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf32"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf33"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf34", "openebs.io/csi-node": "mayastor"} }], preferred: [Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf29"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf30", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf31"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf32"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf33"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf34", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf11", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf12"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf13"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf14"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf15", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf16", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf17"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf18"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf19"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf20", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf21", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf22", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf23", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf24"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf25", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf26", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf27"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf28", "openebs.io/csi-node": "mayastor"} }] }), mutable_parameters: {} }, extensions: Extensions }, volume.uuid: "dbe26fcf-d022-455e-9a80-0478ef83c5c6"
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller   2025-07-22T18:33:11.220502Z ERROR csi_controller::controller: error: status: Unavailable, message: "error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: \"Failed to wait for thread\", message: \"task 223637 panicked with message \\\"user-provided comparison function does not correctly implement a total order\\\"\", kind: Aborted }'", details: [], metadata: MetadataMap { headers: {} }
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller     at control-plane/csi-driver/src/bin/controller/controller.rs:299
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller     in csi_controller::controller::create_volume with request: Request { metadata: MetadataMap { headers: {"content-type": "application/grpc", "user-agent": "grpc-go/1.69.2", "te": "trailers", "grpc-timeout": "35999962u"} }, message: CreateVolumeRequest { name: "pvc-dbe26fcf-d022-455e-9a80-0478ef83c5c6", capacity_range: Some(CapacityRange { required_bytes: 7516192768, limit_bytes: 0 }), volume_capabilities: [VolumeCapability { access_mode: Some(AccessMode { mode: SingleNodeWriter }), access_type: Some(Mount(MountVolume { fs_type: "ext4", mount_flags: [], volume_mount_group: "" })) }], parameters: {"protocol": "nvmf", "ioTimeout": "30", "csi.storage.k8s.io/pvc/namespace": "mayastor", "csi.storage.k8s.io/pvc/name": "pvc-test-1", "repl": "1", "csi.storage.k8s.io/pv/name": "pvc-dbe26fcf-d022-455e-9a80-0478ef83c5c6"}, secrets: {}, volume_content_source: None, accessibility_requirements: Some(TopologyRequirement { requisite: [Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf11", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf12", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf13"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf14"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf15"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf16"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf17", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf18"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf19", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf20"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf21", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf22"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf23", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf24", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf25", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf26"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf27", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf28"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf29"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf30", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf31"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf32"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf33"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf34", "openebs.io/csi-node": "mayastor"} }], preferred: [Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf29"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf30", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf31"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf32"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf33"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf34", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf11", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf12"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf13"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf14"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf15", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf16", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf17"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf18"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf19"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf20", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf21", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf22", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf23", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf24"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf25", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf26", "openebs.io/csi-node": "mayastor"} }, Topology { segments: {"openebs.io/csi-node": "mayastor", "openebs.io/nodename": "prd-app-kub-wsf27"} }, Topology { segments: {"openebs.io/nodename": "prd-app-kub-wsf28", "openebs.io/csi-node": "mayastor"} }] }), mutable_parameters: {} }, extensions: Extensions }, volume.uuid: "dbe26fcf-d022-455e-9a80-0478ef83c5c6"

ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner I0722 18:33:11.205838       1 event.go:389] "Event occurred" object="mayastor/pvc-test-1" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"mayastor/pvc-test-1\""
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller   2025-07-22T18:33:11.208110Z  INFO csi_driver::trace: [ CSI ] Create Volume Request started
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller     at control-plane/csi-driver/src/trace.rs:10

ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller   2025-07-22T18:33:11.220502Z ERROR csi_controller::controller: error: status: Unavailable, message: "error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: \"Failed to wait for thread\", message: \"task 223637 panicked with message \\\"user-provided comparison function does not correctly implement a total order\\\"\", kind: Aborted }'", details: [], metadata: MetadataMap { headers: {} }
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller     at control-plane/csi-driver/src/bin/controller/controller.rs:299
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller     in csi_controller::controller::create_volume with request: Request { metadata: MetadataMap { headers: {"content-type": "application/grpc", "user-agent": "grpc-go/1.69.2", "te": "trailers", "grpc-timeout": "35999962u"} }, message: CreateVolumeRequest { name: "pvc-dbe26fcf-d022-455e-9a80-0478ef83c5c6", capacity_range: Some(CapacityRange { required_bytes: 7516192768, limit_bytes: 0 }), volume_capabilities: [VolumeCapability { access_mode: Some(AccessMode { mode: SingleNodeWriter }), access_type: Some(Mount(MountVolume { fs_type: "ext4", mount_flags: [], volume_mount_group: "" })) }], parameters: {"protocol": "nvmf", "ioTimeout": "30", "csi.storage.k8s.io/pvc/namespace": "mayastor", "csi.storage.k8s.io/pvc/name": "pvc-test-1", "repl": "1", "csi.storage.k8s.io/pv/name": "pvc-dbe26fcf-d022-455e-9a80-0478ef83c5c6"}, secrets: {}, volume_content_source: None, accessibility_requirements: Some(TopologyReq

ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller   2025-07-22T18:35:18.257990Z ERROR csi_controller::controller: error: status: AlreadyExists, message: "Existing volume dbe26fcf-d022-455e-9a80-0478ef83c5c6 is in insufficient state: Deleting", details: [], metadata: MetadataMap { headers: {} }

ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller   2025-07-22T18:34:14.247079Z  INFO csi_driver::trace: [ CSI ] Create Volume Request started

ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner I0722 18:37:26.281779       1 controller.go:1115] "Temporary error received, adding PVC to claims in progress" claimUID="dbe26fcf-d022-455e-9a80-0478ef83c5c6"
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner I0722 18:37:26.281797       1 controller.go:965] "Retrying syncing claim" key="dbe26fcf-d022-455e-9a80-0478ef83c5c6" failures=8
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner E0722 18:37:26.281820       1 controller.go:988] "Unhandled Error" err="error syncing claim \"dbe26fcf-d022-455e-9a80-0478ef83c5c6\": failed to provision volume with StorageClass \"mayastor-repl1\": rpc error: code = Unavailable desc = error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: \"Failed to wait for thread\", message: \"task 227927 panicked with message \\\"user-provided comparison function does not correctly implement a total order\\\"\", kind: Aborted }'" logger="UnhandledError"
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-provisioner I0722 18:37:26.281888       1 event.go:389] "Event occurred" object="mayastor/pvc-test-1" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"mayastor-repl1\": rpc error: code = Unavailable desc = error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: \"Failed to wait for thread\", message: \"task 227927 panicked with message \\\"user-provided comparison function does not correctly implement a total order\\\"\", kind: Aborted }'"
ac-chart-mayastor-csi-controller-7cd9c66c69-frw7f csi-controller   2025-07-22T18:37:26.281552Z ERROR csi_controller::controller: error: status: Unavailable, message: "error in response: status code '503 Service Unavailable', content: 'RestJsonError { details: \"Failed to wait for thread\", message: \"task 227927 panicked with message \\\"user-provided comparison function does not correctly implement a total order\\\"\", kind: Aborted }'", details: [], metadata: MetadataMap { headers: {} }

the complete stack with 3 tries on k delete pvc and k apply

ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    0:     0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    3:     0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    4:     0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    5:     0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    6:     0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    7:     0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    9:     0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   10:     0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   31:     0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    0:     0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    3:     0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    4:     0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    5:     0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    6:     0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    7:     0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    9:     0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   10:     0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   31:     0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   2025-07-22T18:32:55.545216Z ERROR core::controller::reconciler::volume::garbage_collector: Failed to destroy volume, error: Storage Error: Volume Config for Resource id dbe26fcf-d022-455e-9a80-0478ef83c5c6 not committed to the store
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    0:     0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    3:     0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    4:     0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    5:     0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    6:     0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    7:     0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    9:     0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   10:     0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   31:     0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    0:     0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    3:     0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    4:     0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    5:     0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    6:     0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    7:     0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    9:     0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   10:     0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   31:     0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    0:     0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    3:     0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    4:     0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    5:     0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    6:     0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    7:     0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    9:     0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   10:     0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   31:     0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core stack backtrace:
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    0:     0x561052de60ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    3:     0x561052de7bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    4:     0x561052de783e - std::panicking::default_hook::h81c8cd2e7c59ee33
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    5:     0x561052de84cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    6:     0x561052de8183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    7:     0x561052de65a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core    9:     0x561052e12173 - core::panicking::panic_fmt::h3eea515d05f7a35e
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   10:     0x561052e16bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core   31:     0x561052d8594e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
ac-chart-mayastor-agent-core-7cd7969d95-4vkrf agent-core thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:

debian-kilian:~ (admin_apps:mayastor) % k get pvc
NAME                            STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS            VOLUMEATTRIBUTESCLASS   AGE
data-ac-chart-mayastor-etcd-0   Bound     pvc-457c0a61-9cf1-4d11-beec-b35aaa70d673   2Gi        RWO            mayastor-etcd-localpv   <unset>                 18d
data-ac-chart-mayastor-etcd-1   Bound     pvc-9f3a5aca-0e54-40a6-aa02-1ecacca818b0   2Gi        RWO            mayastor-etcd-localpv   <unset>                 18d
data-ac-chart-mayastor-etcd-2   Bound     pvc-b0ff94c8-9266-4c8f-9352-1209b81c1b2e   2Gi        RWO            mayastor-etcd-localpv   <unset>                 18d
pvc-test-1                      Pending                                                                        mayastor-repl1          <unset>                 75s

debian-kilian:~ (admin_apps:mayastor) % k get sc mayastor-repl2 -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"mayastor-repl2"},"parameters":{"ioTimeout":"30","poolHasTopologyKey":"node.k8s.asten/datacenter","protocol":"nvmf","repl":"2"},"provisioner":"io.openebs.csi-mayastor","reclaimPolicy":"Delete","volumeBindingMode":"Immediate"}
  creationTimestamp: "2025-07-22T16:02:35Z"
  name: mayastor-repl2
  resourceVersion: "453750675"
  uid: 76484fa3-da83-4340-bd3d-3d028babbed0
parameters:
  ioTimeout: "30"
  poolHasTopologyKey: node.k8s.asten/datacenter
  protocol: nvmf
  repl: "2"
provisioner: io.openebs.csi-mayastor
reclaimPolicy: Delete
volumeBindingMode: Immediate
debian-kilian:~ (admin_apps:mayastor) % k get sc mayastor-repl1 -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"mayastor-repl1"},"parameters":{"ioTimeout":"30","protocol":"nvmf","repl":"1"},"provisioner":"io.openebs.csi-mayastor","reclaimPolicy":"Delete","volumeBindingMode":"Immediate"}
  creationTimestamp: "2025-07-22T16:02:17Z"
  name: mayastor-repl1
  resourceVersion: "453750284"
  uid: 860fafb9-283f-49f6-884d-adb905a32b97
parameters:
  ioTimeout: "30"
  protocol: nvmf
  repl: "1"
provisioner: io.openebs.csi-mayastor
reclaimPolicy: Delete
volumeBindingMode: Immediate
debian-kilian:~ (admin_apps:mayastor) % k get sc
NAME                    PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ceph-filesystem         rook-ceph.cephfs.csi.ceph.com   Delete          Immediate              true                   193d
local-storage           kubernetes.io/no-provisioner    Delete          WaitForFirstConsumer   false                  452d
mayastor-etcd-localpv   openebs.io/local                Delete          WaitForFirstConsumer   false                  16d
mayastor-repl1          io.openebs.csi-mayastor         Delete          Immediate              true                   160m
mayastor-repl2          io.openebs.csi-mayastor         Delete          Immediate              true                   159m
nfs-csi                 nfs.csi.k8s.io                  Delete          Immediate              false                  432d
openebs-hostpath        openebs.io/local                Delete          WaitForFirstConsumer   false                  218d
openebs-locallvm        local.csi.openebs.io            Delete          Immediate              false                  172d

Maybe the other StorageClass from OpenEBS is causing a conflict? We’re currently migrating from openebs-hostpath to mayastor, so both are still present in the cluster.

debian-kilian:~ (admin_apps:mayastor) % k get ds
NAME                              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                                   AGE
ac-chart-mayastor-agent-ha-node   24        24        24      24           24          kubernetes.io/arch=amd64,node.k8s.asten/infra=v1,openebs.io/csi-node=mayastor   16d
ac-chart-mayastor-csi-node        24        24        24      24           24          kubernetes.io/arch=amd64,node.k8s.asten/infra=v1,openebs.io/csi-node=mayastor   16d
ac-chart-mayastor-io-engine       24        24        24      24            24          kubernetes.io/arch=amd64,openebs.io/engine=mayastor                             16d
debian-kilian:~ (admin_apps:mayastor) % k get sts
NAME                     READY   AGE
ac-chart-mayastor-etcd   3/3     16d
debian-kilian:~ (admin_apps:mayastor) % k get deploy
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
ac-chart-mayastor-agent-core            1/1     1            1           16d
ac-chart-mayastor-api-rest              1/1     1            1           16d
ac-chart-mayastor-csi-controller        1/1     1            1           16d
ac-chart-mayastor-localpv-provisioner   1/1     1            1           16d
ac-chart-mayastor-obs-callhome          1/1     1            1           16d
ac-chart-mayastor-operator-diskpool     1/1     1            1           16d

in values

    node:
      logLevel: info
      topology:
        segments:
          openebs.io/csi-node: mayastor
        # -- Add topology segments to the csi-node and agent-ha-node daemonset node selector
        nodeSelector: True

[kiliandsc]

Here’s the full backtrace I was able to capture. It shows a panic in AddVolumeReplica::with_default_policy during a slice sort operation:

user-provided comparison function does not correctly implement a total order

From what I understand, it seems like the custom sorting function used for selecting volume replicas might be violating Rust’s total ordering rules. This results in a panic that crashes the provisioning.

I’m not entirely sure what's

thread 'tokio-runtime-worker' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
user-provided comparison function does not correctly implement a total order
stack backtrace:
   0:     0x55cd67df00ca - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
   1:     0x55cd67e1e81b - core::fmt::write::hf5713710ce10ff22
   2:     0x55cd67debce3 - std::io::Write::write_fmt::hda708db57927dacf
   3:     0x55cd67df1bd2 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
   4:     0x55cd67df183e - std::panicking::default_hook::h81c8cd2e7c59ee33
   5:     0x55cd67df24cf - std::panicking::rust_panic_with_hook::had2118629c312a4a
   6:     0x55cd67df2183 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
   7:     0x55cd67df05a9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
   8:     0x55cd67df1e44 - rust_begin_unwind
   9:     0x55cd67e1c173 - core::panicking::panic_fmt::h3eea515d05f7a35e
  10:     0x55cd67e20bdb - core::slice::sort::shared::smallsort::panic_on_ord_violation::h972f60d892347b6f
  11:     0x55cd66dfbac2 - core::slice::sort::stable::quicksort::quicksort::h891bee438e3ee28e
  12:     0x55cd66dee558 - core::slice::sort::stable::drift::sort::h2840261b8def17e7
  13:     0x55cd6706df7f - core::slice::sort::stable::driftsort_main::hce08a934693303a4
  14:     0x55cd671f6217 - core::controller::scheduling::volume::AddVolumeReplica::with_default_policy::h97f5333af7d049c6
  15:     0x55cd67221943 - <stor_port::types::v0::transport::volume::CreateVolume as core::volume::operations::CreateVolumeExe>::setup::{{closure}}::h6e4d72f70e22a13f
  16:     0x55cd6721e6fa - core::volume::operations::CreateVolumeExe::run::{{closure}}::h49b64c3a9b167603
  17:     0x55cd66bcef97 - core::volume::operations::<impl core::controller::resources::operations::ResourceLifecycleExt<core::volume::operations::CreateVolumeSource> for core::controller::resources::OperationGuard<core::controller::resources::ResourceMutex<stor_port::types::v0::store::volume::VolumeSpec>,stor_port::types::v0::store::volume::VolumeSpec>>::create_ext::{{closure}}::h9cbaa6de7b8c486a
  18:     0x55cd66bae29a - core::volume::operations::<impl core::controller::resources::operations::ResourceLifecycle for core::controller::resources::OperationGuard<core::controller::resources::ResourceMutex<stor_port::types::v0::store::volume::VolumeSpec>,stor_port::types::v0::store::volume::VolumeSpec>>::create::{{closure}}::he284ae3b6c976fb6
  19:     0x55cd66c54397 - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h5e8333050e1e7c49
  20:     0x55cd6739ec8c - <opentelemetry::trace::context::WithContext<T> as core::future::future::Future>::poll::hc86ec8aeeb889947
  21:     0x55cd67460cb6 - tokio::runtime::task::core::Core<T,S>::poll::heac7b23fa9c1f994
  22:     0x55cd66d11555 - tokio::runtime::task::harness::Harness<T,S>::poll::hd5f5d2b78cc212db
  23:     0x55cd67da50ab - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h9f947b4903e5e002
  24:     0x55cd67da4061 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h64fad4113840e1a3
  25:     0x55cd67da0354 - tokio::runtime::context::runtime::enter_runtime::ha9ceb71500dc86b3
  26:     0x55cd67da3eca - tokio::runtime::scheduler::multi_thread::worker::run::h3505ec8fa10c7037
  27:     0x55cd67d91a67 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::hfeee8786b776e854
  28:     0x55cd67da7673 - tokio::runtime::task::core::Core<T,S>::poll::h4d8d9fe7b4aa06eb
  29:     0x55cd67d89b84 - tokio::runtime::task::harness::Harness<T,S>::poll::h367a206a47f47d13
  30:     0x55cd67d8d41f - tokio::runtime::blocking::pool::Inner::run::h799797b26549b35d
  31:     0x55cd67d8f94e - std::sys::backtrace::__rust_begin_short_backtrace::hdbfad5935d0cad4f
  32:     0x55cd67d900c2 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h4eb31ab3cb5a138f
  33:     0x55cd67df795b - std::sys::pal::unix::thread::Thread::new::thread_start::hcdbd1049068002f4
  34:     0x7f70c3673272 - start_thread
  35:     0x7f70c36eedec - clone3
  36:                0x0 - <unknown>

[kiliandsc]

https://releases.rs/docs/1.81.0/

The new sort implementations may panic if a type’s implementation of Ord (or the given comparison function) does not implement a total order as the trait requires. Ord’s supertraits (PartialOrd, Eq, and PartialEq) must also be consistent. The previous implementations would not “notice” any problem, but the new implementations have a good chance of detecting inconsistencies, throwing a panic rather than returning knowingly unsorted data.

[tiagolobocastro]

Looks like this is our code:

 13:     0x55cd6706df7f - core::slice::sort::stable::driftsort_main::hce08a934693303a4
 14:     0x55cd671f6217 - core::controller::scheduling::volume::AddVolumeReplica::with_default_policy::h97f5333af7d049c6

Would you be able to try a debug image of the agent-core to see if it's capturing some more information helpful to pinpoint the issue?

[kiliandsc]

Actually, i downgraded agent-core to version v2.7.0 and the issue completely disappeared — no more panics so far.

Do you have:
A debug image ?
Or a specific SHA/tag ?

[tiagolobocastro]

Running a test with 600 volumes and 100 pools and using topology and not hitting this :/

I've pushed image openebs/mayastor-agent-core:v2.9.1-dev, which is exactly what it says in the tin, 2.9.1 build in debug mode
Also please ensure env RUST_BACKTRACE is set to full

Thank you

[tiagolobocastro]

We've identified the main culprit, the dual weighted scoring, which by nature is not a total order :/
I think we should refactor that with a new system which normalizes the scores based on the entire range of values.
Meanwhile replacing sort with sort_unstable seems to reduce the panics in my testing, would you be able to test with the image openebs/mayastor-agent-core:v2.9.0-total-order ? (remember, not a fix, but seems to be more "resistent")