[Bug]: Strimzi restart pod with pvc in FileSystemResizePending state before it can be resized online without restart

[hennadii2012]

Bug Description

After resize PVC in some cases (not always) Strimzi is doing pod restart before k8s execute operation of online expansion.

Steps to reproduce

1. Increase disk
2. Wait for resize
3. Check if pod was restarted
4. if not repeat steps 1-3

Expected behavior

Strimzi doesn't restart pod on change PVC size and wait for online resize (probably extra option for such wait time can be added)

Strimzi version

0.45.0

Kubernetes version

1.30

Installation method

Helm chart

Infrastructure

GKE

Configuration files and logs

No response

Additional context

No response

[scholzj]

Some infrastructures (under some circumstances - might depend for example on the volume type / size, etc.) require the Pod to roll in this state. Unfortunately, Stirmzi has no simple way to distinguish when is the restart of the Pod required and when not. So it might restart it even if it is not needed. This has been already discussed plenty times before both here on GitHub as well as on Slack.

[hennadii2012]

Is it possible to add extra parameter (I mean time parameter ) that will be respected in case of rolling restart with event reason resize disk.

[scholzj]

I'm not sure what exactly you mean. There is no watch on the PVCs. It is checked in the regular reconciliation.

[hennadii2012]

Is there a way to do extra check during reconciliation process?

…

________________________________ From: Jakub Scholz ***@***.***> Sent: Friday, March 21, 2025 5:16:32 PM To: strimzi/strimzi-kafka-operator ***@***.***> Cc: Nikitin, Hennadii ***@***.***>; Author ***@***.***> Subject: Re: [strimzi/strimzi-kafka-operator] [Bug]: Strimzi restart pod with pvc in FileSystemResizePending state before it can be resized online without restart (Issue #11279) I'm not sure what exactly you mean. There is no watch on the PVCs. It is checked in the regular reconciliation. — Reply to this email directly, view it on GitHub<#11279 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AVZ2VPRFZJYM3AWSFS7OIJL2VQ3OBAVCNFSM6AAAAABZOHZRDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBTHA2DENJUGQ>. You are receiving this because you authored the thread.Message ID: ***@***.***> [scholzj]scholzj left a comment (strimzi/strimzi-kafka-operator#11279)<#11279 (comment)> I'm not sure what exactly you mean. There is no watch on the PVCs. It is checked in the regular reconciliation. — Reply to this email directly, view it on GitHub<#11279 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AVZ2VPRFZJYM3AWSFS7OIJL2VQ3OBAVCNFSM6AAAAABZOHZRDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBTHA2DENJUGQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[scholzj]

There is nothing to check on because this is all the information the infrastructure gives us.

[hennadii2012]

Infra also provides the time when the event was started, so if add extra parameter `timeout_on_online_resize` and compare those times then rolling restart because of resize can be postponed on time in `timeout_on_online_resize` var

…

________________________________ From: Jakub Scholz ***@***.***> Sent: Friday, March 21, 2025 7:34:17 PM To: strimzi/strimzi-kafka-operator ***@***.***> Cc: Nikitin, Hennadii ***@***.***>; Author ***@***.***> Subject: Re: [strimzi/strimzi-kafka-operator] [Bug]: Strimzi restart pod with pvc in FileSystemResizePending state before it can be resized online without restart (Issue #11279) There is nothing to check on because this is all the information the infrastructure gives us. — Reply to this email directly, view it on GitHub<#11279 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AVZ2VPVXRH4AGKWUSPERMGT2VRLSTAVCNFSM6AAAAABZOHZRDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBUGE2TMNZZGY>. You are receiving this because you authored the thread.Message ID: ***@***.***> [scholzj]scholzj left a comment (strimzi/strimzi-kafka-operator#11279)<#11279 (comment)> There is nothing to check on because this is all the information the infrastructure gives us. — Reply to this email directly, view it on GitHub<#11279 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AVZ2VPVXRH4AGKWUSPERMGT2VRLSTAVCNFSM6AAAAABZOHZRDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBUGE2TMNZZGY>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[scholzj]

That is a lot of effort and maintenance for something what does not break anything. Anyway, this will be triaged on the next community call.

[hennadii2012]

For cluster in high load rolling restart causing extra load on consumers because of necessity of rebalancing and provoke lag increasing

…

________________________________ From: Jakub Scholz ***@***.***> Sent: Friday, March 21, 2025 7:42:29 PM To: strimzi/strimzi-kafka-operator ***@***.***> Cc: Nikitin, Hennadii ***@***.***>; Author ***@***.***> Subject: Re: [strimzi/strimzi-kafka-operator] [Bug]: Strimzi restart pod with pvc in FileSystemResizePending state before it can be resized online without restart (Issue #11279) That is a lot of effort and maintenance for something what does not break anything. Anyway, this will be triaged on the next community call. — Reply to this email directly, view it on GitHub<#11279 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AVZ2VPWNT7VDHPBXGPWBVXL2VRMRLAVCNFSM6AAAAABZOHZRDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBUGE4DCMZXGY>. You are receiving this because you authored the thread.Message ID: ***@***.***> [scholzj]scholzj left a comment (strimzi/strimzi-kafka-operator#11279)<#11279 (comment)> That is a lot of effort and maintenance for something what does not break anything. Anyway, this will be triaged on the next community call. — Reply to this email directly, view it on GitHub<#11279 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AVZ2VPWNT7VDHPBXGPWBVXL2VRMRLAVCNFSM6AAAAABZOHZRDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBUGE4DCMZXGY>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[scholzj]

Yes, but increasing disk storage is usually a relatively rare event. Also, it is easy to argue the opposite - that in some cases it is critical to do it without delays given how badly Kafka handles running out of disk space. So until there is proper solution at Kubernetes level, not sure this is worth doing anything.

[hennadii2012]

In such case this parameter can be set to 0 and ignored

…

________________________________ From: Jakub Scholz ***@***.***> Sent: Friday, March 21, 2025 7:55:18 PM To: strimzi/strimzi-kafka-operator ***@***.***> Cc: Nikitin, Hennadii ***@***.***>; Author ***@***.***> Subject: Re: [strimzi/strimzi-kafka-operator] [Bug]: Strimzi restart pod with pvc in FileSystemResizePending state before it can be resized online without restart (Issue #11279) Yes, but increasing disk storage is usually a relatively rare event. Also, it is easy to argue the opposite - that in some cases it is critical to do it without delays given how badly Kafka handles running out of disk space. So until there is proper solution at Kubernetes level, not sure this is worth doing anything. — Reply to this email directly, view it on GitHub<#11279 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AVZ2VPSQPKB5JSA3V2SPRZL2VROBNAVCNFSM6AAAAABZOHZRDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBUGIYDMOBTHE>. You are receiving this because you authored the thread.Message ID: ***@***.***> [scholzj]scholzj left a comment (strimzi/strimzi-kafka-operator#11279)<#11279 (comment)> Yes, but increasing disk storage is usually a relatively rare event. Also, it is easy to argue the opposite - that in some cases it is critical to do it without delays given how badly Kafka handles running out of disk space. So until there is proper solution at Kubernetes level, not sure this is worth doing anything. — Reply to this email directly, view it on GitHub<#11279 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AVZ2VPSQPKB5JSA3V2SPRZL2VROBNAVCNFSM6AAAAABZOHZRDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBUGIYDMOBTHE>. You are receiving this because you authored the thread.Message ID: ***@***.***>