Volume filling up but pgdata stays the same

[N3zeq]

Environment

• Platform: OpenShift
• Platform Version: 4.14.7
• PGO Image Tag: 5.5.2
• Postgres Version 15
• Storage: thin-csi (vsan)

##Question

I'm running crunchy Postgres 5.5 on openshift on an air gapped environment and I'm running into a very weird problem: my volume is constantly filling up, while my database size doesn't change and my wal files and logs are completely normal. Because I'm running in an air gapped environment I can't paste logs, but all logs on archive-push are returning successful completion. For example, when I run du -sh /pgdatait could return 800mb, but when I run df -h it sometimes returned up to 80Gi (and so did the pgmonitor metrics and the openshift metrics). I'll paste code from a smaller scale example:

[root@host ~]# df /mnt/sdf
Filesystem  1K-Blocks Used Available Use% Mounted on
/dev/sdf  15390744 564988 14117324 4% /var/mnt/sdf

[root@host ~]# du -sc /mnt/sdf/
285544 /mnt/sdf/
285544 total

And since it has even filled up more. Any ideas what could cause this issue?

[tjmoore4]

@N3zeq sorry to hear you're hitting this issue. It's hard for me to say for certain exactly what may be happening without more details from logs, etc, but I would recommend looking into CPK's support of thin provisioning using the auto-growable disk feature. It would require moving to 5.6+ but this feature may help in your case since you appear to be using a thin provisioner (thin-csi (vsan)) in your environment. Details can be found below:
Blog: https://www.crunchydata.com/blog/thin-provisioning-with-crunchy-postgres-for-kubernetes
Docs: https://access.crunchydata.com/documentation/postgres-operator/latest/guides/autogrowable-disk
Hope this helps!

[N3zeq]

@N3zeq sorry to hear you're hitting this issue. It's hard for me to say for certain exactly what may be happening without more details from logs, etc, but I would recommend looking into CPK's support of thin provisioning using the auto-growable disk feature. It would require moving to 5.6+ but this feature may help in your case since you appear to be using a thin provisioner (thin-csi (vsan)) in your environment. Details can be found below:
Blog: https://www.crunchydata.com/blog/thin-provisioning-with-crunchy-postgres-for-kubernetes
Docs: https://access.crunchydata.com/documentation/postgres-operator/latest/guides/autogrowable-disk
Hope this helps!

Hello!
Thanks for the recommendation, I'll look into it. I could print out some logs if needed. The problem is that the Postgres logs are just fine. They don't show any irregularities, errors or warnings. It seems to be that something is going wrong between the filesystem and the kernel with Postgres being the trigger for it. I just wondered if someone encountered this before.

[tjmoore4]

Welcome! Besides trying out the auto-growable disk feature, I'd also try to look through your PVC and PV definitions and see if they are matching what you are seeing via the commands you're running. Another thing I would review is the CSI driver Pod logs for anything out of the ordinary. Hopefully something there might give a better clue as to what's going on.