p2k16 - sometimes the disk fills up, and p2k16 stops working

[tingox]

Sometimes (not very often) the disk drive of the p2k16 serve fills up. this is bad, because the then p2k16 web app stops working.

the server p2k16 runs two services related to the PostgreSQL database:
postgresql@10-main.service
postgresql-base-backup@10-main.service
and also this
postgresql-base-backup@10-main.timer
we also have monitoring (via riemann), but nodbody watches that on a regular basis (not sure if someone gets alarm notificatons).

[tingox]

The directory /var/lib/postgresql/backups/ was filling up with db backups, causing the disk to fill. I cleaned out a few files, the disk is now better:

root@p2k16:~# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  8.9G  9.8G  48% /

and then I restartet postgres via systemctl restart postgresql@10-main.service.

[tingox]

There is a backup service for postegres, I haven't restarted it

root@p2k16:~# systemctl status postgresql-base-backup@10-main.service
● postgresql-base-backup@10-main.service - PostgreSQL base backup
   Loaded: loaded (/etc/systemd/system/postgresql-base-backup@.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2020-04-26 04:00:51 CEST; 7h ago
  Process: 19137 ExecStart=/usr/bin/env bash -c i="10-main"; i=${i/-//}; bin/envdir /etc/wal-e/10-main-env.
 Main PID: 19137 (code=exited, status=1/FAILURE)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

should this serrvice be running, or should we stop it?

[tingox]

p2k16-staging also had the same problem, so I did the same there: clean out most files from db backups, then resstart postgresql.

[rkarlsba]

Perhaps it would be nice to have some monitoring on that, with email alerts to those who run the system - something like zabbix? I have a zabbix VM running…

[tingox]

monitoring is in place, we miss someplace good to send the alerts. Our "IT operations group" is on a volunteer basis...

[rkarlsba]

Where can I see this monitoring status?

[tingox]

monitoring is at riemann.bitraf.no

[tingox]

It happened again; the disk of p2k16 filled up with postgres database backups, and the postgres service failed, causing p2k16 to fail. The disk full error was dutifully recorded by riemann.bitraf.no, but nobody looked at it.
Fix: usual fix - clean out database backups from /var/lib/postgresql/backups/, then restart postgres with sudo systemctl restart postgresql@10-main.service.

[tingox]

perhaps we should add a separate (virtual) disk drive for database backups to the server p2k16. Or better: make sure that the backups go to another server instead. Hmm.

[tingox]

cleaned the backup directory on p2k16-staging, better now

tingo@p2k16-staging:~$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  8.9G  9.8G  48% /

then I restarted postgres with
sudo systemctl restart postgresql@10-main.service
that's all

[tingox]

cleaned backup directory on p2k16-staging again

tingo@p2k16-staging:~$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G   11G  7.9G  58% /

and restarted postgres

[tingox]

Another cleaning of the postgres backup directory on p2k16-staging today:

tingo@p2k16-staging:~$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  9.9G  8.7G  54% /

plus a restart of postgres.

[tingox]

p2k16 had full disk again. As usual, I cleaned out /var/lib/postgresql/backups/, then restart postgres with sudo systemctl restart postgresql@10-main.service. Better now:

tingo@p2k16:~$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  8.3G   11G  45% /

[omega]

We could probably extend the backup service to only retain a set number of base backups, or have a different timer retain only N basebackups.

infrastructure/shared-roles/postgresql-wal-e/tasks/main.yml

Line 85 in 3dabb24

- name: systemd base backup service

Seems to be the code for installing the wal-e backup service, doing something similar for delete might be good enough

ExecStart=/usr/bin/env bash -c 'i="%i"; i=${i/-//}; bin/envdir /etc/wal-e/%i-env.d bin/wal-e delete --retain 5 --confirm'