Skip to content

Conversation

@dir01
Copy link

@dir01 dir01 commented Dec 29, 2025

No description provided.

@dir01
Copy link
Author

dir01 commented Dec 29, 2025

Hi @brokenpip3

We are experiencing an issue where during abnormal termination of backups, old garbage tmp.XXXXXXXXX directories continue to live in our pvc wasting space

At some point, this leads to catastrophic failure where jenkins fails to restore from a backup on startup and so the whole instance remains broken until someone manually cleans up stale directories

_log "INFO" "[backup] running backup ${BACKUP_NUMBER}"
touch "${TRAP_FILE}"

# since at this point we are sure that no other backup process is running, we can clean up stale tmp dirs
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it safe to assume that? I mean, trap file protects us against concurrent runs with the same number, but is it possible that another backup with a lower or higher number runs concurrently?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant