Skip to content

Implement cleanup #7

Description

@natefoo

It appears that on both CFDE instances I never implemented cleanup, both in-Galaxy data cleanup via pgcleanup.py and deletion of data from tmp/jobs/etc. And cleanup_jobs is set to never.

For reference, .org uses this script which templates out as:

#!/usr/bin/env bash
# set -eo pipefail not used here because removals in scratch dirs while running cause find to exit nonzero
set -u

MUTEX="$HOME/.galaxy-clean.lock"
MUTEX_ACQUIRED=false

function trap_handler() {
    $MUTEX_ACQUIRED && rmdir "$MUTEX"
    return 0
}
trap "trap_handler" EXIT

if mkdir "$MUTEX"; then
    MUTEX_ACQUIRED=true
{
    set -x;
    /bin/find /corral4/main/jobs/*/* -mindepth 1 -maxdepth 1 -type d -mtime +7 -exec rm -rvf {} + | tee -a /srv/galaxy/main/log/cleanup_jobs.log && find /corral4/main/jobs/* -mindepth 0 -maxdepth 1 -type d -empty -exec rmdir {} +;
    /bin/find /corral4/main/scratch3 -mindepth 1 -maxdepth 1 -mtime +5 ! -name '.nfs*' -exec rm -rvf {} + | tee -a /srv/galaxy/main/log/cleanup_scratch.log;
    /bin/find /corral4/main/_upload_tus -mindepth 1 -maxdepth 1 -mtime +5 ! -name '.nfs*' -exec rm -rvf {} + | tee -a /srv/galaxy/main/log/cleanup_tus_upload_store.log;
    /bin/find /corral4/main/_upload_job_files -mindepth 1 -maxdepth 1 -mtime +5 ! -name '.nfs*' -exec rm -rvf {} + | tee -a /srv/galaxy/main/log/cleanup_job_files_upload_store.log;
    /bin/find /corral4/main/pulsar_amqp_ack/*/consume -mindepth 1 -maxdepth 1 -mtime +14 ! -name '.nfs*' -exec rm -rvf {} + | tee -a /srv/galaxy/main/log/cleanup_amqp_ack_consume.log;
    /bin/find /corral4/main/celery-scratch -mindepth 1 -maxdepth 1 -mtime +14 ! -name '.nfs*' -exec rm -rvf {} + | tee -a /srv/galaxy/main/log/cleanup_celery_scratch.log;
    /srv/galaxy/main/bin/cleanup_datasets.sh 7 | tee -a /srv/galaxy/main/log/cleanup_datasets_sh.log;
    { set +x; } 2>/dev/null;
} > /srv/galaxy/main/log/cleanup_daily-$(date +%Y-%m-%d).log 2>&1
fi

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions