Skip to content

Commit ad767cf

Browse files
CBroz1CBroz1
authored
Add script for drive space check (#1257)
* Add script for drive space check * Update changelog * Abs space limit and slack message * Drive-specific limits * Add root alias
1 parent a31600c commit ad767cf

File tree

4 files changed

+184
-12
lines changed

4 files changed

+184
-12
lines changed

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
- Ensure merge tables are declared during file insertion #1205
88
- Update URL for DANDI Docs #1210
99
- Add common method `get_position_interval_epoch` #1056
10-
- Improve cron job documentation and script #1226, #1241
10+
- Improve cron job documentation and script #1226, #1241, #1257
1111
- Update export process to include `~external` tables #1239
1212
- Only add merge parts to `source_class_dict` if present in codebase #1237
1313
- Remove cli module #1250

‎maintenance_scripts/README.md‎

Lines changed: 45 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,18 @@ regularly as cron jobs.
1717
- `populate.py` - This script provides an example of how to run computations as
1818
part of cron jobs. This is not currently in use.
1919
- `run_jobs.sh` - This script ...
20+
- Reads from the `.env` file in the same directory.
2021
- Updates the spyglass repository, fetching from the master branch.
2122
- Runs a database connection check (relying on a valid datajoint config).
2223
- Runs the `cleanup.py` script.
24+
- `check_disk_space.sh` - This script ...
25+
- Reads from the same `.env` file as `run_jobs.sh`.
26+
- Checks the disk space of each drive in `SPACE_CHECK_DRIVES`
27+
- Calculate the percent available relative to `SPACE_PERCENT_LIMIT`.
28+
- If above, sends an email for each drive to each recipient listed in
29+
`SPACE_EMAIL_RECIPENTS`.
30+
- If provided with `SLACK_TOKEN` and `SLACK_CHANNEL`, posts a message to
31+
slack.
2332

2433
## Setup
2534

@@ -28,26 +37,51 @@ regularly as cron jobs.
2837
`dj_local_conf.json` and filling in the necessary information.
2938
3. Copy the `example.env` file to `.env` in the `maintenance_scripts` directory
3039
and fill in the necessary information, including...
31-
- `SPYGLASS_CONDA_ENV`: the name of the conda environment with Spyglass and
32-
DataJoint installed.
33-
- `SPYGLASS_REPO_PATH`: the path to the Spyglass repository.
34-
- `SPYGLASS_LOG`: the path to the log file.
35-
- Optional email settings. If not set, email notifications will not be sent.
36-
- `SPYGLASS_EMAIL_SRC`: The email address from which to send notifications.
37-
- `SPYGLASS_EMAIL_PASS`: the password for the email address.
38-
- `SPYGLASS_EMAIL_DEST`: the email address to which to send notifications.
39-
4. Set up a cron job to run `run_jobs.sh` at the desired interval by running
40+
- Items for running cleanup jobs:
41+
- `SPYGLASS_CONDA_ENV`: the name of the conda environment with Spyglass and
42+
DataJoint installed.
43+
- `SPYGLASS_REPO_PATH`: the path to the Spyglass repository.
44+
- `SPYGLASS_LOG`: the path to the log file.
45+
- Optional email settings. If not set, email notifications will not be
46+
sent.
47+
- `SPYGLASS_EMAIL_SRC`: The email address from which to send
48+
notifications.
49+
- `SPYGLASS_EMAIL_PASS`: the password for the email address.
50+
- `SPYGLASS_EMAIL_DEST`: the email address to which to send
51+
notifications.
52+
- Items for checking disk space:
53+
- `TZ`: the timezone to use reporting local times.
54+
- `SPACE_PERCENT_LIMIT`: the percentage of disk space below which to send
55+
notifications.
56+
- `SPACE_CHECK_DRIVES`: a space-separated list of drives to check.
57+
- `SPACE_LOG`: the path to the log file.
58+
- `SPACE_EMAIL_SRC`/`SPACE_EMAIL_PASS`: email sender settings for disk
59+
space notifications.
60+
- `SPACE_EMAIL_RECIPENTS`: a space-separated list of email addresses to
61+
receive disk space notifications.
62+
- Items for posting to slack:
63+
- `SLACK_TOKEN`: the token for the slack app.
64+
- `SLACK_CHANNEL`: the channel to post to.
65+
4. Set up a cron job to run each shell script at the desired interval by running
4066
`crontab -e` and adding the script.
4167

4268
Note that the log file will automatically be truncated to `SPYGLASS_MAX_LOG`
4369
lines on each run. 1000 lines should be sufficient.
4470

45-
### Example Cron Job
71+
To enable slack notifications, you will need to create a slack app and generate
72+
a token following the instructions
73+
[here](https://api.slack.com/tutorials/tracks/posting-messages-with-curl).
74+
For posting to a private channel, you will need to invite the app to the
75+
relevant channel before attempting to post.
4676

47-
In the following example, the script is set to run every Monday at 4:00 AM.
77+
### Example Cron Jobs
78+
79+
In the following example, the cleanup script is set to run every Monday at 4:00
80+
AM, and the disk space check is set to run every day at 8:00 AM.
4881

4982
```text
5083
0 4 * * 1 /path/to/run_jobs.sh
84+
0 8 * * * /path/to/check_disk_space.sh
5185
```
5286

5387
### Email Service

‎maintenance_scripts/check_disk_space.sh‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
#!/bin/bash
2+
# This script will...
3+
# 1. Read variables from a .env file
4+
# 2. Loop through $SPACE_CHECK_DRIVES
5+
# 3. Compare available space relative to $SPACE_LIMIT
6+
# 4. If above, send an email to $SPACE_EMAIL_RECIPIENTS
7+
8+
# Load env
9+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
10+
source "$SCRIPT_DIR/.env" # load environment variables from this directory
11+
12+
# Check for required variables
13+
if [[ -z "${SPACE_CHECK_DRIVES}" \
14+
|| -z "${SPACE_EMAIL_SRC}" \
15+
|| -z "${SPACE_EMAIL_PASS}" \
16+
|| -z "${SPACE_EMAIL_RECIPIENTS}" \
17+
|| -z "${SPACE_DRIVE_LIMITS}" \
18+
|| -z "${SPACE_LOG}" \
19+
|| -z "${SPACE_ROOT_NAME}" ]] ; then
20+
echo "Error: Missing one or more variables required for check_disk_space.sh"
21+
exit 1
22+
fi
23+
24+
# Inputs to arrays
25+
IFS=' ' read -r -a DRIVE_LIST <<< "$SPACE_CHECK_DRIVES"
26+
IFS=' ' read -r -a LIMIT_LIST <<< "$SPACE_DRIVE_LIMITS"
27+
28+
if [[ "${#SPACE_CHECK_DRIVES[@]}" -ne "${#SPACE_DRIVE_LIMITS[@]}" ]]; then
29+
echo "Error: Number of drives does not match number of limits"
30+
exit 1
31+
fi
32+
33+
echo "SPACE CHECK: $(date)" > "$SPACE_LOG"
34+
35+
# Email template
36+
EMAIL_TEMPLATE=$(cat <<-EOF
37+
From: "Spyglass" <$SPACE_EMAIL_SRC>
38+
To: %s
39+
Subject: Drive almost full: %s
40+
41+
%s
42+
EOF
43+
)
44+
45+
# Send email alert
46+
send_email_message() {
47+
local RECIPIENT="$1"
48+
local SUBJECT="$2"
49+
local BODY="$3"
50+
EMAIL=$(printf "$EMAIL_TEMPLATE" "$RECIPIENT" "$SUBJECT" "$BODY")
51+
curl -s --url "smtps://smtp.gmail.com:465" \
52+
--ssl-reqd \
53+
--user "$SPACE_EMAIL_SRC:$SPACE_EMAIL_PASS" \
54+
--mail-from "$SPACE_EMAIL_SRC" \
55+
--mail-rcpt "$RECIPIENT" \
56+
-T <(echo "$EMAIL")
57+
}
58+
59+
# Send slack message
60+
send_slack_message() {
61+
if [[ -z "$SLACK_TOKEN" || -z "$SLACK_CHANNEL" ]]; then
62+
return 0
63+
fi
64+
local MESSAGE="$1"
65+
curl -d "text=$MESSAGE" \
66+
-d "channel=$SLACK_CHANNEL" \
67+
-H "Authorization: Bearer $SLACK_TOKEN" \
68+
-X POST https://slack.com/api/chat.postMessage
69+
}
70+
71+
# Find the longest drive name for padding
72+
MAX_DRIVE_LEN=0
73+
for DRIVE in $SPACE_CHECK_DRIVES; do
74+
LEN=${#DRIVE}
75+
[[ $LEN -gt $MAX_DRIVE_LEN ]] && MAX_DRIVE_LEN=$LEN
76+
done
77+
78+
# Check each drive
79+
for i in "${!DRIVE_LIST[@]}"; do
80+
DRIVE="${DRIVE_LIST[$i]}"
81+
LIMIT="${LIMIT_LIST[$i]}"
82+
83+
# Convert limit to bytes
84+
SPACE_LIMIT_BYTES=$(numfmt --from=iec "$LIMIT")
85+
86+
# Skip nonexistent drives
87+
if [[ ! -d "$DRIVE" ]]; then
88+
echo "WARNING: Drive $DRIVE not found. Skipping." >> "$SPACE_LOG"
89+
continue
90+
fi
91+
92+
# Get free space in bytes. Multiply by 1024 to convert kilo -> bytes.
93+
FREE_BYTES=$(df --output=avail "$DRIVE" | awk 'NR==2 {print $1 * 1024}')
94+
TOTAL_BYTES=$(df --output=size "$DRIVE" | awk 'NR==2 {print $1 * 1024}')
95+
FREE_HUMAN=$(numfmt --to=iec "$FREE_BYTES")
96+
TOTAL_HUMAN=$(numfmt --to=iec "$TOTAL_BYTES")
97+
98+
# Log with left-padded drive names
99+
if [[ "$DRIVE" == "/" ]]; then
100+
NAME="$SPACE_ROOT_NAME" # Use custom name for root
101+
else
102+
NAME="${DRIVE:1}" # Assumes first char is `/`
103+
fi
104+
printf "%-*s: %s/%s\n" "$MAX_DRIVE_LEN" "$NAME" \
105+
"$FREE_HUMAN" "$TOTAL_HUMAN" >> "$SPACE_LOG"
106+
107+
# Do nothing if under capacity
108+
if [[ "$FREE_BYTES" -gt "$SPACE_LIMIT_BYTES" ]]; then
109+
continue
110+
fi
111+
112+
# Send email alert
113+
BODY="Low space warning: ${NAME} has ${FREE_HUMAN}/${TOTAL_HUMAN} free"
114+
SUBJ="LOW SPACE: ${NAME}"
115+
116+
echo $BODY >> "$SPACE_LOG"
117+
118+
send_slack_message "$BODY"
119+
120+
for RECIPIENT in $SPACE_EMAIL_RECIPIENTS; do
121+
send_email_message "$RECIPIENT" "$SUBJ" "$BODY"
122+
done
123+
124+
done
125+

‎maintenance_scripts/example.env‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,21 @@
11
# NOTE: Use quotes for strings with spaces
2+
# ----------------------------------------
3+
# For cleanup:
24
export SPYGLASS_CONDA_ENV=spyglass # name of conda environment
35
export SPYGLASS_REPO_PATH=/home/franklab/spyglass # path to spyglass repo
46
export SPYGLASS_LOG=/home/franklab/spyglass.log # log file
57
export SPYGLASS_EMAIL_SRC=[email protected] # optional, email sender
68
export SPYGLASS_EMAIL_PASS="example password" # optional, email password
79
export SPYGLASS_EMAIL_DEST=[email protected] # optional, email recipient
810
export SPYGLASS_MAX_LOG=1000 # number of lines to keep in log
11+
# For checking disk space. Use iec format (e.g. 10T for 10 terabytes) for limits
12+
export TZ=America/Los_Angeles # see `tzselect`
13+
export SPACE_ROOT_NAME="root" # name of root drive
14+
export SPACE_CHECK_DRIVES="/drive1 /drive2 /" # may err or drive with space
15+
export SPACE_DRIVE_LIMITS="10T 10G 3G" # space separated, same order as above
16+
export SPACE_LOG=$SPYGLASS_LOG
17+
export SPACE_EMAIL_SRC=$SPYGLASS_EMAIL_SRC
18+
export SPACE_EMAIL_PASS=$SPYGLASS_EMAIL_PASS
19+
export SPACE_EMAIL_RECIPIENTS="[email protected] [email protected]" # space separated
20+
export SLACK_TOKEN="xoxb-not-a-real-token-this-will-not-work" # optional, slack token
21+
export SLACK_CHANNEL="B056AQM5VFZ" # optional, slack channel

0 commit comments

Comments
 (0)