Add nightly Presto benchmark script for NVL72 Slurm cluster - #382
Add nightly Presto benchmark script for NVL72 Slurm cluster#382misiugodfrey wants to merge 13 commits into
Conversation
Adds presto/slurm/presto-nvl72/nightly-benchmark.sh, a daemon script
intended to run in a persistent tmux/screen session. It sleeps until
3 AM EST (08:00 UTC) each night, then:
1. Pulls the latest nightly-pinned coordinator and gpu-cuda13.1 worker
images from GHCR into ${IMAGE_DIR}/nightly-temp (leaving the
cluster's main IMAGE_DIR untouched).
2. Runs a 1-node / 4-GPU TPC-H sf1000 benchmark via launch-run.sh.
3. Runs a 2-node / 8-GPU TPC-H sf3000 benchmark via launch-run.sh.
4. Posts both results to the benchmarking DB via post_results.py.
5. Cleans up the temporary images.
A failed night's run prints a warning and retries the next night without
killing the daemon. All output is tee'd to a persistent log file under
RESULTS_BASE.
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
These will be auto-populated from image metadata by the time this script is in use.
…ning
Set RESULTS_BASE to ${VT_ROOT}/results before sourcing defaults.env so the
workspace derivation logic (which converts path separators to dashes) doesn't
produce a non-existent path when the repo is nested more than one level under
$HOME.
| BENCHMARK_TEST_DIR=${TEST_DIR}/performance_benchmarks | ||
| pytest -q -s ${BENCHMARK_TEST_DIR}/${BENCHMARK_TYPE}_test.py ${PYTEST_ARGS[*]} | ||
| pytest_exit=0 | ||
| pytest -q -s ${BENCHMARK_TEST_DIR}/${BENCHMARK_TYPE}_test.py ${PYTEST_ARGS[*]} || pytest_exit=$? |
There was a problem hiding this comment.
Needed to avoid an early exit here on failure since that prevents the logs from being moved. This is to enable the results to be pushed even if there was a pytest failure when running the benchmarks.
| [[ "${JOB_STATE}" == "COMPLETED" ]] || exit 1 | ||
|
|
||
| if [[ -n "${OUTPUT_PATH}" ]]; then | ||
| if [[ -n "${OUTPUT_PATH}" && -f "result_dir/benchmark_result.json" ]]; then |
There was a problem hiding this comment.
Only copy the results over if there are results to copy - this is instead of checking if the job state was "COMPLETED" (which means without failures).
|
This is designed to run with PR #338 so it will need re-testing once that lands. |
patdevinwilson
left a comment
patdevinwilson
left a comment
There was a problem hiding this comment.
Useful daemon for NVL72 nightlies. A couple of correctness issues around failure handling and LOGS_DIR (esp. relative to #338) before this should land. Details inline.
| export PRESTO_DATA_DIR=/var/lib/presto/data/hive/data/user_data; \ | ||
| export MINIFORGE_HOME=/workspace/miniforge3; \ | ||
| export HOME=/workspace; \ | ||
| export LOGS_DIR=/var/log/nsys; \ |
There was a problem hiding this comment.
This conflicts with the direction in #338, which mounts ${LOGS}:/opt/presto-server/logs and exports LOGS_DIR=/opt/presto-server/logs so provenance + server logs are visible to pytest.
/var/log/nsys is the nsys token/report mount, not the general logs dir used by launch_presto_servers.sh / run_context.py. If this merges before/without #338's /opt/presto-server/logs mount on the CLI container, log snapshotting / GPU detection / provenance may miss files depending on which run_coord_image path is used.
Prefer aligning with #338 (LOGS_DIR=/opt/presto-server/logs + the matching mount), or land #338 first and rebase this.
There was a problem hiding this comment.
I think we should wait for #338 to land and then I'll rebase this.
- Add || echo_warning to both post_results calls so a DB posting failure (API down, bad SKU, etc.) warns but does not abort the remaining run - Add trap to run cleanup_images on SIGINT/SIGTERM so temp .sqsh files are removed when the daemon is stopped manually in tmux/screen - Log active COORD_TAG and WORKER_TAG at startup so stale image tags are visible in the log
|
Went through 555869a — replied inline, but summarizing:
|
Resolves conflict in presto/scripts/run_benchmark.sh: take main's PYTEST_EXIT variable name and improved exit logic that preserves both benchmark and validation exit codes.
mattgara
left a comment
mattgara
left a comment
There was a problem hiding this comment.
LGTM, there are three items that may need another look.
- Capture run_nightly exit status explicitly so set -e is active inside the subshell rather than being suppressed by the || conditional - Gate post_results on benchmark_result.json (not just the output dir) so partial/empty result dirs are not posted - Copy all artifacts to OUTPUT_PATH in launch-run.sh regardless of whether benchmark_result.json exists, preserving logs from failed runs - Run cleanup_images via srun so deletions execute on the compute node where IMAGE_DIR is mounted; source launcher_common.sh for partition/ account resolution
| [[ "${RUN_NOW}" -eq 1 ]] || sleep_until_next_run | ||
| RUN_NOW=0 | ||
| nightly_exit=0 | ||
| (run_nightly) || nightly_exit=$? |
There was a problem hiding this comment.
Can we please move (run_nightly) out of the OR-list and capture its status with errexit (-e) disabled only in the parent. I think the new assignment changes the right-hand side, but the subshell is still the left-hand side of ||, so bash suppresses errexit throughout run_nightly (I assume this is not the intended case.)
There was a problem hiding this comment.
I think I better understand the ask here. I've isolated this idiom and put it in it's own function run_isolated().
| [[ "${JOB_STATE}" == "COMPLETED" ]] || exit 1 | ||
|
|
||
| if [[ -n "${OUTPUT_PATH}" ]]; then | ||
| if [[ -n "${OUTPUT_PATH}" && -d "result_dir" ]]; then |
There was a problem hiding this comment.
I believe this still loses logs from failures before collect_results. Consider copying the available job and server logs into ${OUTPUT_PATH} before returning the failed job status, or collecting them from an EXIT trap. AFAICT, result_dir is created unconditionally at startup, but populated from logs/ only after run_queries succeeds.
There was a problem hiding this comment.
I think the overall logging for the slurm scripts needs a bit of an update. For now I've moved the log collection into the same trap that handles metadata injection - so the logs should always get copied over if they exist.
…lated - Refactor run_nightly invocation into run_isolated() which runs a function in a subshell with errexit explicitly active, calls a cleanup function unconditionally, and calls an optional failure callback — all without suppressing errexit via || - Add collect_logs_on_exit EXIT trap in run-presto-benchmarks.sh so logs are always copied into result_dir regardless of where the job fails (result_dir is created unconditionally by the launcher) - Remove log copying from collect_results() since the EXIT trap handles it; collect_results() now only copies configs - Simplify launch-run.sh OUTPUT_PATH copy to not gate on result_dir (which always exists) or benchmark_result.json
|
Holding out on merging until I can re-test the final version on the cluster. |
3 participants
Summary
presto/slurm/presto-nvl72/nightly-benchmark.sh, a daemon script for running automated nightly Presto benchmarks on the NVL72 Slurm clustergpu-cuda13.1worker images from GHCR into a temporary directory (leaves the cluster's mainIMAGE_DIRuntouched), then removes them after both runs completepost_results.pyafter each runtmux/screensession; sleeps until 3 AM EST (08:00 UTC) each night usingdatearithmeticTest plan
bash -n nightly-benchmark.shRUN_HOUR_UTC=$(date -u +%H) ./nightly-benchmark.sh— should fire within seconds.sqshfiles appear in${IMAGE_DIR}/nightly-temp/during a run and are removed on completionpost_results.pyis invoked with correct--benchmark-name,--cache-state warm, and storage config per run