[Feature]: Improve zstash testing

[forsyth2]

How will this affect the next version number?

New feature (increment MINOR version)

Is your feature request related to a problem?

Current testing has a number of problems:

• So-called "unit" testing is really more integration testing because it involves I/O rather than pure function testing. This leads to two points:
  • It may perhaps be better to convert the "unit" tests to bash scripts that check output. This would allow more transparency in what workflows are being tested. For example, one implementation of this is tests/scripts/globus_auth.bash.
  • Actual unit tests should test functions used by zstash rather than the actual zstash commands called from the command line.
• The current test framework uses unittest rather than pytest. I recall trying to modernize that in the past, but running into problems.
• The current test framework always uses the GitHubActions or conda/dev.yml Python version. We should simulate running zstash create with an older version of Python and/or zstash and then running zstash extract on a later version. For example, see the thread at Update Python support #384 (comment). It's a common use case that someone would be extracting data that was archived a while ago.

Describe the solution you'd like

Described above.

Describe alternatives you've considered

No response

Additional context

No response

[forsyth2]

Standard unit tests

Expand

# Perlmutter
cd ~/ez/zstash
git status
# No uncommited changes
git fetch upstream main
git checkout -b test-main-20251003 upstream/main
git log
# Last commit: Update docutils constraint (#386)
# Good, matches https://github.com/E3SM-Project/zstash/commits/main
nersc_conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zstash-main-20251003
conda activate zstash-main-20251003
pre-commit run --all-files
python -m pip install .
python -m unittest tests/test_*.py
# Ran 70 tests in 330.113s
# OK

# Chrysalis
cd ~/ez/zstash
git status
# No uncommited changes
git fetch upstream main
git checkout -b test-main-20251003 upstream/main
git log
# Last commit: Update docutils constraint (#386)
# Good, matches https://github.com/E3SM-Project/zstash/commits/main
lcrc_conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zstash-main-20251003
conda activate zstash-main-20251003
pre-commit run --all-files
python -m pip install .
python -m unittest tests/test_*.py
# Ran 70 tests in 52.831s
# OK (skipped=32)

Extra tests

Expand

# Chrysalis
cd ~/ez/zstash/tests/scripts
# To start fresh with Globus:
# 1. Log into endpoints (LCRC Improv DTN, NERSC Perlmutter) at globus.org: File Manager > Add the endpoints in the "Collection" fields
# 2. To start fresh, with no consents: https://auth.globus.org/v2/web/consents > Manage Your Consents > Globus Endpoint Performance Monitoring > rescind all"
# Edit ~/ez/zstash/tests/scripts/globus_auth.bash
TRY_NUM=9
git diff # Make sure we edited on Chrysalis, not Perlmutter
./globus_auth.bash
# Success

# Chrysalis
cd ~/ez/zstash/tests/scripts
./symlinks.sh
# Success; note -- test is really just a check that it runs

# Perlmutter
cd ~/ez/zstash/tests/scripts
# Edit ~/ez/zstash/tests/scripts/follow_symlinks.sh
# conda_env=zstash-main-20251003
# source /global/homes/f/forsyth/miniforge3/etc/profile.d/conda.sh
# cd /global/homes/f/forsyth/ez/zstash/
# python -m pip install ../..
git diff # Make sure we edited on Perlmutter, not Chrysalis
./follow_symlinks.sh
# Success; note -- test is really just a check that it runs

# Perlmutter
cd /global/homes/f/forsyth/zstash/tests/
# Empty, no need for:
# rm -rf test_database_corruption
# Chrysalis
cd ~/ez/zstash/tests/scripts
./database_corruption.bash
# Success count: 25
# Fail count: 0

# Perlmutter
# Log into endpoints (Globus Tutorial Collection 1) at globus.org: File Manager > Add the endpoints in the "Collection" fields
cd ~/ez/zstash/tests/scripts_unit_tests
./test_ls_globus.bash
# Note -- test is really just a check that it runs
# Fails on final line because of a missing path
# Presumably, the hpss_path needs to be replaced with a different string
rm -rf zstash_test/

# Perlmutter
cd ~/ez/zstash/tests/scripts_unit_tests
./test_update_non_empty_hpss.bash
# Success; note -- test is really just a check that it runs

# Perlmutter
cd ~/ez/zstash
python -m unittest tests2/test_*.py
# Ran 4 tests in 3.514s
# OK

# Chrysalis
cd ~/ez/zstash
python -m unittest tests2/test_*.py
# Ran 4 tests in 5.943s
# OK

# Perlmutter
ls ~/ez/zstash/tests3
# This was Tony's testing for a PR

Test summary

The current state of zstash testing, with tests ordered by decreasing usability. Currently, python -m unittest tests/test_*.py is the primary test suite, with the other tests being extra and/or used in specific PRs.

	Has automatic assertions/checks	Runs on Perlmutter	Runs on Chrysalis	Works without test file edits	Specific PR?
cd ~/ez/zstash; python -m unittest tests/test_*.py HPSS tests	✅	✅		✅
cd ~/ez/zstash; python -m unittest tests/test_*.py non-HPSS tests	✅	✅	✅	✅
cd ~/ez/zstash; python -m unittest tests2/test_*.py	✅	✅	✅	✅
cd tests/scripts; ./globus_auth.bash (Replaced Globus unit test)	✅		✅	❌	#380
cd tests/scripts; ./database_corruption.bash	✅		✅	✅	#378
cd tests/scripts; ./symlinks.sh	❌		✅	✅	#343
cd tests/scripts_unit_tests; ./test_update_non_empty_hpss.bash	❌	✅		✅	#363
cd tests/scripts; ./follow_symlinks.sh	❌	✅		❌	#261
cd tests/scripts_unit_tests; ./test_ls_globus.bash	❌	✅		❌ (still needs editing)	#363
~/ez/zstash/tests3 (need to review testing instructions)	❓	❓	❓	❓	#355

[forsyth2]

Based on the primary rows, we can proceed with making rc1 but we should at some point resolve this issue of improving the tests... and also making them more uniform in terms of how to run, where to run from, etc.

[forsyth2]

#393 improved test structure, but it doesn't implement all the ideas listed above, so I'm re-opening this.