Improve tests (#393)

* Add test assertions

* Restructure testing directories

* Add more test assertions

* Fix GitHub Actions and update README

* Update Chrysalis tests

* Add blocking test notes

Author: forsyth2

.github/workflows/build_workflow.yml

@@ -82,9 +82,12 @@ jobs:
         run: |
           python -m pip install .
 
+      # Run machine-independent tests
       - name: Run Tests
         run: |
-          python -m unittest tests/test_*.py
+          pytest tests/unit/test_*.py
+          python -m unittest tests/integration/python_tests/group_by_command/test_*.py
+          python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
 
   # Build documentation on all PRs and pushes; only publish (commit to gh-pages) on pushes.
   build-docs:

.gitignore

@@ -1,7 +1,7 @@
 build/
 dist/
-tests/test_follow_symlinks/
-tests/test_follow_symlinks_non_archived/
+tests/utils/test_follow_symlinks
+tests/utils/test_follow_symlinks_non_archived
 zstash.egg-info/
 *.pyc
 *~

conda/dev.yml

@@ -8,7 +8,7 @@ dependencies:
   - python >=3.11,<3.14
   - sqlite
   - six >=1.16.0
-  - globus-sdk >=3.15.0
+  - globus-sdk >=3.15.0,<4.0
   # Developer Tools
   # =================
   # If versions are updated, also update 'rev' in `.pre-commit.config.yaml`
@@ -18,6 +18,10 @@ dependencies:
   - mypy ==1.18.2
   - pre-commit ==4.3.0
   - tbump >=6.9.0
+  # Testing
+  # =======================
+  - pytest
+  - pytest-cov
   # Documentation
   # =================
   # If versions are updated, also update in `.github/workflows/workflow.yml`

tests/README.md

@@ -0,0 +1,120 @@
+# zstash Test Suite
+
+## The test directory structure
+
+```
+tests/
+  integration/ # Tests that call zstash from the command line
+    bash_tests/ # Test zstash commands via bash scripts
+      run_from_chrysalis/ # Run these from Chrysalis (these use Globus and/or require the Chrysalis file system)
+      run_from_perlmutter/ # Run these from Perlmutter (these use `hsi` directly and/or require the Perlmutter file system)
+    python_tests/ # Test zstash commands via Python unittest wrappers
+      group_by_command # Tests organized by command
+      group_by_workflow # Tests organized by workflow
+  unit/ # Tests of pure functions (uses pytest, not unittest)
+  utils/ # Utilities for testing
+```
+
+## Testing example for Perlmutter
+
+```bash
+rm -rf build
+conda clean --all --y
+conda env create -f conda/dev.yml -n zstash_dev_20251017_test1
+conda activate zstash_dev_20251017_test1
+pre-commit run --all-files
+python -m pip install .
+pytest tests/unit/test_*.py
+# 1 passed in 0.19s
+python -m unittest tests/integration/python_tests/group_by_command/test_*.py
+# Ran 69 tests in 327.570s
+# OK
+python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
+# Ran 4 tests in 2.666s
+# OK
+cd tests/integration/bash_tests/run_from_perlmutter/
+time ./follow_symlinks.sh # NOTE: you will have to change out paths for your username
+# real	0m31.851s
+# No errors
+time ./test_update_non_empty_hpss.bash
+# real	0m10.062s
+# No errors
+
+# Log into globus.org
+# Log into endpoints (NERSC Perlmutter, Globus Tutorial Collection 1) at globus.org: File Manager > Add the endpoints in the "Collection" fields
+time ./test_ls_globus.bash
+# real	0m26.930s
+# No errors
+```
+
+## Testing example for Chrysalis
+
+```bash
+rm -rf build
+conda clean --all --y
+conda env create -f conda/dev.yml -n zstash_dev_20251017_test1
+conda activate zstash_dev_20251017_test1
+pre-commit run --all-files
+python -m pip install .
+pytest tests/unit/test_*.py
+# 1 passed in 0.84s
+python -m unittest tests/integration/python_tests/group_by_command/test_*.py
+# Ran 69 tests in 110.139s
+# OK (skipped=32)
+# NOTE: Some tests are skipped because Chrysalis doesn't have direct `hsi`/HPSS access
+python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
+# Ran 4 tests in 6.889s
+# OK
+cd tests/integration/bash_tests/run_from_chrysalis/
+
+# Log into globus.org
+# 1. Log into endpoints (LCRC Improv DTN, NERSC Perlmutter) at globus.org: File Manager > Add the endpoints in the "Collection" fields
+# 2. To start fresh, with no consents: https://auth.globus.org/v2/web/consents > Manage Your Consents > Globus Endpoint Performance Monitoring > rescind all"
+# Then, increment `try_num` below to avoid using an old directory.
+# Alternatively, start fresh by deleting the directory on Perlmutter:
+# `rm -rf /global/homes/f/forsyth/zstash/tests/test_globus_auth_try{try_num}`
+time ./globus_auth.bash try_num # NOTE: you will have to change out paths for your username
+# Paste the URL into your browser
+# Log into Argonne
+# Log into NERSC
+# Provide a label
+# Copy the auth code to the command line
+#
+# real	2m45.954s
+# No errors
+
+# If not done above, do the following:
+# Log into globus.org
+# Log into endpoints (LCRC Improv DTN, NERSC Perlmutter) at globus.org: File Manager > Add the endpoints in the "Collection" fields
+
+# In all cases, do:
+# Then, increment `try_num` below to avoid using an old directory.
+# Alternatively, start fresh by deleting the directory on Perlmutter:
+# `rm -rf /global/homes/f/forsyth/zstash/tests/test_database_corruption_try{try_num}`
+time ./database_corruption.bash try_num # NOTE: you will have to change out paths for your username
+# Success count: 25
+# Fail count: 0
+# real	6m43.994s
+
+time ./symlinks.sh # NOTE: you will have to change out paths for your username
+# real	0m1.346s
+# No errors
+
+cd blocking_test_scripts
+# Review README_TEST_BLOCKING
+# This uses "12 piControl ocean monthly files, 49 GB",
+# so processing may take a long time.
+# TODO (later PR): Confirm this test works
+```
+
+## Testing with GitHub Actions
+
+GitHub Actions runs the tests according to `.github/workflows/build_workflow.yml`:
+```
+      # Run machine-independent tests
+      - name: Run Tests
+        run: |
+          pytest tests/unit/test_*.py
+          python -m unittest tests/integration/python_tests/group_by_command/test_*.py
+          python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
+```

tests/integration/__init__.py

@@ -67,15 +67,7 @@ II. Running the Test Script
 
 The test script "test_zstash_blocking.sh" accepts two positional parameters:
 
-    test_zstash_blocking.sh (BLOCKING|NON_BLOCKING) [NEW_CREDS]
-
-On an initial run, or whenever Globus complains of authentication failures,
-add "NEW_CREDS" as the second parameter.  This will act to delete your
-cached Globus credentials and trigger prompts for you to paste login URLs
-to your browser (generally one per endpoint) which requires that you conduct
-a login sequence, and then paste a returned key-value at the bash command
-prompt.  After both keys are accepted, you can re-run the test script
-without "NEW_CREDS", until the credentials expire (usually 24 hours.)
+    test_zstash_blocking.sh (BLOCKING|NON_BLOCKING)
 
 If "BLOCKING" is selected, zstash will run in default mode, waiting for
 each tar file to complete transfer before generating another tar file.
@@ -93,7 +85,7 @@ so that your command prompt returns and you can monitor progress with
     snapshot.sh
 
 which will provide a view of both the tarfile cache and the destination
-directory for delivred tar files. It is also suggested that you name your
+directory for delivered tar files. It is also suggested that you name your
 logfile to reflect the date, and whether BLOCKING or not was specified.