E3SM Unified 1.12.0 testing

[forsyth2]

Chrysalis

Unit tests pass.

Testing steps

cd ~/ez/zstash
git status
# branch: test-main-20251003
# Handle uncommitted changes
git add -A
git commit -m "Testing" --no-verify
git status
# branch: test-main-20251003
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zstash/commits/main
# Bump to 1.5.0rc1 => bfa5cad3d72ad2cceb74fb5f211fecfef4c1937b
# e3sm_unified_1_12_0rc1
git checkout -b test_unified_1.12.0rc1_chrysalis bfa5cad3d72ad2cceb74fb5f211fecfef4c1937b
git log
# Good, last commit: Bump to 1.5.0rc1 (#389)
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc1_chrysalis.sh
python -m unittest tests/test_*.py
# Ran 70 tests in 131.154s
# OK (skipped=32)

cd ~/ez/zstash
python -m unittest tests2/test_*.py
# Ran 4 tests in 8.546s
# OK

Possible additional tests to run (some can only be run on certain machines): see list at #385 (comment).

[forsyth2]

e3sm_unified_1.12.0rc2

Chrysalis

zstash

Testing steps

See test improvements at #393. We can follow the new test README at https://github.com/E3SM-Project/zstash/blob/main/tests/README.md.

Remember, zstash rc3 is included in Unified rc2.

cd ~/ez/zstash
git status
# branch: issue-385-blocking-test
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zstash/commits/main
# Bump to 1.5.0rc3 => ce18814aa6b91af4eee09bb883c03141c2a355bd
# e3sm_unified_1_12_0rc2
git checkout -b test_unified_1.12.0rc2_chrysalis ce18814aa6b91af4eee09bb883c03141c2a355bd
git log
# Good, last commit: Bump to 1.5.0rc3 (#394)
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc2_chrysalis.sh

# Begin following directions from README:
pytest tests/unit/test_*.py
# 1 passed in 3.18s
python -m unittest tests/integration/python_tests/group_by_command/test_*.py # Will skip hsi tests
# Ran 69 tests in 56.870s
# OK (skipped=32)
python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
# Ran 4 tests in 3.510s
# OK
cd tests/integration/bash_tests/run_from_chrysalis/
time ./globus_auth.bash 15 # try_num=15
# real	2m27.691s
# No errors
time ./database_corruption.bash 15 # try_num=15
# Success count: 25
# Fail count: 0
# real	6m45.018s
time ./symlinks.sh 
# real	0m1.679s
# No errors

Success

[forsyth2]

e3sm_unified_1.12.0rc2

Compy (rc2)

zstash

Testing steps

See test improvements at #393. We can follow the new test README at https://github.com/E3SM-Project/zstash/blob/main/tests/README.md.

Remember, zstash rc3 is included in Unified rc2.

cd ~/ez/zstash
git status
# branch: main
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zstash/commits/main
# Bump to 1.5.0rc3 => ce18814aa6b91af4eee09bb883c03141c2a355bd
# e3sm_unified_1_12_0rc2
git checkout -b test_unified_1.12.0rc2_compy ce18814aa6b91af4eee09bb883c03141c2a355bd
git log
# Good, last commit: Bump to 1.5.0rc3 (#394)
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc2_compy.sh

# Begin following directions from README:
pytest tests/unit/test_*.py
# 1 passed in 1.18s
python -m unittest tests/integration/python_tests/group_by_command/test_*.py # # Will skip hsi tests
# Ran 69 tests in 140.506s
# OK (skipped=32)
python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
# Ran 4 tests in 7.469s
# OK

Success

[forsyth2]

e3sm_unified_1.12.0rc2

Perlmutter (rc2)

zstash

Testing steps

See test improvements at #393. We can follow the new test README at https://github.com/E3SM-Project/zstash/blob/main/tests/README.md.

Remember, zstash rc3 is included in Unified rc2.

cd ~/ez/zstash
git status
# branch: main
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zstash/commits/main
# Bump to 1.5.0rc3 => ce18814aa6b91af4eee09bb883c03141c2a355bd
# e3sm_unified_1_12_0rc2
git checkout -b test_unified_1.12.0rc2_perlmutter ce18814aa6b91af4eee09bb883c03141c2a355bd
git log
# Good, last commit: Bump to 1.5.0rc3 (#394)
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc2_pm-cpu.sh

# Begin following directions from README:
pytest tests/unit/test_*.py
# 1 passed in 0.49s
python -m unittest tests/integration/python_tests/group_by_command/test_*.py
# Ran 69 tests in 297.714s
# OK
python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
# Ran 4 tests in 2.547s
# OK
cd tests/integration/bash_tests/run_from_perlmutter/
time ./follow_symlinks.sh
# real	0m27.438s
# Good, no errors
time ./test_update_non_empty_hpss.bash
# real	0m8.740s
# Good, no errors

# Log into globus.org
# Log into endpoints (NERSC Perlmutter, Globus Tutorial Collection 1) at globus.org: File Manager > Add the endpoints in the "Collection" fields
time ./test_ls_globus.bash
#     raise self.error_class(r)
# globus_sdk.services.auth.errors.AuthAPIError: ('POST', 'https://auth.globus.org/v2/oauth2/token', None, 400, 'Error', 'Bad Request')
# Expected grep 'INFO: Adding 000000.tar' not found in create.log. Test failed.
#
# real	0m1.160s
# user	0m0.167s
# sys	0m0.182s

./test_ls_globus.bash failed. I'm seeing some auth issues to debug.

(Note that the issue didn't occur on Chrysalis or Compy because zstash has a number of machine-specific tests. To an extent, this can be improved in the future, but some tests will have to remain machine-specific because Perlmutter is the only machine with direct hsi access).

[forsyth2]

Fixes PR: #397

[forsyth2]

e3sm_unified_1.12.0rc3

Chrysalis

zstash

Testing steps

See test improvements at #393. We can follow the new test README at https://github.com/E3SM-Project/zstash/blob/main/tests/README.md.

Remember, zstash rc4 is included in Unified rc3.

cd ~/ez/zstash
git status
# branch main
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zstash/commits/main
# Bump to 1.5.0rc4 => e7a1e2253a00a9d0898ae799245df8216fe4a286
git checkout -b test_unified_1.12.0rc3_chrysalis e7a1e2253a00a9d0898ae799245df8216fe4a286
git log
# Good, last commit: Bump to 1.5.0rc4 (#399)
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc3_chrysalis.sh

# Begin following directions from README:
pytest tests/unit/test_*.py
# 1 passed in 2.31s
python -m unittest tests/integration/python_tests/group_by_command/test_*.py # Will skip hsi tests
# Ran 69 tests in 55.914s
# OK (skipped=32)
python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
# Ran 4 tests in 3.511s
# OK

cd tests/integration/bash_tests/run_from_any/
# Use try_num parameter as a more descriptive unique id: test_unified_1.12.0rc3_chrysalis
# Reminder 1:
# When NERSC asks for password it means without the MFA token. It will ask for that later.
# Compy, however, wants them combined.
# Reminder 2:
# If a test hangs, check if https://app.globus.org/activity reports any errors on your transfers.
./globus_auth.bash test_unified_1.12.0rc3_chrysalis chrysalis /home/ac.forsyth2/ez/zstash /home/ac.forsyth2/zstash_tests /global/homes/f/forsyth/zstash_tests /home/f/forsyth/zstash_tests /compyfs/fors729/zstash_tests
# All globus_auth tests completed successfully.

cd ~/ez/zstash/tests/integration/bash_tests/run_from_chrysalis/
# Reset completely:
# Revoke consents: https://auth.globus.org/v2/web/consents > Globus Endpoint Performance Monitoring > rescind all
# Run the following lines to set up the database_corruption test
# Alternative option: Rerun the `test_single_auth_code ${path_to_repo} NERSC_PERLMUTTER_ENDPOINT ${perlmutter_dst_dir}` line from `globus_auth.bash`
rm ~/.zstash_globus_tokens.json
mkdir zstash_demo; echo 'file0 stuff' > zstash_demo/file0.txt
# NERSC_PERLMUTTER_ENDPOINT=6bdc7956-fc0f-4ad2-989c-7aa5ee643a79
zstash create --hpss=globus://6bdc7956-fc0f-4ad2-989c-7aa5ee643a79//global/homes/f/forsyth/zstash/tests/test_database_corruption_setup23 zstash_demo
# You'll have to paste an auth code here, but NOT during the database_corruption test.
rm -rf zstash_demo/
# Use try_num parameter as a more descriptive unique id: test_unified_1.12.0rc3_chrysalis
time ./database_corruption.bash test_unified_1.12.0rc3_chrysalis
# Success count: 25
# Fail count: 0
# real	9m46.528s
time ./symlinks.sh
# real	0m1.629s
# No errors

Success

[forsyth2]

e3sm_unified_1.12.0rc3

Compy

zstash

Testing steps

See test improvements at #393. We can follow the new test README at https://github.com/E3SM-Project/zstash/blob/main/tests/README.md.

Remember, zstash rc4 is included in Unified rc3.

cd ~/ez/zstash
git status
# branch update-authentications
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zstash/commits/main
# Bump to 1.5.0rc4 => e7a1e2253a00a9d0898ae799245df8216fe4a286
git checkout -b test_unified_1.12.0rc3_chrysalis e7a1e2253a00a9d0898ae799245df8216fe4a286
git log
# Good, last commit: Bump to 1.5.0rc4 (#399)
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc3_compy.sh

# Begin following directions from README:
pytest tests/unit/test_*.py
# 1 passed in 1.28s
python -m unittest tests/integration/python_tests/group_by_command/test_*.py # # Will skip hsi tests
# Ran 69 tests in 149.817s
# OK (skipped=32)
python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
# Ran 4 tests in 7.383s
# OK

cd tests/integration/bash_tests/run_from_any/
# Skip ./globus_auth.bash test on Compy, because the transfer node uses a different file system/storage area than my workspace.

Success

[forsyth2]

e3sm_unified_1.12.0rc3

Perlmutter

zstash

Testing steps

See test improvements at #393. We can follow the new test README at https://github.com/E3SM-Project/zstash/blob/main/tests/README.md.

Remember, zstash rc4 is included in Unified rc3.

cd ~/ez/zstash
git status
# branch update-authentications
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zstash/commits/main
# Bump to 1.5.0rc4 => e7a1e2253a00a9d0898ae799245df8216fe4a286
git checkout -b test_unified_1.12.0rc3_perlmutter e7a1e2253a00a9d0898ae799245df8216fe4a286
git log
# Good, last commit: Bump to 1.5.0rc4 (#399)
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc3_pm-cpu.sh

# Begin following directions from README:
pytest tests/unit/test_*.py
# 1 passed in 0.54s
python -m unittest tests/integration/python_tests/group_by_command/test_*.py
# Ran 69 tests in 312.958s
# OK
python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
# Ran 4 tests in 2.360s
# OK

cd tests/integration/bash_tests/run_from_any/
# Use try_num parameter as a more descriptive unique id: test_unified_1.12.0rc3_pm-cpu
# Reminder 1:
# When NERSC asks for password it means without the MFA token. It will ask for that later.
# Compy, however, wants them combined.
# Reminder 2:
# If a test hangs, check if https://app.globus.org/activity reports any errors on your transfers.
./globus_auth.bash test_unified_1.12.0rc3_pm-cpu perlmutter /global/homes/f/forsyth/ez/zstash /home/ac.forsyth2/zstash_tests /global/homes/f/forsyth/zstash_tests /home/f/forsyth/zstash_tests /compyfs/fors729/zstash_tests
# Expected grep 'file_empty.txt' not found in run1_ls.log. Test failed.
#
# ERROR: ('POST', 'https://transfer.api.globus.org/v0.10/transfer', 'Bearer', 502, 'ExternalError', "Error validating login to endpoint 'NERSC Perlmutter (6bdc7956-fc0f-4ad2-989c-7aa5ee643a79)', Error (connect)\nEndpoint: NERSC Perlmutter (6bdc7956-fc0f-4ad2-989c-7aa5ee643a79)\nServer: 128.55.64.33:443\nMessage: The operation timed out\n",
# run2 completed successfully. Checking run2.log now.
#
# Try again?
rm -rf ../../../utils/globus_auth/
./globus_auth.bash _test_unified_1.12.0rc3_perlmutter perlmutter /global/homes/f/forsyth/ez/zstash /home/ac.forsyth2/zstash_tests /global/homes/f/forsyth/zstash_tests /home/f/forsyth/zstash_tests /compyfs/fors729/zstash_tests
# Expected grep 'file_empty.txt' not found in run1_ls.log. Test failed.
# 
# globus_sdk.services.transfer.errors.TransferAPIError: ('GET', 'https://transfer.api.globus.org/v0.10/operation/endpoint/6bdc7956-fc0f-4ad2-989c-7aa5ee643a79/ls?path=%2F%2Fglobal%2Fhomes%2Ff%2Fforsyth%2Fzstash_tests%2Ftest_globus_auth_try_test_unified_1.12.0rc3_perlmutter%2Frun1', 'Bearer', 404, 'ClientError.NotFound', "Directory '//global/homes/f/forsyth/zstash_tests/test_globus_auth_try_test_unified_1.12.0rc3_perlmutter/run1' not found on endpoint '6bdc7956-fc0f-4ad2-989c-7aa5ee643a79'",
ls /global/homes/f/forsyth/zstash_tests/test_globus_auth_try_test_unified_1.12.0rc3_perlmutter/
# run2
#
# So, it's correct that there is no run1 subdir.

# Let's come back to this tomorrow. Maybe something is wrong on the Globus side?

# Coming back to this
cd ~/ez/zstash
git status
# branch test_unified_1.12.0rc3_perlmutter
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc3_pm-cpu.sh
cd tests/integration/bash_tests/run_from_any/
rm -rf ../../../utils/globus_auth/
./globus_auth.bash _test_unified_1.12.0rc3_pm-cpu_try3 perlmutter /global/homes/f/forsyth/ez/zstash /home/ac.forsyth2/zstash_tests /global/homes/f/forsyth/zstash_tests /home/f/forsyth/zstash_tests /compyfs/fors729/zstash_tests
# ERROR: ('POST', 'https://transfer.api.globus.org/v0.10/transfer', 'Bearer', 502, 'ExternalError', 'Error validating login to endpoint \'NERSC Perlmutter (6bdc7956-fc0f-4ad2-989c-7aa5ee643a79)

# Make edits
rm -rf ../../../utils/globus_auth/
./globus_auth.bash test_unified_1.12.0rc3_pm-cpu_try4 perlmutter /global/homes/f/forsyth/ez/zstash /home/ac.forsyth2/zstash_tests /global/homes/f/forsyth/zstash_tests /home/f/forsyth/zstash_tests /compyfs/fors729/zstash_tests
# All globus_auth tests completed successfully.

cd ~/ez/zstash/tests/integration/bash_tests/run_from_perlmutter/
time ./follow_symlinks.sh # NOTE: you will have to change out paths for your username
# real	0m30.815s
# No errors
time ./test_update_non_empty_hpss.bash
# real	0m9.352s
# No errors

# Log into globus.org
# Log into endpoints (NERSC Perlmutter, Globus Tutorial Collection 1) at globus.org: File Manager > Add the endpoints in the "Collection" fields
time ./test_ls_globus.bash # NOTE: You may be asked to paste an auth-code
# real	0m56.596s
# No errors

# Push changes
cd ~/ez/zstash
nersc_conda # Activate conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zstash_dev_20251031
conda activate zstash_dev_20251031
pre-commit run --all-files
python -m pip install .
git add -A
git commit -m "Test updates for zstash rc5"
git branch -m test-updates-for-rc5
git push upstream test-updates-for-rc5

Success. #400 makes some minor improvements to the tests.

[forsyth2]

@TonyB9000 Once you've completed the long-running tests for zstash using the E3SM Unified 1.12.0rc2 or 1.12.0rc3, can you post results on this thread?

Expected behavior:

• You should be able to launch a zstash command, authenticate once, and have it continue. That is, there should be no need to delete a token file or authenticate on a "toy" problem before doing the real thing.

A couple things to note:

• You may have to remove the token file (rm ~/.zstash_globus_tokens.json) if you're switching between destination endpoints. This is to be handled later in [Feature]: Better handle the Globus token file #398
• At this point, I do not expect that we've resolved the tokens-not-refreshing problem (step 6 of [Feature]: Add refresh tokens for better Globus functionality #338). But let me know if the recent authentication fixes have had the side-effect of resolving that as well.

[TonyB9000]

@forsyth2 My base environment has no zstash, and my dev and production environments give zstash 1.4.4. I don't usually employ "E3SM Unified anything". How should I activate? I vaguely recall the last time I used "Unified" and it altered my terminal setting so I could no longer bring up "vi/vim" without clobbering my terminal scroll-back (it would overwrite anything I'd had on the screen, which would not return after exiting vi/vim).

[forsyth2]

How should I activate?

I tagged you on Slack, referencing the source commands listed for each machine. For the machines we test on, there's:

# Chrysalis 
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc3_chrysalis.sh
# Compy
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc3_compy.sh
# Perlmutter
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc3_pm-cpu.sh

[forsyth2]

Once you're in the Unified environment, you can run zstash as normal.

I vaguely recall the last time I used "Unified" and it altered my terminal setting so I could no longer bring up "vi/vim" without clobbering my terminal scroll-back

Hmm, I haven't had any issues with that. @andrewdnolan @xylar Has Tony's issue ever come up for either of you?

[xylar]

Nothing I've ever experienced.

[TonyB9000]

I guess I'll try my luck. I operate from Chrysalis.

I will switch to my "base" environment before I "source" unified. I expect to delete about 100 TB of my (recent) v3_LR native data, and then attempt to employ "zstash check" (path to NERSC) to retrieve. Hopefully, that will take a few days (~50+ hrs).

[forsyth2]

I will switch to my "base" environment before I "source" unified

Unified sets up its own conda. I have aliases/functions on all 3 machines (Chrysalis, Compy, Perlmutter) to set up conda for dev environments, but they're not needed when sourcing Unified. That is, I haven't ran into issues sourcing Unified regardless if I'm in a dev environment, base environment, or no environment at all.

Hopefully, that will take a few days (~50+ hrs).

Ok, sounds like a good test, thanks!

[TonyB9000]

@forsyth2 I just eliminated 180T of my local archive storage (e3sm2 now lists 360 TB free space).

I'll share my "zstash check" command (and anticipated volume) as soon as I craft and launch it. I will wipe my tokens file, just to ensure a fresh start. I've only used "globus web" the past few months, and I've only exercised (NERSC <--> Chrysalis) and (Chrysalis<-->Eagle) during that time.

[TonyB9000]

@forsyth2 Consider that I want to fetch the following archives:

LR:DECK,v3.LR.piControl,70,na,na,/home/g/golaz/E3SMv3.LR/v3.LR.piControl
(or)
LR:Historical,v3.LR.historical_0051,19,na,na,/home/w/wlin/E3SMv3/v3.LR.historical_0051
LR:Historical,v3.LR.historical_0101,19,na,na,/home/w/wlin/E3SMv3/v3.LR.historical_0101
LR:Historical,v3.LR.historical_0151,19,na,na,/home/w/wlin/E3SMv3/v3.LR.historical_0151
LR:Historical,v3.LR.historical_0201,19,na,na,/home/w/wlin/E3SMv3/v3.LR.historical_0201
LR:Historical,v3.LR.historical_0251,19,na,na,/home/w/wlin/E3SMv3/v3.LR.historical_0251

The first one is 70TB. The other 5 sum to 95TB.
Is there a way to do this as a single "zstash check" call? If I use a bash-loop to fetch each one sequentially, is that a valid test? (I could also launch each one in the background, 5 parallel "zstash check" calls. What happens if the console prompt to updated credentials occurs for parallel zstash calls? Do they interfere)

[forsyth2]

Is there a way to do this as a single "zstash check" call?

No, I wouldn't think so. Everything about zstash revolves around processing with one index.db (i.e., one archive).

If I use a bash-loop to fetch each one sequentially, is that a valid test?

Yes, I think so. Definitely from the perspective of testing that steps 2-4 of #339 have been condensed:

source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc3_chrysalis.sh

# If you want a COMPLETELY fresh start: 
rm ~/.zstash_globus_tokens.json
# Reset Globus consents: https://auth.globus.org/v2/web/consents > Globus Endpoint Performance Monitoring > rescind all

zstash check v3.LR.historical_0051 # You should be prompted to paste an auth code exactly once.
zstash check v3.LR.historical_0101 # For the 2nd and all following, you should NOT need to paste an auth code.
zstash check ...

I'm not sure if the refresh token problem (step 6 of #339) is tied to the token simply existing for too long or being used specifically for one long-running task. I imagine it's the former though, in which case again, the sequential approach would be fine. (Remember, we haven't done anything to expect the refresh token problem is fixed, but it'd be good to test if it got fixed by side-effect of other recent changes).

What happens if the console prompt to updated credentials occurs for parallel zstash calls?

Theoretically, if you authenticate on one of them, it would hold for the others, as long as it was for the same Globus endpoint.

[TonyB9000]

@forsyth2 @xylar Probably harmless, but:

Before and after "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc3_chrysalis.sh"

65,66c64,65
< -rw-------    1 ac.bartoletti1 cels       34688 Nov  5 14:44 .viminfo
< drwxr-xr-x   28 ac.bartoletti1 cels        8192 Nov  5 14:44 .
---
> -rw-------    1 ac.bartoletti1 cels       34123 Nov  5 14:46 .viminfo
> drwxr-xr-x   28 ac.bartoletti1 cels        8192 Nov  5 14:46 

My ".viminfo" just lost 500 bytes. Can we make these codes NOT change the user's viminfo files?

Proceeding with testing: I'll provide a blow-by-blow trace of what I do and what I see.

[TonyB9000]

@forsyth2 I'm not having much luck. When it gets to "paste this URL and login", and then "paste the code you get", it actys as if I'm pasting the code at the back commandline:

(e3sm_unified_1.12.0rc3_login) [ac.bartoletti1@chrlogin1 1_DataAcquisition]$ zstash version
For help, please see https://e3sm-project.github.io/zstash. Ask questions at https://github.com/E3SM-Project/zstash/discussions/categories/q-a.
v1.5.0rc4

(e3sm_unified_1.12.0rc3_login) [ac.bartoletti1@chrlogin1 1_DataAcquisition]$ ps gx
    PID TTY      STAT   TIME COMMAND
1297615 ?        S      0:00 sshd: ac.bartoletti1@pts/17
1297628 pts/17   Ss     0:00 -bash
1301497 ?        Rs     0:00 SCREEN -S zstash_test
1301498 pts/40   Ss     0:00 /bin/bash
1552443 pts/17   S+     0:00 screen -r 1301497.zstash_test
1557104 pts/40   R+     0:00 ps gx
3782353 ?        Ss     0:07 /usr/lib/systemd/systemd --user
3782355 ?        S      0:00 (sd-pam)
3782404 ?        Ss     0:00 ssh-agent -s

(e3sm_unified_1.12.0rc3_login) [ac.bartoletti1@chrlogin1 1_DataAcquisition]$ ./run_zstash_test.sh &
[1] 1557590
(e3sm_unified_1.12.0rc3_login) [ac.bartoletti1@chrlogin1 1_DataAcquisition]$ ISSUING zstash check command: zstash check -v --keep --cache archives --hpss=globus://9cd89cfd-6d04-11e5-ba46-22000b92c6ec//home/g/golaz/E3SMv3.LR/v3.LR.piControl
For help, please see https://e3sm-project.github.io/zstash. Ask questions at https://github.com/E3SM-Project/zstash/discussions/categories/q-a.
DEBUG: Opening index database
INFO: 20251105_213300_471113: in hpss_transfer, prev_transfers is starting as []
INFO: Transferring file from HPSS: archives/index.db
INFO: 20251105_213300_471426: DIVING: hpss calls globus_transfer(name=index.db)
INFO: 20251105_213300_471496: Entered globus_transfer() for name = index.db
DEBUG: 20251105_213300_471541: non_blocking = False
INFO: /home/ac.bartoletti1/.zstash.ini does NOT exist. This means we won't be able to read the local endpoint ID from it.
INFO: Token file /home/ac.bartoletti1/.zstash_globus_tokens.json does NOT exist. This means we won't be able to load tokens from it.
INFO: Writing to empty /home/ac.bartoletti1/.zstash.ini
INFO: Setting local_endpoint_id based on FQDN chrlogin1.lcrc.anl.gov: 15288284-7006-4041-ba1a-6b52501e49f1
INFO: No stored tokens found - starting authentication
Please go to this URL and login:
https://auth.globus.org/v2/oauth2/authorize?client_id=6c1629cf-446c-49e7-af95-323c6412397f&redirect_uri=https%3A%2F%2Fauth.globus.org%2Fv2%2Fweb%2Fauth-code&scope=urn%3Aglobus%3Aauth%3Ascope%3Atransfer.api.globus.org%3Aall%5B%2Ahttps%3A%2F%2Fauth.globus.org%2Fscopes%2F15288284-7006-4041-ba1a-6b52501e49f1%2Fdata_access+%2Ahttps%3A%2F%2Fauth.globus.org%2Fscopes%2F9cd89cfd-6d04-11e5-ba46-22000b92c6ec%2Fdata_access%5D&state=_default&response_type=code&code_challenge=KOZbHq_VeT-hXXgW4z0ipw0NtfeHscWCjR1sp1K2iQU&code_challenge_method=S256&access_type=offline
Please enter the code you get after login here: ufhSTot5ER9kLxHc8gb32hoQFPrDAN
bash: ufhSTot5ER9kLxHc8gb32hoQFPrDAN: command not found

[1]+  Stopped                 ./run_zstash_test.sh

(e3sm_unified_1.12.0rc3_login) [ac.bartoletti1@chrlogin1 1_DataAcquisition]$ history | tail -5
 1098  vi TEST_TRACE
 1099  ./run_zstash_test.sh &
 1100  ufhSTot5ER9kLxHc8gb32hoQFPrDAN
 1101  vi TEST_TRACE
 1102  history | tail

(e3sm_unified_1.12.0rc3_login) [ac.bartoletti1@chrlogin1 1_DataAcquisition]$ ps gx
    PID TTY      STAT   TIME COMMAND
1297615 ?        S      0:00 sshd: ac.bartoletti1@pts/17
1297628 pts/17   Ss     0:00 -bash
1301497 ?        Rs     0:00 SCREEN -S zstash_test
1301498 pts/40   Ss     0:00 /bin/bash
1552443 pts/17   S+     0:00 screen -r 1301497.zstash_test
1557590 pts/40   T      0:00 /bin/bash ./run_zstash_test.sh
1557591 pts/40   T      0:00 /lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/bin/python /lcrc/soft/climate/e3sm-unified/e3
1560638 pts/40   R+     0:00 ps gx
3782353 ?        Ss     0:07 /usr/lib/systemd/systemd --user
3782355 ?        S      0:00 (sd-pam)
3782404 ?        Ss     0:00 ssh-agent -s

Let me know what you think I should adjust.

[TonyB9000]

Addendum: When I paste the given URL, my globus page looks like this:

I select OK (should I?) and then get this page:

If that helps at all . . .

[forsyth2]

Can you please either give me the path to ./run_zstash_test.sh or copy its content here?

my globus page looks like this:

Yes that all looks right. The bigger issue seems to be that the script is set up such that the code is not being taken as input properly.

[TonyB9000]

@forsyth2 Sorry for the delay. Mu original try was to run this script:

#!/bin/bash

src_path=/home/g/golaz/E3SMv3.LR/v3.LR.piControl

NERSC_HPSS="9cd89cfd-6d04-11e5-ba46-22000b92c6ec"
local_cache="archives"

cmd="zstash check -v --keep --cache $local_cache --hpss=globus://$NERSC_HPSS/$src_path"

echo "ISSUING zstash check command: $cmd"

$cmd

echo "Launched command in background"

(I ran THAT script as "./run_zstash_test.sh &")

I just tried again, but this time put the "&" in the script itself, as in "$cmd &". The result threw an exception:

(e3sm_unified_1.12.0rc3_login) [ac.bartoletti1@chrlogin1 1_DataAcquisition]$ For help, please see https://e3sm-project.github.io/zstash. Ask questions at https://github.com/E3SM-Project/zstash/discussions/categories/q-a.
DEBUG: Opening index database
INFO: 20251105_234255_129608: in hpss_transfer, prev_transfers is starting as []
INFO: Transferring file from HPSS: archives/index.db
INFO: 20251105_234255_129954: DIVING: hpss calls globus_transfer(name=index.db)
INFO: 20251105_234255_130030: Entered globus_transfer() for name = index.db
DEBUG: 20251105_234255_130083: non_blocking = False
INFO: /home/ac.bartoletti1/.zstash.ini exists. We can try to read the local endpoint ID from it.
INFO: Token file /home/ac.bartoletti1/.zstash_globus_tokens.json does NOT exist. This means we won't be able to load tokens from it.
INFO: Setting local_endpoint_id based on /home/ac.bartoletti1/.zstash.ini:
INFO: Setting local_endpoint_id based on FQDN chrlogin1.lcrc.anl.gov: 15288284-7006-4041-ba1a-6b52501e49f1
INFO: No stored tokens found - starting authentication
Please go to this URL and login:
https://auth.globus.org/v2/oauth2/authorize?client_id=6c1629cf-446c-49e7-af95-323c6412397f&redirect_uri=https%3A%2F%2Fauth.globus.org%2Fv2%2Fweb%2Fauth-code&scope=urn%3Aglobus%3Aauth%3Ascope%3Atransfer.api.globus.org%3Aall%5B%2Ahttps%3A%2F%2Fauth.globus.org%2Fscopes%2F15288284-7006-4041-ba1a-6b52501e49f1%2Fdata_access+%2Ahttps%3A%2F%2Fauth.globus.org%2Fscopes%2F9cd89cfd-6d04-11e5-ba46-22000b92c6ec%2Fdata_access%5D&state=_default&response_type=code&code_challenge=EmcnjSyw1UM_cGPqEF7jRKnZdHgmRqS7y44lmuS8oBg&code_challenge_method=S256&access_type=offline
Please enter the code you get after login here: Traceback (most recent call last):
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/bin/zstash", line 9, in <module>
    sys.exit(main())
             ~~~~^^
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/zstash/main.py", line 73, in main
    check()
    ~~~~~^^
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/zstash/check.py", line 12, in check
    extract.extract(keep_files=False)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/zstash/extract.py", line 45, in extract
    failures: List[FilesRow] = extract_database(args, cache, keep_files)
                               ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/zstash/extract.py", line 178, in extract_database
    hpss_get(hpss, get_db_filename(cache), cache)
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/zstash/hpss.py", line 166, in hpss_get
    hpss_transfer(hpss, file_path, "get", cache, False)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/zstash/hpss.py", line 111, in hpss_transfer
    globus_status = globus_transfer(
        endpoint, url_path, name, transfer_type, non_blocking
    )
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/zstash/globus.py", line 82, in globus_transfer
    globus_activate("globus://" + remote_ep)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/zstash/globus.py", line 49, in globus_activate
    transfer_client = get_transfer_client_with_auth(both_endpoints)
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/zstash/globus_utils.py", line 168, in get_transfer_client_with_auth
    auth_code = input("Please enter the code you get after login here: ").strip()
                ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
EOFError: EOF when reading a line

To be honest, I should launch zstash from a small python "subprocess" call, as that would more closely match the true use-case.

QUESTION: When you need to launch a job that will last several days, do you use "nohup" or "screen" (or something else) to protect your process from terminal hang-ups? I am using "screen".

[forsyth2]

At the risk of this becoming AI talking to AI, here's my thread:

Me: Why is my colleague encountering the attached issue? zstash asked him to paste a Globus auth code, but when he did, it seemed to interpret it as a command: command not found; My bash script works fine, also attached.

(Note: attached bash script was the auth test)

Claude's response:

---

The issue your colleague is encountering is due to running the zstash command in the background with &.

When they modified the script to include $cmd &, the zstash process was sent to the background. The problem is that zstash needs to read user input (the Globus authentication code) from stdin, but background processes cannot read from the terminal's stdin.

Looking at the error:

Please enter the code you get after login here: Traceback (most recent call last):
...
EOFError: EOF when reading a line

The input() function in Python tried to read from stdin but got an EOF (End Of File) because the process was backgrounded and stdin was closed/unavailable.

Solution

For your colleague's use case, there are a few approaches:

1. Don't background the initial authentication step: Run the command normally the first time to complete authentication, which will save tokens to ~/.zstash_globus_tokens.json. Subsequent runs won't need interactive input and can be safely backgrounded.

2. Use screen or tmux (recommended for long-running jobs): Your colleague mentioned using screen, which is perfect. They should:

   • Start a screen session: screen
   • Run the zstash command normally (without &)
   • Complete the authentication when prompted
   • Detach with Ctrl+A, D
   • The process continues running even if they disconnect

3. Use nohup with a wrapper script (for subprocess approach): If launching from Python subprocess, ensure the authentication happens in an interactive session first, then subsequent calls can use nohup.

The key insight from your bash test script is that it expects interactive authentication for the first run, then stores tokens for subsequent runs. Background execution (&) breaks this interactive authentication flow.

Answer to the specific question: Yes, for multi-day jobs, screen (or tmux) is the recommended approach over nohup when you need initial interactive authentication, because it maintains the terminal session including stdin/stdout connectivity.

[TonyB9000]

@forsyth2 ASIDE: I asked the great AI for some guidance regarding python subprocess in this scenario:

ME:

When using python "subprocess" (with NO waiting) to launch a (long-running) child background process, and that child will occasionally issue a "prompt" to the console (for re-authorization on expired credentials or tokens), is there a special way to issue the subprocess call to handle this behavior?

THE AI:

Great question! If you want to launch a long-running child process using Python’s subprocess without waiting (i.e., non-blocking), and you want the child’s console prompts (such as for re-authorization) to appear and be interactive on your console, you need to let the child process inherit the parent’s standard input, output, and error streams.

By default, if you use subprocess.Popen without specifying stdin, stdout, or stderr, the child will use the parent’s console. This means prompts will appear and you can interact with them as if you launched the process manually.
Recommended Usage

import subprocess

# Launch child process in background, inheriting console I/O
p = subprocess.Popen(['python', 'my_app.py'])
# Parent continues running; child can prompt on console as needed

    Do NOT set stdin=subprocess.PIPE, stdout=subprocess.PIPE, or stderr=subprocess.PIPE if you want the child to interact with the console.
    The child process will display prompts and accept input from the terminal as normal.

Special Notes

    If you launch from a script running in a non-interactive environment (like a cron job), the child won’t have a console to prompt.
    If you want to programmatically handle the child’s prompts, you’d need to use stdin=subprocess.PIPE and communicate with the process, but this is more complex.

In summary:
Just use subprocess.Popen([...]) with default arguments, and your child process will be able to prompt and interact with the console as expected.

ME:

What if the parent process is run in a detachable "screen", and that screen is later resumed (re-attached). The child (and parent) terminal is never itself closed. Upon re-attachment, any prompt issues should still appear (as would stdout, etc). Can this work?

THE AI:

Yes, this will work as you expect!

If you launch your parent Python process inside a detachable terminal multiplexer like screen (or tmux), and you use subprocess.Popen without redirecting stdin, stdout, or stderr, both the parent and child processes will use the same terminal session provided by screen.
How It Works

    Parent and child processes inherit the terminal file descriptors from screen.
    When you detach and later re-attach to the screen session, any output, prompts, or input requests from either the parent or child will appear in the terminal, just as if you never left.
    Prompts for credentials or other interactive input from the child process will still show up on the terminal when you re-attach, and you can respond to them.

Example

screen -S mysession
python parent_script.py
# parent_script.py launches child with subprocess.Popen(['python', 'child_script.py'])
# Detach with Ctrl-A D
# Re-attach later with screen -r mysession
# Any prompts from child_script.py will be visible and interactive

Key Points

    Do not redirect stdin, stdout, or stderr in your Python subprocess.Popen call.
    As long as the screen session is running, the terminal is "alive" and handles all I/O for both parent and child.
    If you re-attach, you see all output and can respond to prompts.

In short:
As long as you launch your processes inside a screen session and do not redirect their I/O, interactive prompts and output will appear as expected after you re-attach. This is a common and reliable workflow for long-running or interactive processes on Linux systems.

I think I should work on launching this from a python parent, rather than a bash script, to more closely emulate how I will need to do it from the "process wrappers" I will use in production.

[TonyB9000]

@forsyth2 Regarding Claude's solution: If during a long run, credentials (or token) expire and require an interactive shell to supply input, we would need a way to bring the background job to the foreground (first, I believe, because otherwise its prompt will get bogus input (EOF).

A solution might be to NEVER place the processing in the background, and simply detach/reattach the "screen" employed for that job. Upon re-attach, that prompt should be waiting there for input.

[forsyth2]

A solution might be to NEVER place the processing in the background, and simply detach/reattach the "screen"

This is how I use screen, yes.

[forsyth2]

e3sm_unified_1.12.0rc4

Chrysalis

zstash

Testing steps

Following the test README at https://github.com/E3SM-Project/zstash/blob/main/tests/README.md.

cd ~/ez/zstash
git status
# branch test_unified_1.12.0rc3_chrysalis
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zstash/commits/main
# Test updates for zstash rc5 (#400)
# Bump to 1.5.0rc4 (#399)

# #400 only changed files in `tests/` so we can include thatas well
git checkout -b test_unified_1.12.0rc4_chrysalis upstream/main
git log
# Good, commits match
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc4_chrysalis.sh

pytest tests/unit/test_*.py
# 1 passed in 2.19s
python -m unittest tests/integration/python_tests/group_by_command/test_*.py
# Ran 69 tests in 56.885s
# OK (skipped=32)
python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
# Ran 4 tests in 3.258s
# OK

cd tests/integration/bash_tests/run_from_any/
# Review the directions at the bottom of globus_auth.bash
# Run `./globus_auth.bash` with the appropriate parameters:
./globus_auth.bash test_unified_1.12.0rc4_chrysalis chrysalis /home/ac.forsyth2/ez/zstash /home/ac.forsyth2/zstash_tests /global/homes/f/forsyth/zstash_tests /home/f/forsyth/zstash_tests /compyfs/fors729/zstash_tests
# All globus_auth tests completed successfully.

cd ~/ez/zstash
cd tests/integration/bash_tests/run_from_chrysalis/
# Reset completely:
# Revoke consents: https://auth.globus.org/v2/web/consents > Globus Endpoint Performance Monitoring > rescind all
# Run the following lines to set up the database_corruption test
rm ~/.zstash_globus_tokens.json
mkdir zstash_demo; echo 'file0 stuff' > zstash_demo/file0.txt
# NERSC_PERLMUTTER_ENDPOINT=6bdc7956-fc0f-4ad2-989c-7aa5ee643a79
zstash create --hpss=globus://6bdc7956-fc0f-4ad2-989c-7aa5ee643a79//global/homes/f/forsyth/zstash/tests/test_database_corruption_setup23 zstash_demo
# You'll have to paste an auth code here, but NOT during the database_corruption test.
rm -rf zstash_demo/
time ./database_corruption.bash database_corruption_test_unified_1.12.0rc4_chrysalis
# Success count: 25
# Fail count: 0
# real	6m37.189s
time ./symlinks.sh
# real	0m5.459s
# No errors

Success

[forsyth2]

e3sm_unified_1.12.0rc4

Perlmutter

zstash

Testing steps

Following the test README at https://github.com/E3SM-Project/zstash/blob/main/tests/README.md.

cd ~/ez/zstash
git status
# branch test-updates-for-rc5
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zstash/commits/main
# Test updates for zstash rc5 (#400)
# Bump to 1.5.0rc4 (#399)

# #400 only changed files in `tests/` so we can include thatas well
git checkout -b test_unified_1.12.0rc4_perlmutter upstream/main
git log
# Good, commits match
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc4_pm-cpu.sh

pytest tests/unit/test_*.py
#  1 passed in 0.54s
python -m unittest tests/integration/python_tests/group_by_command/test_*.py
# Ran 69 tests in 302.179s
# OK
python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
# Ran 4 tests in 2.301s
# OK

cd tests/integration/bash_tests/run_from_any/
# Review the directions at the bottom of globus_auth.bash
# Run `./globus_auth.bash` with the appropriate parameters:
./globus_auth.bash globus_auth_test_unified_1.12.0rc4_pm perlmutter /global/homes/f/forsyth/ez/zstash /home/ac.forsyth2/zstash_tests /global/homes/f/forsyth/zstash_tests /home/f/forsyth/zstash_tests /compyfs/fors729/zstash_tests
# All globus_auth tests completed successfully.

cd ~/ez/zstash
cd tests/integration/bash_tests/run_from_perlmutter/
time ./follow_symlinks.sh
# real	0m28.343s
# No errors
time ./test_update_non_empty_hpss.bash
# real	0m8.961s
# No errors

# Log into globus.org
# Log into endpoints (NERSC Perlmutter, Globus Tutorial Collection 1) at globus.org: File Manager > Add the endpoints in the "Collection" fields
time ./test_ls_globus.bash
# Paste auth code
# real	0m40.364s
# No errors

Success

[forsyth2]

e3sm_unified_1.12.0rc4

Compy

zstash

Testing steps

Following the test README at https://github.com/E3SM-Project/zstash/blob/main/tests/README.md.

cd ~/ez/zstash
git status
# branch test_unified_1.12.0rc3_chrysalis # Mislabeled
# nothing to commit

# REMINDER: `git fetch upstream main` is NOT guaranteed to work on Compy!
# Checking out a branch from upstream/main might not actually 
# base it on ALL the most recent commits!
# Not specifying a branch at all seems to be the best method:
git fetch upstream 
# https://github.com/E3SM-Project/zstash/commits/main
# Test updates for zstash rc5 (#400)
# Bump to 1.5.0rc4 (#399)

# #400 only changed files in `tests/` so we can include thatas well
git checkout -b test_unified_1.12.0rc4_compy upstream/main
git log
# Good, commits match
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc4_compy.sh

pytest tests/unit/test_*.py
# 1 passed in 1.07s
python -m unittest tests/integration/python_tests/group_by_command/test_*.py
# Ran 69 tests in 136.090s
# OK (skipped=32)
python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
# Ran 4 tests in 6.662s
# OK

Success

[TonyB9000]

@forsyth2 When I made a recent try, I was prompted to apply the URL and submit the code, I did so, and is was "accepted" (tokens file updated or something), but then immediately issues a series of exceptions, the final element of which reads:

  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/globus_sdk/client.py", line 522, in request
    raise self.error_class(r)
globus_sdk.services.transfer.errors.TransferAPIError: ('GET', 'https://transfer.api.globus.org/v0.10/operation/endpoint/9cd89cfd-6d04-11e5-ba46-22000b92c6ec/ls?path=%2F%2Fhome%2Fg%2Fgolaz%2FE3SMv3.LR%2Fv3.LR.piControl', 'Bearer', 502, 'ExternalError.DirListingFailed.LoginFailed', 'Command Failed: Error (login)\nEndpoint: NERSC HPSS (9cd89cfd-6d04-11e5-ba46-22000b92c6ec)\nServer: 128.55.205.31:443\nMessage: Login Failed\n---\nDetails: 530-Login incorrect. : GlobusError: v=1 c=LOGIN_DENIED\\r\\n530-GridFTP-Message: None of your authenticated identities are from domains allowed by resource policies\\r\\n530-GridFTP-JSON-Result: {"DATA_TYPE": "result#1.1.0", "authorization_parameters": {"session_required_single_domain": ["nersc.gov"]}, "code": "permission_denied", "detail": {"DATA_TYPE": "not_from_allowed_domain#1.0.0", "allowed_domains": ["nersc.gov"]}, "has_next_page": false, "http_response_code": 403, "message": "None of your authenticated identities are from domains allowed by resource policies"}\\r\\n530 End.\\r\\n\n', '2EzmqAF9P')

When I tried again, it did not even prompt me with a URL - it used the "cashed tokens" and threw the same exceptions.

I wanted to capture all of it, but have difficulty (loss of scrollback) so I asked AI:

My terminal (Windows Powershell), logged into a remote linux system and having invoked "screen", will no longer support output-scrollback. I need to launch a long-running python program that will first prompt me to supply input at the command line. The input is accepted, but soon a long list of exceptions are generated, and I cannot capture the beginning of that output as it has scrolled beyond the current screenful.

Is there a command (like "tee") I can use, both to redirect console output to a file and yet not miss the prompt for input I must satisfy?

The AI replied:

Yes, you can use the tee command to capture output to a file while still seeing it on your terminal and being able to interact with the program. However, there’s a caveat:

    If you simply run python myscript.py | tee output.txt, this will not work well for interactive programs that prompt for input, because the prompt may not appear, or input may not be handled correctly (the program’s stdin is not the terminal anymore).

Solution: Use script or unbuffer (from expect or moreutils)
1. Using script

The script command records your entire terminal session, including both input and output, and handles interactive programs well.

script -c "python myscript.py" output.txt

    This will run your Python program, let you interact as normal, and everything (input and output) will be saved in output.txt.

2. Using unbuffer (from expect package)

If you want to use tee and your program buffers output, you can use unbuffer to force line buffering:

unbuffer python myscript.py | tee output.txt

    This lets you see output and also saves it, and should preserve interactivity.

3. Using stdbuf (for line buffering)

If you have stdbuf available:

stdbuf -oL python myscript.py | tee output.txt

    This sets Python’s stdout to be line-buffered, which helps with prompt visibility.

SO ... I will try again with "script"

THE RESULT:

(e3sm_unified_1.12.0rc3_login) [ac.bartoletti1@chrlogin2 1_DataAcquisition]$ script -c "./run_zstash_test.sh" OUTPUT.TXT

(more OUTPUT.TXT)

Script started on 2025-11-06 16:00:50-06:00
ISSUING zstash check command: zstash check -v --keep --cache archives --hpss=globus://9cd89cfd-6d04-11e5-ba46-22000b92c6ec//home/g/golaz/E3SMv3.LR/v3.LR.piControl
For help, please see https://e3sm-project.github.io/zstash. Ask questions at https://github.com/E3SM-Project/zstash/discussions/categories/q-a.
DEBUG: Opening index database
INFO: 20251106_220054_905896: in hpss_transfer, prev_transfers is starting as []
INFO: Transferring file from HPSS: archives/index.db
INFO: 20251106_220054_906088: DIVING: hpss calls globus_transfer(name=index.db)
INFO: 20251106_220054_906124: Entered globus_transfer() for name = index.db
DEBUG: 20251106_220054_906149: non_blocking = False
INFO: /home/ac.bartoletti1/.zstash.ini exists. We can try to read the local endpoint ID from it.
INFO: Token file /home/ac.bartoletti1/.zstash_globus_tokens.json exists. We can try to load tokens from it.
INFO: Setting local_endpoint_id based on /home/ac.bartoletti1/.zstash.ini:
INFO: Setting local_endpoint_id based on FQDN chrlogin2.lcrc.anl.gov: 15288284-7006-4041-ba1a-6b52501e49f1
INFO: Found stored refresh token - using it
Traceback (most recent call last):
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/bin/zstash", line 9, in <module>
    sys.exit(main())
             ~~~~^^
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/zstash/main.py", line 73
, in main
    check()
    ~~~~~^^
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/zstash/check.py", line 12
, in check
    extract.extract(keep_files=False)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/zstash/extract.py", line 4
5, in extract
    failures: List[FilesRow] = extract_database(args, cache, keep_files)
                               ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/zstash/extract.py", line 1
78, in extract_database
    hpss_get(hpss, get_db_filename(cache), cache)
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/zstash/hpss.py", line 166
, in hpss_get
    hpss_transfer(hpss, file_path, "get", cache, False)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/zstash/hpss.py", line 111
, in hpss_transfer
    globus_status = globus_transfer(
        endpoint, url_path, name, transfer_type, non_blocking
    )
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/zstash/globus.py", line 88
, in globus_transfer
    archive_directory_listing = transfer_client.operation_ls(
        remote_endpoint, remote_path
    )
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/globus_sdk/services/transfer/clie
nt.py", line 1378, in operation_ls
    self.get(f"operation/endpoint/{endpoint_id}/ls", query_params=query_params)
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/globus_sdk/client.py", line
334, in get
    return self.request(
           ~~~~~~~~~~~~^
        "GET",
        ^^^^^^
    ...<3 lines>...
        automatic_authorization=automatic_authorization,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )e--(77%)
    ^e--(77%)
  File "/lcrc/soft/climate/e3sm-unified/e3smu_1_12_0rc3/chrysalis/conda/envs/e3sm_unified_1.12.0rc3_login/lib/python3.13/site-packages/globus_sdk/client.py", line
5m522, in request
    raise self.error_class(r)
globus_sdk.services.transfer.errors.TransferAPIError: ('GET', 'https://transfer.api.globus.org/v0.10/operation/endpoint/9cd89cfd-6d04-11e5-ba46-22000b92c6ec/ls
?path=%2F%2Fhome%2Fg%2Fgolaz%2FE3SMv3.LR%2Fv3.LR.piControl', 'Bearer', 502, 'ExternalError.DirListingFailed.LoginFailed', 'Command Failed: Error (login)\nEndpoint: NERSC HP
SS (9cd89cfd-6d04-11e5-ba46-22000b92c6ec)\nServer: 128.55.205.31:443\nMessage: Login Failed\n---\nDetails: 530-Login incorrect. : GlobusError: v=1 c=LOGIN_DENIED\\r\\n530-G
ridFTP-Message: None of your authenticated identities are from domains allowed by resource policies\\r\\n530-GridFTP-JSON-Result: {"DATA_TYPE": "result#1.1.0", "authorizati
on_parameters": {"session_required_single_domain": ["nersc.gov"]}, "code": "permission_denied", "detail": {"DATA_TYPE": "not_from_allowed_domain#1.0.0", "allowed_domains":
["nersc.gov"]}, "has_next_page": false, "http_response_code": 403, "message": "None of your authenticated identities are from domains allowed by resource policies"}\\r\\n53
0 End.\\r\\n\n', '3UzJzQxr9')

Script done on 2025-11-06 16:00:57-06:00

Suggestions? Are there files I should purge for a fresh start?

[forsyth2]

Great, when did you launch this? Nov. 6? So it's been 4 days without a token expiration? In #339, Chris stated:

Note that this transfer is limited to 48 hours due to Globus token expiration. Given the relatively slow transfer speeds between chrysalis and NERSC HPSS (~100 MB/s), 48 hours is often insufficient for a large simulation.

So maybe we have in fact fixed that issue!

[TonyB9000]

The first SUCCEEDED was 11/07 at 1:42 AM UTC

TASK DETAILS
Task ID: 71ab0409-bb76-11f0-9dcb-027493648695
Task Type: TRANSFER
Status: SUCCEEDED
Source: NERSC HPSS
Destination: LCRC Improv DTN
Label: v3LRpiControl 000000
Request Time: 2025-11-07 01:09:44.582228 (UTC) Completion Time: 2025-11-07 01:42:57.357059 (UTC) Files Transfered: 1 Directories Transfered: 0 Bytes Transfered: 274792304640 Effective Speed: 137894307 Bytes per Second Transfer Settings:
 - transfer is not encrypted
 - preserve source file modification times
 - quota errors prevent transfer retry
 - overwriting all files on destination

and the latest is this morning (11/10 at 14:49 PM UTC)

TASK DETAILS
Task ID: 6734803f-be44-11f0-ae37-0affdd0cd947
Task Type: TRANSFER
Status: SUCCEEDED
Source: NERSC HPSS
Destination: LCRC Improv DTN
Label: v3LRpiControl 00008d
Request Time: 2025-11-10 14:49:08.393180 (UTC) Completion Time: 2025-11-10 15:14:06.843429 (UTC) Files Transfered: 1 Directories Transfered: 0 Bytes Transfered: 269077340160 Effective Speed: 179570420 Bytes per Second Transfer Settings:
 - transfer is not encrypted
 - preserve source file modification times
 - quota errors prevent transfer retry
 - overwriting all files on destination

(the notifications say "0 bytes transferred", but the tar files are arriving like clockwork, at least up until now. 3.5 days running.

I need to log in to the right "chrlogin(1,2)" or else I cannot reattach the screen to look for a prompt..

Back in 1 hour...

[TonyB9000]

@forsyth2 Bad News. No further transfers after 00008d.tar, and after logging in BOTH to chrlogin1 and chrlogin2, neither shows zstash running, and neither shows any "screen" I had used to protect the session from hang-up (perhaps "maintenance monday" is at fault, but I did not think that affected the login nodes.) Because I neglected to "script" or "tee", I have no log-output to examine.

I am going to delete the ~40 TB of files h=that were transferred, and retry. (At least we know it ran 3.5 days without incident.)

This time, I will be sure to record the node I'm on, to record the name of the screen invoked, and use script to "tee" the output for tracing.

[forsyth2]

At least we know it ran 3.5 days without incident.

Yes, this at least confirms that #380 worked (thus reducing steps 2-4 of #339). So, that is good.

[TonyB9000]

@forsyth2 @chengzhuzhang Very Frustrating.. This morning, I find I am "logged off" of chrysalis (chrlogin2, half-expected), finally log back in (after "file-system issues"), locate the "detached" screen session, and re-attach. The console shows a "failed write" to my OUTPUT_2.txt file (invoked with "script" to capture "tee'd" zstash output) and the process table shows zstash no longer active. Up until 7:30 AM local time, I was receiving a tar file every 25 minutes, but nothing since.

For long-running jobs (2-4+ days), we need a truly-robust solution for processing - one that can withstand system crashes/restarts and likewise file-system breakdowns.

Right now, I should be able to "re-start" my previous zstash invocation, and have zstash "discover" that tar-files 000000.tar through 000015.tar (21 tar files) were successfully fetched, and pick up fetching the remaining 235 tar files.

Even my proposed "Transfer Service via Tickets" solution would need to have generated tickets "per tarfile" in order to automate the proper restart activity. Where MY invocation had "archive-granularity" ("get me this archive"), zstash itself would need to retain a sense of job-id and "files processed", in order to be invoked with the same job-id under a "resume" operation.

Yes, we can manually discover which tar-files remain, and (somehow) specify only the remaining files be fetched - but with a more robust solution, we could cron a "No zstash Service? Then Restart" capability, and then a breakdown mid-vacation would not interrupt a 7-day process, stretching it to 20+ days. At least, that should be the goal.

ASIDE: What if I were to turn my "fetch archive" (zstash check --hpss=path_to_archive) into 256 individual "fetch tarfile" (zstash check --hpss=path_to_tarfile)and submit each one as a separate slurm-srun job? I'm guessing that the slurm system is build to withstand downtime and recovery automatically.