Unexpected file size during zstash check

[darincomeau]

I'm running zstash check, and have come across an error:

INFO: zstash/000201.tar exists. Checking expected size matches actual size.
INFO: 000201.tar: expected size=274534901760 != 274548275200=actual_size

What's the best way to proceed here?

Thanks.

[forsyth2]

Do you have more of the output? Does an exception actually get raised?

From https://github.com/E3SM-Project/zstash/blob/main/zstash/extract.py:

                if not os.path.exists(tfname):
                    do_retrieve = True
                else:
                    do_retrieve = not check_sizes_match(cur, tfname)

                try:
                    if test_retry:
                        test_retry = False
                        raise RuntimeError
                    if do_retrieve:
                        hpss_get(hpss, tfname, cache)
                        if not check_sizes_match(cur, tfname):
                            raise RuntimeError(
                                f"{tfname} size does not match expected size."
                            )

If there is a size mismatch, zstash retrieves the file. If there is still a size mismatch after that, an error gets raised.

[darincomeau]

Sorry, I should have just posted the full traceback:

INFO: Globus transfer 4156f442-fef9-11ed-9bbc-c9bb788c490e, from 9cd89cfd-6d04-11e5-ba46-22000b92c6ec to 61f9954c-a4fa-11ea-8f07-0a21f750d19b: 20221116CRYO1950ne30pg2_SOwISC12to60E2r4N2Dependentsubmesochrysalis 000201 succeeded
INFO: zstash/000201.tar exists. Checking expected size matches actual size.
INFO: 000201.tar: expected size=274534901760 != 274548275200=actual_size
Traceback (most recent call last):
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.8.0_nompi/bin/zstash", line 10, in <module>
    sys.exit(main())
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.8.0_nompi/lib/python3.10/site-packages/zstash/main.py", line 68, in main
    check()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.8.0_nompi/lib/python3.10/site-packages/zstash/check.py", line 12, in check
    extract.extract(keep_files=False)
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.8.0_nompi/lib/python3.10/site-packages/zstash/extract.py", line 45, in extract
    failures: List[FilesRow] = extract_database(args, cache, keep_files)
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.8.0_nompi/lib/python3.10/site-packages/zstash/extract.py", line 280, in extract_database
    failures = extractFiles(matches, keep_files, keep, cache, cur, args)
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.8.0_nompi/lib/python3.10/site-packages/zstash/extract.py", line 501, in extractFiles
    raise e
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.8.0_nompi/lib/python3.10/site-packages/zstash/extract.py", line 490, in extractFiles
    raise RuntimeError(
RuntimeError: zstash/000201.tar size does not match expected size.

[forsyth2]

In that case, I'm not sure. The tar file zstash/000201.tar already exists and is larger than what we would expect. I think that might be happening because there was a random hsi get failure, so zstash tried again and then found that the sizes don't match. Looking at comments in #218 (which introduced this behavior), it seems we were more concerned about the opposite case though (expected size is larger than what was actually returned).

I might suggest rerunning with --retries set to a high number. This is documented at https://e3sm-project.github.io/zstash/_build/html/main/usage.html#extract but the same behavior applies to check (I just made #268 to update the check documentation).

@golaz Do you have any thoughts on this?