IndexError on extraction

[forsyth2]

@lee1046 brought up this error:

zstash/zstash/extract.py

Line 395 in 0d082ed

cur.execute(f"select size from tars where name is '{name_only}';")

[extract.py for the last zstash release]: It looks to me like 000014.tar exists yet, for some reason, zstash can't find it in the table of tars to look up an expected size.

I tried running the exact command (zstash extract -v --hpss=/home/j/jlee1046/ECP/E3SM_MAML.ne30pg2_r05_oECv3.F-MMF1-MAML.201223.RRTMGP.crm32.nrad16.01 "E3SM_MAML.ne30pg2_r05_oECv3.F-MMF1-MAML.201223.RRTMGP.crm32.nrad16.01.elm_*.h1.0001-*.nc" > zstash_extract.log), but it looks like I don't have access to /home/j/jlee1046/ECP/.

I suggested the following as a non-zstash workaround. @lee1046 says that it works but is impractically slow.

hsi -q "cd <HPSS directory>; get <tar file name>" # Get tar
tar -xvf <tar file name> # Untar

[forsyth2]

@lee1046 Do you know when the HPSS archive /home/j/jlee1046/ECP/E3SM_MAML.ne30pg2_r05_oECv3.F-MMF1-MAML.201223.RRTMGP.crm32.nrad16.01 was created? I see you're extracting from the archive using E3SM Unified 1.8.1 which includes zstash v1.3.0.

I discussed this issue with @chengzhuzhang and @golaz. We came up with four possible causes of the error.

1. The archive was created prior to the introduction of tars_table. That would be before 2021-07-16 (zstash v1.1.0 includes Add checksums for tar files #140, which merged on 2021-04-19).
2. The archive was created prior to the introduction of check_sizes_match. That would be before 2022-12-01 (zstash v1.3.0 includes hsi get retry #218, which merged on 2022-10-12).
3. zstash update was run on the archive, using a pre-zstash v1.1.0 version, meaning the tars_table would not be updated appropriately.
4. Somehow, an incorrect index.db is being used.

I tried running these cases myself, but running older versions of zstash is proving to be a challenge, presumably because 1) we recently changed the build process from conda to mamba, and 2) certain older versions of third party packages might be unavailable now.

Looking at the directory you ran the command from, /pscratch/sd/j/jlee1046/cori_scratch/e3sm_scratch/cori-knl/E3SM_MAML.ne30pg2_r05_oECv3.F-MMF1-MAML.201223.RRTMGP.crm32.nrad16.01/run, I notice the following:

$ ls
# Includes `zstash` directory
$ cd zstash
$ ls
# Includes `zstash` and `zstash_extract` directories.
$ ls zstash
ls: cannot open directory 'zstash': Permission denied
$ ls zstash_extract/
ls: cannot open directory 'zstash_extract/': Permission denied

It looks to me like you ran the extraction from the directory that the HPSS archive was created from. Is that the case? That can cause issues. See https://e3sm-project.github.io/zstash/_build/html/main/tutorial.html -- "we will extract into a new directory because zstash will not extract files that are already present"

[forsyth2]

Somehow, an incorrect index.db is being used.

This can happen if zstash extract is run for multiple archives into the same extraction directory. zstash won't load the new index.db because it sees one already in the directory.