Corrupted zstash data when extracting

[whannah1]

Question criteria

• I added a descriptive title here.
• I searched the zstash GitHub Discussions to find a similar question and didn't find it.
• I searched the zstash documentation.

What is the deadline?

N/A

Describe your question

Recently I was trying to extract some archived data that was added via "zstash update" and hit errors like this:

> zstash extract --hpss=E3SM/2024-RCEMIP2/E3SM.2024-RCEMIP2.FRCE-MW_305dT1p25 "data_final/E3SM_MW_305dT1p25_3D_cldfrac.nc"
For help, please see https://e3sm-project.github.io/zstash. Ask questions at https://github.com/E3SM-Project/zstash/discussions/categories/q-a.
INFO: Transferring file from HPSS: zstash/00000a.tar
INFO: zstash/00000a.tar exists. Checking expected size matches actual size.
INFO: Opening tar archive zstash/00000a.tar
INFO: Extracting data_final/E3SM_MW_305dT1p25_3D_cldfrac.nc
Traceback (most recent call last):
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.10.0_login/lib/python3.10/site-packages/zstash/extract.py", line 535, in extractFiles
    tarinfo: tarfile.TarInfo = tar.tarinfo.fromtarfile(tar)
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.10.0_login/lib/python3.10/tarfile.py", line 1293, in fromtarfile
    obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors)
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.10.0_login/lib/python3.10/tarfile.py", line 1237, in frombuf
    raise InvalidHeaderError("bad checksum")
tarfile.InvalidHeaderError: bad checksum
ERROR: Retrieving data_final/E3SM_MW_305dT1p25_3D_cldfrac.nc
ERROR: Encountered an error for files:
ERROR: data_final/E3SM_MW_305dT1p25_3D_cldfrac.nc in 00000a.tar
ERROR: The following tar archives had errors:
ERROR: 00000a.tar

This is for a simulation campaign with several runs, and more than half of the runs are exhibiting this issue, while a handful of them are working perfectly as expected.

I guess my question is whether there is anything I can do about this - or is the data just hopelessly lost?

This is not the first time I've had issues with corrupted data when trying to extract it, and the frequency of how often this happens is very troubling. Does this happen to other zstash users? Or is there possibly something about how I use zstash that makes my data more prone to corruption?

Are there are any possible answers you came across?

No response

What machine were you running on?

Perlmutter

Environment

zstash v1.4.2 (e3sm_unified_1.10.0)

Minimal Complete Verifiable Example (MCVE)

No response

Relevant log output

No response

Anything else we need to know?

No response

[forsyth2]

@golaz Have you encountered problems like this before?

[forsyth2]

@whannah1 Do you have any information about how the data was initially archived?

A couple things come to mind:

• Make sure the HPSS path is correct. Does --hpss=E3SM/2024-RCEMIP2/E3SM.2024-RCEMIP2.FRCE-MW_305dT1p25 need a prefix like /home/...?
• Was the log file archived during the creation step? Then, it may be possible to use the steps listed at [Bug]: Log file cannot currently be in directory being archived #335 to recover.

[whannah1]

What sort of info are you talking about?
In all of these cases I made the initial archive when the simulations were done and then did a zstash update command after the post processing was done.

HPSS path is definitely correct. I've tried manually downloading the file and trying to unzip it, but it throws an error no matter what I do.

I assume you mean the zstash log file? In that other post they just manually try to un-tar the data and it fails, but they still get the data. I need the data and can't get it. Not sure if I recover the log file, but either way I don't see how that would help.

I'm worried that this might be a systemic issue with the update process. I came across another thread about something similar, can't find it now though, but they mentioned that it had something to do with needing a bigger header, and without that step the data become un-recoverable.

[forsyth2]

What sort of info are you talking about?
E.g., what the exact zstash create, zstash update commands were.

I assume you mean the zstash log file? [...] Not sure if I recover the log file, but either way I don't see how that would help.

The issue in #335 was that sending output to a log file in the directory being archived ended up corrupting the tars. That is, if you those initial zstash create or zstash update commands included a > zstash_log.txt component, that could be responsible.

it had something to do with needing a bigger header, and without that step the data become un-recoverable.

@golaz I haven't seen anything about that, have you?

[whannah1]

Ah, I misunderstood that other issue then. It seems relevant then, but all RCEMIP cases were treated the same, so they should all have that problem. The data corruption appears to be somewhat "random" across these cases. The archiving and other special post processing tasks were controlled by a python script to ensure consistent treatment for all the data.

[tangq]

@whannah1 , did you zstash check after the simulations were archived? For important simulations, I always do this as it confirms that the tar files were not corrupted during transfer to the HPSS tape. (I learned it the hard way.)

[chengzhuzhang]

@whannah1 sorry to hear important data being corrupted.. have you tried to use the cpio utility (e.g cpio -ivd -H ustar < 000000.tar) to recover good files from tar? This was discovered and suggested by @golaz earlier when there were cases when tar xvf would fail to extract.

Any case, it is stressful to lose data like this! The zstash check step @tangq brought up should be helpful to avoid this in the future. @forsyth2 we do have a zstsah check session in the best practice doc, but it doesn't explain why this step is needed, and is very important to protect data. Could you maybe add note to emphasize this?

[whannah1]

I haven't tried that, but I will, thanks for the suggestion!

[forsyth2]

have you tried to use the cpio utility (e.g cpio -ivd -H ustar < 000000.tar) to recover good files from tar?

This is the process described in the earlier link: #335 (cpio -ivd -H ustar < 000000.tar)

it doesn't explain why this step is needed

I created E3SM-Project/E3SM-Project.github.io#11 to track that on the E3SM workflow docs. I suppose we could also update "After you run zstash create it’s highly recommended that you run zstash check" on the zstash docs to more explicitly state why it's recommended.

[chengzhuzhang]

@forsyth2 Thanks! At the same time, we should have an eye out for potential systematic issue with zstash upadate. I don't use this feature and not sure if we have good tests that covers this feature.

[forsyth2]

@chengzhuzhang There are automated tests of zstash update, but one problem with zstash testing in general is that it is often tricky to recreate real world use cases.

E.g., how do we produce so much testing data that it gets auto-split into multiple tars, how can we force a server disconnect mid-Globus transfer, how can we force a data corruption? Many tests have ended up being shell scripts rather than pytest functions (e.g., for #355).

For what it's worth, I'm wondering if it may be better to refactor the tests to organize around common workflows rather than zstash subcommands (see #367).