Add BGC and Cryo v1 data to centralized archive

[forsyth2]

Add BGC and Cryo v1 data to centralized archive, building on #59, and #60

[forsyth2]

What paths exists currently?

HPSS Paths are on the e3sm_data_docs simulation table

• /home/projects/e3sm/www/WaterCycle/E3SMv1/LR/
• /home/projects/e3sm/www/WaterCycle/E3SMv1/HR/

hsi
ls /home/projects/e3sm/www/BGC/E3SMv2_1/ # No other subdirectories of BGC
# CBGCv2.r05.chrysalis.I20TRGSWCNP/  v2.LR.BGC-LNDATM.CONTRL/           v2.LR.BGC-LNDATM.SSP370/ 
ls /home/projects/e3sm/www/Cryosphere/v1_supplemental/ # No other subdirectories of Cryosphere
# 20190923.GMPAS-IAF.T62_oEC60to30v3wLI.cori-knl/                                        20211020.GMPAS-DIB-IAF-ISMF.T62_oEC60to30v3wLI.CGM.DIBbugfix-EastWedd0.5758myr.anvil/  
# 20191003.GMPAS-IAF-ISMF.T62_oEC60to30v3wLI.cori-knl/                                   20211020.GMPAS-DIB-IAF-ISMF.T62_oEC60to30v3wLI.VGM.DIBbugfix-EastWedd0.5758myr.anvil/  
# 20210730.GMPAS-DIB-IAF-ISMF.T62_oEC60to30v3wLI.DIBbugfix.anvil/                        20211025.GMPAS-DIB-IAF-ISMF.T62_oEC60to30v3wLI.CGM.DIBbugfix-EastWedd4myr.anvil/       
# 20210901.GMPAS-DIB-IAF-ISMF.T62_oEC60to30v3wLI.VGM.DIBbugfix.anvil/                    20211025.GMPAS-DIB-IAF-ISMF.T62_oEC60to30v3wLI.CGM.DIBbugfix-EastWedd8myr.anvil/       
# 20211006.GMPAS-DIB-IAF-ISMF.T62_oEC60to30v3wLI.VGM.DIBbugfix-EastWedd16myr.anvil/      20211026.GMPAS-DIB-IAF-ISMF.T62_oEC60to30v3wLI.CGM.DIBbugfix-EastWedd16myr.anvil/      
# 20211008.GMPAS-DIB-IAF-ISMF.T62_oEC60to30v3wLI.VGM.DIBbugfix-EastWedd4myr.anvil/       20211026.GMPAS-DIB-IAF-ISMF.T62_oEC60to30v3wLI.CGM.DIBbugfix-EastWedd2myr.anvil/       
# 20211008.GMPAS-DIB-IAF-ISMF.T62_oEC60to30v3wLI.VGM.DIBbugfix-EastWedd8myr.anvil/       20211104.GMPAS-DIB-IAF-ISMF.T62_oEC60to30v3wLI.CGM.DIBbugfix-EastWedd1myr.anvil/       
# 20211017.GMPAS-DIB-IAF-ISMF.T62_oEC60to30v3wLI.VGM.DIBbugfix-EastWedd2myr.anvil/ 

Paths from Confluence:

• v1-BGC: source location + data publication archives
• v1 CRYO (low-resolution): source location only

Initial work on making the data accessible from a central location

showquota --hpss
# +---------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
# |                           File system | Space used | Space quota | Space used (%) | Inode used | Inode quota | Inode used (%) |
# +---------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
# |                                  home |   13.64GiB |    40.00GiB |          34.1% |    329.81K |       1.00M |          33.0% |
# |                              pscratch |  125.92GiB |    20.00TiB |           0.6% |     117.00 |      10.00M |           0.0% |
# | forsyth usage on HPSS charged to e3sm | 1019.11TiB |   323.72TiB |         314.8% |          - |           - |              - |
# +---------------------------------------+------------+-------------+----------------+------------+-------------+----------------+

# Check if I can make the necessary subdirs ###################################
hsi
cd /home/projects/e3sm/www/BGC/
mkdir E3SMv1
# *** hpss_Mkdir: Access denied [-13: HPSS_EACCES] 
#     /home/projects/e3sm/www/BGC/E3SMv1
# mkdir: /home/projects/e3sm/www/BGC/E3SMv1
cd /home/projects/e3sm/www/Cryosphere/
# *** hpss_Mkdir: Access denied [-13: HPSS_EACCES] 
#     /home/projects/e3sm/www/Cryosphere/E3SMv1
# mkdir: /home/projects/e3sm/www/Cryosphere/E3SMv1
cd /home/projects/e3sm/www
ls -l
# drwxr-xr-x    3 feng809   e3sm             512 Mar 14 09:35 BGC
# drwxr-xr-x    3 dcomeau   e3sm             512 Jun 20  2024 Cryosphere
# drwxrwxr-x    5 golaz     e3sm             512 Jun  2 14:57 WaterCycle
cd /home/projects/e3sm/www/WaterCycle/
mkdir test_dir
# Works! Looks like BGC and Cryosphere need Group write access
rmdir test_dir
mkdir temp_bgc_v1_subdir
# Check if I can make hard links ##############################################
# ln -s => symbolic link (file path points to some other file path that itself points to an inode)
# ln => hard link (file path points to the same inode as another file path)
# ln <src> <target>
ln /home/projects/e3sm/www/publication-archives/pub_archive_E3SM_1_1_piControl/ /home/projects/e3sm/www/WaterCycle/temp_bgc_v1_subdir
ls temp_bgc_v1_subdir:
# Still looks empty
cd temp_bgc_v1_subdir
ls
# Still nothing
exit

showquota --hpss # Appears to show no changes
# +---------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
# |                           File system | Space used | Space quota | Space used (%) | Inode used | Inode quota | Inode used (%) |
# +---------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
# |                                  home |   13.64GiB |    40.00GiB |          34.1% |    329.81K |       1.00M |          33.0% |
# |                              pscratch |  125.92GiB |    20.00TiB |           0.6% |     117.00 |      10.00M |           0.0% |
# | forsyth usage on HPSS charged to e3sm | 1019.11TiB |   323.72TiB |         314.8% |          - |           - |              - |
# +---------------------------------------+------------+-------------+----------------+------------+-------------+----------------+

hsi
cd /home/projects/e3sm/www/WaterCycle/temp_bgc_v1_subdir
# Let's try a symlink
ln -s /home/projects/e3sm/www/publication-archives/pub_archive_E3SM_1_1_piControl/ 20181217.CNTL_CNPCTC1850_OIBGC.ne30_oECv3.edison
ls
# /home/projects/e3sm/www/WaterCycle/temp_bgc_v1_subdir:
# 20181217.CNTL_CNPCTC1850_OIBGC.ne30_oECv3.edison@
#
# NOTICE: the `@` indicates its a symlink
exit

showquota --hpss # Appears to show no changes
# +---------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
# |                           File system | Space used | Space quota | Space used (%) | Inode used | Inode quota | Inode used (%) |
# +---------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
# |                                  home |   13.64GiB |    40.00GiB |          34.1% |    329.81K |       1.00M |          33.0% |
# |                              pscratch |  125.92GiB |    20.00TiB |           0.6% |     117.00 |      10.00M |           0.0% |
# | forsyth usage on HPSS charged to e3sm | 1019.11TiB |   323.72TiB |         314.8% |          - |           - |              - |
# +---------------------------------------+------------+-------------+----------------+------------+-------------+----------------+

hsi
cd /home/projects/e3sm/www/WaterCycle/temp_bgc_v1_subdir
# Let's try a hard link again
ln /home/projects/e3sm/www/publication-archives/pub_archive_E3SM_1_1_piControl/ 20181217.CNTL_CNPCTC1850_OIBGC.ne30_oECv3.edison_hard_link
cd 20181217.CNTL_CNPCTC1850_OIBGC.ne30_oECv3.edison_hard_link
# *** cd: No such file or directory [-2: HPSS_ENOENT] 
#     20181217.CNTL_CNPCTC1850_OIBGC.ne30_oECv3.edison_hard_link

Here's what Claude has to say about hard links on NERSC

Based on my research of the NERSC documentation and HPSS system information, here's what I found regarding hard links on NERSC HPSS:

Hard links are not supported on HPSS at NERSC. While you've successfully created symlinks, hard links don't work because HPSS (High Performance Storage System) is fundamentally different from a traditional POSIX filesystem.

Here's why hard links don't work on HPSS:

1. Tape-based hierarchical storage: HPSS is a hierarchical storage management system designed for long-term archival storage on tape, not a traditional filesystem. It uses a different storage model where files are managed across multiple storage tiers (disk cache and tape).

2. Limited POSIX compliance: While the hsi command shell supports many standard Linux commands like ls, cd, rm, chmod, mkdir, rmdir, it doesn't support all POSIX filesystem features. The documentation shows that most of the standard Linux commands work in hsi, but hard links are notably absent from the supported operations.

3. Archive-oriented design: HPSS is intended for long-term storage of data that is not frequently accessed and requires storing files in appropriately sized chunks between 100 GB and 2 TB. This design philosophy is incompatible with hard links, which require shared inode structures typical of regular filesystems.

What works instead:

• Symlinks: As you've discovered, these work fine
• Copying files: Using hsi put and hsi get commands
• Bundling with htar: htar is designed for storing groups of files and preserves directory structure

The HPSS system is optimized for archival storage rather than providing full POSIX filesystem semantics, which is why hard links aren't supported while symlinks are.

@chengzhuzhang So, 2 problems before I can really proceed:

1. /home/projects/e3sm/www/BGC/ and /home/projects/e3sm/www/Cryosphere/ need to expand write-access to group e3sm so I can add a E3SMv1 subdirectory in each.
2. It appears that while NERSC supports symlinks (as we used in Add ESGF links and more simulations to v1 data #60), it does not support hard links (which I suppose makes sense given the inode columns of showquota are marked as non-applicable). That's really a problem if that data ever gets accidentally deleted, as @golaz has noted.

[forsyth2]

That's really a problem if that data ever gets accidentally deleted

Less of a problem if we're symlinking to the publication archives, since I imagine those are staying around for a while. But for the cyro simulations, for example, we're relying on users to not delete data.

[forsyth2]

More from Claude re: hard links not working

HPSS actually does have something analogous to inodes, but it's designed very differently from traditional POSIX filesystems, which is exactly why hard links don't work the same way.

Here's how HPSS actually works:

HPSS Metadata Architecture

HPSS uses IBM Db2 relational databases to manage metadata describing files, with separate core servers that use metadata to coordinate data flow. The system has what's called bitfile identifiers rather than traditional inodes.

Key Architectural Differences

1. Relational Database vs. Filesystem Metadata: Metadata describing the files in HPSS are managed by Db2 relational databases with tables and index files, transaction log files, archived transaction log files, and database backup files.

2. Hierarchical Storage Model: HPSS implements hierarchical storage management (HSM) with tiered storage from solid-state disk to magnetic disk to tape. Files migrate between tiers, and their physical location changes over time.

3. Bitfiles and Object Tracking: Instead of inodes pointing to specific disk blocks, HPSS uses a more complex system where:

   • Each file gets a unique identifier in the database
   • The system tracks which tape cartridge(s) contain the data
   • The disk cache retains many days' worth of new data, and the tape subsystem provides long-term scalable data storage

Why Hard Links Don't Work

The problem isn't that there's no equivalent to inodes - it's that HPSS's design philosophy is fundamentally different:

• Traditional filesystem: Multiple directory entries → same inode → same disk blocks
• HPSS: Database entries → bitfile records → tape locations that can migrate

Since files in HPSS can migrate between storage tiers (disk cache to tape) and even between different tape cartridges during data management operations, the concept of multiple directory entries pointing to the same "piece of tape" becomes problematic. The system is designed for data mobility and long-term archival, not for the fixed relationships that hard links require.

So while there IS metadata tracking each piece of data (analogous to inodes), the architecture prioritizes data management and migration over POSIX filesystem semantics like hard links.

[forsyth2]

/home/projects/e3sm/www/BGC/ and /home/projects/e3sm/www/Cryosphere/ need to expand write-access to group e3sm so I can add a E3SMv1 subdirectory in each.

@BunnyVon & @darincomeau it looks like you're the respective owners of those directories on NERSC HPSS. Can you expand write-access to the group on those? If not, I can pass along instructions on how to add the v1 data symlinks. Thanks!

[darincomeau]

Done

drwxrwxr-x    3 dcomeau   e3sm             512 Aug 22 14:31 Cryosphere

[forsyth2]

Thanks, looks like I can create a directory in it now.

[BunnyVon]

done.
drwxrwxr-x 3 feng809 e3sm 512 Sep 15 14:49 BGC