diff --git a/docs/source/index.rst b/docs/source/index.rst index cc38f2a..51de892 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -18,6 +18,7 @@ simulations. v1/index v2/index v2.1/index + v3/index SCREAMv0/index SCREAMv1/index diff --git a/docs/source/v2/WaterCycle/simulation_data/index.rst b/docs/source/v2/WaterCycle/simulation_data/index.rst index 8dbc908..8f1f23a 100644 --- a/docs/source/v2/WaterCycle/simulation_data/index.rst +++ b/docs/source/v2/WaterCycle/simulation_data/index.rst @@ -42,6 +42,12 @@ For more information, refer to `zstash usage `_. Note that this will be slow and inefficient since you'll have to download the tar files. +**Tip for users without NERSC access**: Before downloading large tar files, you can first download the ``index.db`` file and use sqlite to check the archive contents: :: + + sqlite3 index.db "SELECT tar,name,size from files;" > archive_contents.txt + +This will help you identify which specific tar files contain the data you need before downloading. + **v2.LR** simulations data has been archived on NERSC HPSS under: :: /home/projects/e3sm/www/WaterCycle/E3SMv2/LR diff --git a/docs/source/v3/CoupledSystem/index.rst b/docs/source/v3/CoupledSystem/index.rst new file mode 100644 index 0000000..5bfa552 --- /dev/null +++ b/docs/source/v3/CoupledSystem/index.rst @@ -0,0 +1,21 @@ +E3SMv3 +================ + +The simulation campaign for E3SMv3 was performed initially with the +**v3.LR** (lower resolution) model configuration. + +If you use data from this simulation campaign, please cite the relevant overview +manuscripts. + +* For a general E3SMv3 overview and v3.LR simulations: + + * Xie et al. (2025). The Energy Exascale Earth System Model Version 3. Part I: Overview of the Atmospheric Component. Accepted by JAMES. DOI to follow. + * Golaz et al. (2025). The Energy Exascale Earth System Model Version 3. Part II: Overview of the Coupled system. Submitted to JAMES. + +For information on how to access and use the simulation data, see :doc:`here <../../v2/WaterCycle/simulation_data/index>`. + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + simulation_data/index diff --git a/docs/source/v3/CoupledSystem/simulation_data/index.rst b/docs/source/v3/CoupledSystem/simulation_data/index.rst new file mode 100644 index 0000000..b34464e --- /dev/null +++ b/docs/source/v3/CoupledSystem/simulation_data/index.rst @@ -0,0 +1,7 @@ +Simulation Data +=============== + +.. toctree:: + :maxdepth: 2 + + simulation_table \ No newline at end of file diff --git a/docs/source/v3/CoupledSystem/simulation_data/simulation_table.rst b/docs/source/v3/CoupledSystem/simulation_data/simulation_table.rst new file mode 100644 index 0000000..94743c3 --- /dev/null +++ b/docs/source/v3/CoupledSystem/simulation_data/simulation_table.rst @@ -0,0 +1,77 @@ +********************************** +v3 CoupledSystem simulation table +********************************** + ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| Simulation | Data Size (TB) | ESGF Links | HPSS Path | HPSS URL | ++===================================================================+=================+==============================================================================================================================================================================================================================+==========================================================================================================================+==============================================================================================================================================+ +| **LR > DECK** | | | | | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.piControl | 69 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piControl | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.abrupt-4xCO2_0101_bcdt15m | 15 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.abrupt-4xCO2_0101_bcdt15m | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.1pctCO2_0101_bcdt15m | 16 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.1pctCO2_0101_bcdt15m | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| **LR > Historical** | | | | | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.historical_0051 | 19 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.historical_0051 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.historical_0101 | 19 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.historical_0101 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.historical_0151 | 19 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.historical_0151 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.historical_0201 | 19 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.historical_0201 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.historical_0251 | 19 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.historical_0251 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| **LR > Single-forcing (DAMIP-like)** | | | | | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.hist-GHG_0101 | 20 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-GHG_0101 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.hist-GHG_0151 | 20 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-GHG_0151 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.hist-GHG_0201 | 20 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-GHG_0201 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.hist-aer_0101 | 19 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-aer_0101 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.hist-aer_0151 | 19 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-aer_0151 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.hist-aer_0201 | 19 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-aer_0201 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.hist-xGHG-xaer_0101 | 19 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-xGHG-xaer_0101 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.hist-xGHG-xaer_0151 | 20 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-xGHG-xaer_0151 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.hist-xGHG-xaer_0201 | 19 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-xGHG-xaer_0201 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| **LR > AMIP** | | | | | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.amip_0101 | 8 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.amip_0101 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.amip_0151 | 6 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.amip_0151 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.amip_0201 | 6 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.amip_0201 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| **LR > RFMIP** | | | | | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.piClim-control-iceini | 3 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-control-iceini | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.piClim-histall/v3.LR.piClim-histall_0101 | 9 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histall/v3.LR.piClim-histall_0101 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.piClim-histall/v3.LR.piClim-histall_0151 | 9 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histall/v3.LR.piClim-histall_0151 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.piClim-histall/v3.LR.piClim-histall_0201 | 9 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histall/v3.LR.piClim-histall_0201 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0101 | 9 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0101 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0151 | 9 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0151 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0201 | 9 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0201 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.piClim-histaer/v3.LR.piClim-histaer_0101 | 8 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histaer/v3.LR.piClim-histaer_0101 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.piClim-histaer/v3.LR.piClim-histaer_0151 | 8 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histaer/v3.LR.piClim-histaer_0151 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ +| v3.LR.piClim-histaer/v3.LR.piClim-histaer_0201 | 9 | | /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histaer/v3.LR.piClim-histaer_0201 | `HPSS URL `_ | ++-------------------------------------------------------------------+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------+ diff --git a/docs/source/v3/index.rst b/docs/source/v3/index.rst new file mode 100644 index 0000000..602fb6f --- /dev/null +++ b/docs/source/v3/index.rst @@ -0,0 +1,8 @@ +v3.LR +========================================== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + CoupledSystem/index diff --git a/machine_readable_data/v3_CoupledSystem_simulations.csv b/machine_readable_data/v3_CoupledSystem_simulations.csv new file mode 100644 index 0000000..68bd0ae --- /dev/null +++ b/machine_readable_data/v3_CoupledSystem_simulations.csv @@ -0,0 +1,31 @@ +Simulation,Data Size (TB),ESGF Links,HPSS Path,HPSS URL +v3.LR.piControl,69,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piControl,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piControl +v3.LR.abrupt-4xCO2_0101_bcdt15m,15,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.abrupt-4xCO2_0101_bcdt15m,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.abrupt-4xCO2_0101_bcdt15m +v3.LR.1pctCO2_0101_bcdt15m,16,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.1pctCO2_0101_bcdt15m,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.1pctCO2_0101_bcdt15m +v3.LR.historical_0051,19,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.historical_0051,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.historical_0051 +v3.LR.historical_0101,19,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.historical_0101,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.historical_0101 +v3.LR.historical_0151,19,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.historical_0151,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.historical_0151 +v3.LR.historical_0201,19,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.historical_0201,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.historical_0201 +v3.LR.historical_0251,19,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.historical_0251,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.historical_0251 +v3.LR.hist-GHG_0101,20,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-GHG_0101,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-GHG_0101 +v3.LR.hist-GHG_0151,20,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-GHG_0151,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-GHG_0151 +v3.LR.hist-GHG_0201,20,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-GHG_0201,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-GHG_0201 +v3.LR.hist-aer_0101,19,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-aer_0101,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-aer_0101 +v3.LR.hist-aer_0151,19,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-aer_0151,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-aer_0151 +v3.LR.hist-aer_0201,19,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-aer_0201,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-aer_0201 +v3.LR.hist-xGHG-xaer_0101,19,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-xGHG-xaer_0101,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-xGHG-xaer_0101 +v3.LR.hist-xGHG-xaer_0151,20,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-xGHG-xaer_0151,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-xGHG-xaer_0151 +v3.LR.hist-xGHG-xaer_0201,19,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-xGHG-xaer_0201,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.hist-xGHG-xaer_0201 +v3.LR.amip_0101,8,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.amip_0101,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.amip_0101 +v3.LR.amip_0151,6,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.amip_0151,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.amip_0151 +v3.LR.amip_0201,6,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.amip_0201,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.amip_0201 +v3.LR.piClim-control-iceini,3,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-control-iceini,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-control-iceini +v3.LR.piClim-histall/v3.LR.piClim-histall_0101,9,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histall/v3.LR.piClim-histall_0101,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histall/v3.LR.piClim-histall_0101 +v3.LR.piClim-histall/v3.LR.piClim-histall_0151,9,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histall/v3.LR.piClim-histall_0151,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histall/v3.LR.piClim-histall_0151 +v3.LR.piClim-histall/v3.LR.piClim-histall_0201,9,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histall/v3.LR.piClim-histall_0201,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histall/v3.LR.piClim-histall_0201 +v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0101,9,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0101,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0101 +v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0151,9,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0151,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0151 +v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0201,9,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0201,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0201 +v3.LR.piClim-histaer/v3.LR.piClim-histaer_0101,8,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histaer/v3.LR.piClim-histaer_0101,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histaer/v3.LR.piClim-histaer_0101 +v3.LR.piClim-histaer/v3.LR.piClim-histaer_0151,8,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histaer/v3.LR.piClim-histaer_0151,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histaer/v3.LR.piClim-histaer_0151 +v3.LR.piClim-histaer/v3.LR.piClim-histaer_0201,9,,/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histaer/v3.LR.piClim-histaer_0201,https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piClim-histaer/v3.LR.piClim-histaer_0201 diff --git a/utils/generate_tables.py b/utils/generate_tables.py index 1f21f3b..199a024 100644 --- a/utils/generate_tables.py +++ b/utils/generate_tables.py @@ -173,6 +173,9 @@ def __init__(self, simulation_dict): self.esgf = get_esgf(self.model_version, self.resolution, self.simulation_name, self.experiment, self.ensemble_num, self.link_type, self.node) + # Generate web interface URL from HPSS path + self.web_interface = self.get_web_interface_url() + self.run_script_original = get_run_script_original(self.model_version, self.simulation_name) self.run_script_reproduction = get_run_script_reproduction(self.model_version, self.simulation_name) @@ -183,9 +186,20 @@ def __init__(self, simulation_dict): if not self.run_script_original: self.run_script_original = "N/A" + def get_web_interface_url(self) -> str: + """Generate web interface URL from HPSS path""" + if self.hpss and self.data_size: + # Convert HPSS path to web interface URL + # /home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piControl -> https://portal.nersc.gov/archive/home/projects/e3sm/www/CoupledSystem/E3SMv3/LR/v3.LR.piControl + hpss_clean = self.hpss.replace("(symlink) ", "") # Remove symlink prefix if present + # Use the full path - each simulation gets its own distinct URL + web_url = f"https://portal.nersc.gov/archive{hpss_clean}" + return f"`HPSS URL <{web_url}>`_" + return "" + def get_row(self, output_file, minimal_content: bool = False) -> List[str]: if "simulation" in output_file: - row = [self.simulation_name, self.data_size, self.esgf, self.hpss] + row = [self.simulation_name, self.data_size, self.esgf, self.hpss, self.web_interface] if minimal_content: match_object: re.Match = re.match("`.*<(.*)>`_", self.esgf) if match_object: @@ -195,6 +209,10 @@ def get_row(self, output_file, minimal_content: bool = False) -> List[str]: # Since we don't want that in the csv output, # which a computer reads. row[3] = row[3].replace("(symlink) ", "") + # Extract web interface URL for CSV + web_match: re.Match = re.match("`.*<(.*)>`_", self.web_interface) + if web_match: + row[4] = web_match.group(1) # Extract URL from the web interface link return row elif "reproduction" in output_file: return [self.simulation_name, self.machine, self.checksum, self.run_script_reproduction, self.run_script_original] @@ -261,6 +279,8 @@ def read_simulations(csv_file): simulation_dict["link_type"] = "both" elif simulation_dict["cmip_only"] == "cmip_only": simulation_dict["link_type"] = "cmip" + elif simulation_dict["cmip_only"] == "none": + simulation_dict["link_type"] = "none" else: raise ValueError(f"Invalid cmip_only={simulation_dict['cmip_only']}") simulation_dicts.append(simulation_dict) @@ -324,6 +344,10 @@ def pad_cells_row_dividers(marker: str, cell_paddings: List[int]) -> str: return string def generate_table(page_type: str, resolutions: OrderedDict[str, Category], header_cells: List[str], output_file: str, cell_paddings: List[int]): + + # Create output directory if it doesn't exist + os.makedirs(os.path.dirname(output_file), exist_ok=True) + with open(output_file, "w") as file_write: # Page Title file_write.write("**********************************\n") @@ -349,7 +373,7 @@ def generate_table(page_type: str, resolutions: OrderedDict[str, Category], head def construct_pages(csv_file: str, model_version: str, group_name: str, include_reproduction_scripts: bool = False): versions: OrderedDict[str, ModelVersion] = read_simulations(csv_file) resolutions: OrderedDict[str, Category] = versions[model_version].groups[group_name].resolutions - header_cells: List[str] = ["Simulation", "Data Size (TB)", "ESGF Links", "HPSS Path"] + header_cells: List[str] = ["Simulation", "Data Size (TB)", "ESGF Links", "HPSS Path", "HPSS URL"] construct_output_csv(resolutions, header_cells, f"../machine_readable_data/{model_version}_{group_name}_simulations.csv") print(f"csv of the simulations will be available at https://github.com/E3SM-Project/e3sm_data_docs/blob/main/machine_readable_data/{model_version}_{group_name}_simulations.csv") # TODO: add proper subdirs and index.rst files in docs/ @@ -358,7 +382,7 @@ def construct_pages(csv_file: str, model_version: str, group_name: str, include_ resolutions, header_cells, f"../docs/source/{model_version}/{group_name}/simulation_data/simulation_table.rst", - [85, 15, 400, 140] + [65, 15, 220, 120, 140] ) if include_reproduction_scripts: header_cells_reproduction: List[str] = ["Simulation", "Machine", "10 day checksum", "Reproduction Script", "Original Script (requires significant changes to run!!)",] @@ -383,4 +407,5 @@ def construct_pages(csv_file: str, model_version: str, group_name: str, include_ # Sources for v1 data # https://acme-climate.atlassian.net/wiki/spaces/ED/pages/4495441922/V1+Simulation+backfill+WIP # https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/1271169273/v1+High+Res+Coupled+Run+Output+HPSS+Archive - construct_pages("input/simulations_v1_water_cycle.csv", "v1", "WaterCycle") + # construct_pages("input/simulations_v1_water_cycle.csv", "v1", "WaterCycle") + construct_pages("input/simulations_v3_LR_coupled.csv", "v3", "CoupledSystem") diff --git a/utils/input/simulations_v3_LR_coupled.csv b/utils/input/simulations_v3_LR_coupled.csv new file mode 100644 index 0000000..21be674 --- /dev/null +++ b/utils/input/simulations_v3_LR_coupled.csv @@ -0,0 +1,31 @@ +model_version, group, resolution, category, simulation_name, machine, checksum, experiment, ensemble_num, cmip_only, +v3, CoupledSystem, LR, DECK, v3.LR.piControl, chrysalis, , piControl, 1, none, +v3, CoupledSystem, LR, DECK, v3.LR.abrupt-4xCO2_0101_bcdt15m, chrysalis, , abrupt-4xCO2, 1, none, +v3, CoupledSystem, LR, DECK, v3.LR.1pctCO2_0101_bcdt15m, chrysalis, , 1pctCO2, 1, none, +v3, CoupledSystem, LR, Historical, v3.LR.historical_0051, chrysalis, , historical, 1, none, +v3, CoupledSystem, LR, Historical, v3.LR.historical_0101, chrysalis, , historical, 2, none, +v3, CoupledSystem, LR, Historical, v3.LR.historical_0151, chrysalis, , historical, 3, none, +v3, CoupledSystem, LR, Historical, v3.LR.historical_0201, chrysalis, , historical, 4, none, +v3, CoupledSystem, LR, Historical, v3.LR.historical_0251, chrysalis, , historical, 5, none, +v3, CoupledSystem, LR, Single-forcing (DAMIP-like), v3.LR.hist-GHG_0101, chrysalis, , hist-GHG, 1, none, +v3, CoupledSystem, LR, Single-forcing (DAMIP-like), v3.LR.hist-GHG_0151, chrysalis, , hist-GHG, 2, none, +v3, CoupledSystem, LR, Single-forcing (DAMIP-like), v3.LR.hist-GHG_0201, chrysalis, , hist-GHG, 3, none, +v3, CoupledSystem, LR, Single-forcing (DAMIP-like), v3.LR.hist-aer_0101, chrysalis, , hist-aer, 1, none, +v3, CoupledSystem, LR, Single-forcing (DAMIP-like), v3.LR.hist-aer_0151, chrysalis, , hist-aer, 2, none, +v3, CoupledSystem, LR, Single-forcing (DAMIP-like), v3.LR.hist-aer_0201, chrysalis, , hist-aer, 3, none, +v3, CoupledSystem, LR, Single-forcing (DAMIP-like), v3.LR.hist-xGHG-xaer_0101, chrysalis, , hist-nat, 1, none, +v3, CoupledSystem, LR, Single-forcing (DAMIP-like), v3.LR.hist-xGHG-xaer_0151, chrysalis, , hist-nat, 2, none, +v3, CoupledSystem, LR, Single-forcing (DAMIP-like), v3.LR.hist-xGHG-xaer_0201, chrysalis, , hist-nat, 3, none, +v3, CoupledSystem, LR, AMIP, v3.LR.amip_0101, chrysalis, , amip, 1, none, +v3, CoupledSystem, LR, AMIP, v3.LR.amip_0151, chrysalis, , amip, 2, none, +v3, CoupledSystem, LR, AMIP, v3.LR.amip_0201, chrysalis, , amip, 3, none, +v3, CoupledSystem, LR, RFMIP, v3.LR.piClim-control-iceini, chrysalis, , piClim-control, 1, none, +v3, CoupledSystem, LR, RFMIP, v3.LR.piClim-histall/v3.LR.piClim-histall_0101, chrysalis, , piClim-histall, 1, none, +v3, CoupledSystem, LR, RFMIP, v3.LR.piClim-histall/v3.LR.piClim-histall_0151, chrysalis, , piClim-histall, 2, none, +v3, CoupledSystem, LR, RFMIP, v3.LR.piClim-histall/v3.LR.piClim-histall_0201, chrysalis, , piClim-histall, 3, none, +v3, CoupledSystem, LR, RFMIP, v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0101, chrysalis, , piClim-histGHG, 1, none, +v3, CoupledSystem, LR, RFMIP, v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0151, chrysalis, , piClim-histGHG, 2, none, +v3, CoupledSystem, LR, RFMIP, v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0201, chrysalis, , piClim-histGHG, 3, none, +v3, CoupledSystem, LR, RFMIP, v3.LR.piClim-histaer/v3.LR.piClim-histaer_0101, chrysalis, , piClim-histaer, 1, none, +v3, CoupledSystem, LR, RFMIP, v3.LR.piClim-histaer/v3.LR.piClim-histaer_0151, chrysalis, , piClim-histaer, 2, none, +v3, CoupledSystem, LR, RFMIP, v3.LR.piClim-histaer/v3.LR.piClim-histaer_0201, chrysalis, , piClim-histaer, 3, none,