Merge pull request #202 from ESGF/pid_url_fix

Pid url fix

Author: sashakames

docs/index.rst

@@ -1,5 +1,5 @@
-The esgcet package for ESGF Publication
-=======================================
+The esgcet package for ESGF Publication - Beta release 13
+=========================================================
 
 Esgcet is a package of publisher commands for publishing to the `ESGF
 <https://esgf-node.llnl.gov/projects/esgf-llnl/>`_ search database.
@@ -8,10 +8,12 @@ Esgcet is a package of publisher commands for publishing to the `ESGF
 TL;DR
 -----
 
-if you have conda you can install the publisher wih the following into a fresh environment:
+if you have conda you can install the publisher wih the following into a fresh environment, and update to the latest version:
 ::
      conda create -n esgf-pub -c conda-forge -c esgf-forge esgcet
      conda activate esgf-pub
+     pip install esgcet # upgrade
+     esgpublish --version #  Ensure you have upgraded to v5.1.0-b13
      esgpublish # will print the usage information.
 
 You may also look at the inital ``~/.esg/esg.ini`` and fill in the missing information based on the provided examples.

docs/install.rst

@@ -1,15 +1,6 @@
 Installation
 ============
 
-You can install the ``esg-publisher`` (aka the ``esgcet`` Python package) one of two ways: conda, or git.
-
-For the **basic** installation: to install esgcet using conda, use the following to create a new environment populated with the publisher software and required packages: ::
-
-    conda create -n esgf-pub -c esgf-forge -c conda-forge esgcet
-    conda activate esgf-pub
-
-Other installation methods require several pre-requisites, as described below.
-
 Conda & Required Packages
 -------------------------
 
@@ -18,21 +9,20 @@ We recommend creating a conda env before installing ``esgcet`` ::
     conda create -n esgf-pub -c conda-forge -c esgf-forge pip libnetcdf cmor autocurator esgconfigparser
     conda activate esgf-pub
 
-NOTE: if you install esgcet using conda below, the cmor package (different from tables) should be installed at the time you install esgcet automatically, and having cmor in your env may cause conflicts (but not always).
 
 You will also need to install ``esgfpid`` using pip::
 
     pip install esgfpid
 
-NOTE: you will need a functioning version of ``autocurator`` in order to run the publisher, in addition to downloading the CMOR tables. See those pages for more info.
+NOTE: you will need a functioning version of ``autocurator`` in order to run the publisher, in addition to downloading the CMOR tables. See those pages for more info.  The ``autocurator`` package in the ``esgf-forge`` conda channel provides a working albeit not the most recent version of this module.
 
-Conda Install
--------------
+Pip Install
+-----------
 
 Use the following command to install ``esgcet`` into a previously created conda environment: ::
 
-    conda install -c esgf-forge -c conda-forge esgcet
-
+    conda activate esgf-pub
+    pip install esgcet
 
 
 Installing esgcet via git

docs/whatsnew.rst

@@ -1,6 +1,13 @@
 Release Notes
 =============
 
+b5.1.0-b13
+----------
+
+* **BUGFIX**: corrected file URL format for PID/Handle publishing (previously published URLs via v5.* were malformed).
+* CMIP6 Cloned project support 
+* **NOTE**:  this version is unavailable on Conda (``esgf-forge`` channel), please use ``pip install esgcet`` and confirm the upgrade with ``esgpublish --version``.
+
 b5.1.0-b11
 ----------
 

pkg/esgcet/__init__.py

@@ -1 +1 @@
-__version__ = "5.1.0b12"
+__version__ = "5.1.0b13"

pkg/esgcet/mk_dataset.py

@@ -248,6 +248,7 @@ def get_file(self, mapdata, fn_trid):
                 exit(1)
 
         ret["url"] = self.gen_urls(self.data_roots[proj_root], rel_path)
+        ret["publish_path"] = f"{self.data_roots[proj_root]}/{rel_path}"
         if "number_of_files" in ret:
             ret.pop("number_of_files")
         else:

pkg/esgcet/pid_cite_pub.py

@@ -3,16 +3,14 @@
 import traceback
 import esgcet.logger as logger
 
+
 log = logger.Logger()
 
 silent = False
 verbose = False
 pid_creds = {}
 import traceback
 
-def get_url(arr):
-    return arr[0].split('|')[0]
-
 class ESGPubPidCite(object):
     """ for PID services wraps calls to obtain a PID, add to records and generate citiation metadata """
 
@@ -122,7 +120,7 @@ def pid_flow_code(self):
                                         file_handle=file_rec['tracking_id'],
                                         checksum=file_rec['checksum'],
                                         file_size=file_rec['size'],
-                                        publish_path=get_url(file_rec['url']),
+                                        publish_path=file_rec['publish_path'],
                                         checksum_type=file_rec['checksum_type'],
                                         file_version=file_rec['version'] )
             else:
@@ -131,7 +129,7 @@ def pid_flow_code(self):
                                    file_handle=file_rec['tracking_id'],
                                    checksum=file_rec['checksum'],
                                    file_size=file_rec['size'],
-                                   publish_path=get_url(file_rec['url']),
+                                   publish_path=file_rec['publish_path'],
                                    checksum_type=file_rec['checksum_type'],
                                    file_version=file_rec['version'])
             if pid_wizard: