Replicas' retracted status not synced to their origins that were published on non-DOE nodes

[minxu74]

Some CMIP6 replicas in the index show their retracted as False, but their origins that were published on non-DOE nodes have been retracted.

• How to sync the retracted status of these replicas in the index whose origins were not published on the DOE nodes
• Shall we replicate their new origins if they are published on non-DOE nodes?

[jpnavarro]

We should implement the ESGF policy that all the dataset and file metadata for all the replicas is owned by the project that originally published it, except for the replica related fields identifying location and access: data_node, url, access, and replica.

The implementation should propagate all metadata changes to the original publication to all the replicas, excluding replica related fields data_node, url, access, and replica.

So, I think the answers to the two bullets are: the original publication (whether on a DOE node or otherwise) is authoritative, and the retraction of the original must apply to all the replicas on all nodes.

[bstrdsmkr]

My recollection is that we are doing away with the concept of replicas (vs origins) entirely. In the new system, a "replica" is just another entry in the list of URLs the file can be retrieved from

Given that, I believe that:

1. The decision to RETRACT a file (or even whole dataset) is up to the project that produced the data
   • This is purely a metadata action on the index to set the retracted field to True for all affected documents
2. The decision to DELETE a file is up to the individual node operators
   • This is both a metadata action and a node operations action:
     1. Update the list of URLs at which a file is available to remove the nodes that are deleting the file
     2. The deleting node removes the file from disk

As for the two questions:

1. For the current state of things were we still have repetitive file records that only differ by download location, the retracted field MUST be the same value for all records of the same file.
2. The purpose of the queue system was to allow different indices to sync, so I believe we should index all available download locations. Currently, I think that involves publishing a new file record for the new site, but all sites should be equal

[jpnavarro]

@minxu74 , are there fields in the metadata that identify origin versus replica?

We also have repetitive dataset records that only differ by download location. So the retracted field MUST be the same value for all records of the same dataset.

As we are talking about ESGF-1.5, we don't have a queuing system to "help" us in this situation. In ESGF-1.5, replication involves grabbing all the dataset and file metadata from the source, including the retracted field, plugging in new locations values, and publishing the new replica metadata. ESGF-1.5 duplicates entire dataset and file entries with new URLs, and doesn't use multiple URLs in a single metadata entry to point to multiple replicas.

[minxu74]

@jpnavarro as I know, the replica field is used to identify origin (replica=False) and replica (replica=True) metadata if person who published or replicated data followed the conventions.