DUM loads duplicate files

[plawton-umd]

Checked for duplicates

No - I haven't checked

🐛 Describe the bug

When I did loaded bundle nh_documents::1.0 (which includes collections mission::1.0 and ralph::1.0) and then loaded bundle nh_documents::2.0 (which includes collections mission::2.0, ralph::1.0, and several new version 1.0 collections), I noticed the files in ralph:1.0 were loaded both times and display on the AWS console as separate copies.

🕵️ Expected behavior

I expected the contents of ralph::1.0 to be skipped when nh_documents:2.0 was loaded.

📜 To Reproduce

1. Download https://pdssbn.astro.umd.edu/holdings/pds4-nh_documents-v1.0/ and https://pdssbn.astro.umd.edu/holdings/pds4-nh_documents-v2.0/
2. use DUM to load nh_documents::1.0
3. use DUM to load nh_documents::2.0
4. review logs
   ...

🖥 Environment Info

• Version of this software Data Upload Manager Client v2.1.2
• Operating System: Linux
  ...

📚 Version of Software Used

No response

🩺 Test Data / Additional context

No response

🦄 Related requirements

🦄 #xyz

⚙️ Engineering Details

No response

🎉 Integration & Test

No response

[jordanpadams]

@collinss-jpl @viviant100 this is a much larger issue we need to talk about in the breakout: symlinks. how do we prevent duplication of data and handle symlinks of archives in PDC? this is going to have significant impacts on both DUM and the S3 browser

[jordanpadams]

@plawton-umd do you have to rename the directory where the new version of the data goes? Or can you keep the path the same?

like what if the path to the data remained:

https://pdssbn.astro.umd.edu/holdings/pds4-nh_documents/

And then you just moved the older versions elsewhere if you are concerned about that causing confusion?

[jordanpadams]

@collinss-jpl @viviant100 actually, this may also become a different problem when we check for object checksums using the registry: #99

[plawton-umd]

@jordanpadams I ran some tests using a non-versions soft link to the versioned directory.
The already loaded collection's content was skipped - good.
New collection loaded - good.
The new products in existing collections were loaded - ok.
Existing files that had been updated - including bundle label, collection label, and inventory file - were overwritten
and earlier versions do not appear to be available - issue. Assuming something similar would happen with data products - issue.

[jordanpadams]

@plawton-umd copy that. You will need to manually move/rename those files in the bucket prior to uploading new versions of those products, similar to what you do on a file system now in order to avoid overwriting of files. We can eventually look to enable versioning on archive buckets, but right now we don't know if we can afford that.

[plawton-umd]

@jordanpadams We do not do that now. A new versioned directory is created with links to the already existing files.

If the files in the Cloud have to be renamed, once all the parts of the process are functional, where else will the filename need to be changed please?

[jordanpadams]

@plawton-umd you will most likely need to re-register all the data with the new filenames so that someone can get to those older versions using the API

[tbarnes4]

@jordanpadams @plawton-umd I will mention that sometimes we have accumulating collections, where each version mostly/entirely just adds new products to the collection. In this case there may be very very many old products that do not need to be reuploaded. What is the point of checksums and LIDVIDs if we cannot use them? Is the LIDVID not supposed to be a unique identifier, which we could use, instead of path name? I understand the desire to do a sanity check with checksums which would warn to replace an existing uploaded product that may have a different checksum, but same LIDVID.

[jordanpadams]

@tbarnes4 The short answer is we built this as a tool to upload files to S3, not to do PDS-specific stuff. We intentionally avoided PDS-isms in the design in order to maximize performance. If we add all these PDS checks and reads into the workflow, it will dramatically slow down the ability for DUM to get data to the cloud as fast is possible.

Additionally, we need the ability to support the upload of non-PDS4 files. We can add in some more robust checksum checking once we are actually processing and registering the data. Right now, it is sitting in a staging bucket so we don't have a database to query for whether or not the file has been uploaded to the cloud already.