DUM failed to load the last several files in Batch 0 #279
Description
Checked for duplicates
No - I haven't checked
🐛 Describe the bug
When I did loaded a "small" ATLAS collection, I noticed 171 of the 177 files loaded. 6 files failed. All the
error messages were similar to
Giving up ingress_file
_to_s3(...) after 1 tries (requests.exceptions.HTTPError: 403 Client Error: Forbidden
for url: https://pds-sbn-staging-mcp-test.s3.amazonaws.com/sbn/244/58786/data/01a58786
o0014o.ddc.csv?....
I wondered why the access was suddenly forbidden when it had just worked. When reviewing the log, I
noticed that the 6 failed to load files were the last 6 file in Batch 0. When I looked at the last load entry
for Batch 1, I noticed it finished about a minute before the Batch 0 failures started. Did Batch 1's end impact Batch 0's access?
🕵️ Expected behavior
I expected Batch 0's uploads to complete.
📜 To Reproduce
- Run an upload requiring 2 batches with the later batch having being able to finish first?
- Only run 2 threads
...
🖥 Environment Info
- Version of this software [e.g. vX.Y.Z]
- Operating System: [e.g. MacOSX with Docker Desktop vX.Y]
...
📚 Version of Software Used
No response
🩺 Test Data / Additional context
No response
🦄 Related requirements
🦄 #xyz
⚙️ Engineering Details
No response
🎉 Integration & Test
No response
Metadata
Metadata
Assignees
Type
Projects
Status
Status