MICNO-684: Added check that input file not already exist in input_dir/archive#157
Merged
linnbjornerud merged 2 commits intomainfrom Mar 27, 2025
Merged
MICNO-684: Added check that input file not already exist in input_dir/archive#157linnbjornerud merged 2 commits intomainfrom
linnbjornerud merged 2 commits intomainfrom
Conversation
…fore moving file to input_dir/archive
DanielElisenberg
previously approved these changes
Mar 27, 2025
Collaborator
DanielElisenberg
left a comment
DanielElisenberg
left a comment
There was a problem hiding this comment.
This is so simple and does actually fix the whole issue 💯 10/10 would review again
Co-authored-by: Daniel Elisenberg <33904479+DanielElisenberg@users.noreply.github.com>
|
DanielElisenberg
approved these changes
Mar 27, 2025
Collaborator
DanielElisenberg
left a comment
DanielElisenberg
left a comment
There was a problem hiding this comment.
Actually got to review it again 🎉
Still 10/10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added a small check that a dataset does not already exist in input_dir/archive before moving it there.
This is to avoid error when trying to reimport a dataset that is already in the archived-folder (i.e. from a previously failed import).
Since datastore admin allows reimport of archived files/failed imports, now job-executor will only try to import the archived dataset even if a dataset with the same name also exists in the input directory. And there will be no conflicts.
If I have understood the dataflow and problem correctly - is this a sufficient solution?