product skipping and general uplift #62
Conversation
sar_pipeline/aws/cli.py
Outdated
| required=False, | ||
| is_flag=True, | ||
| default=False, | ||
| help="Skip the creating burst products if it already exist in the desired s3 bucket path. " |
There was a problem hiding this comment.
Skip creating the burst products if they already exist in the desired S3 bucket path.
| default=False, | ||
| help="Skip the creating burst products if it already exist in the desired s3 bucket path. " | ||
| "Warning - false may result in duplicate products", | ||
| ) |
There was a problem hiding this comment.
WARNING - setting this argument to False may result in duplicate products.
| "--skip-existing-products", | ||
| required=False, | ||
| is_flag=True, | ||
| default=False, |
There was a problem hiding this comment.
Do we want the default to be False?
There was a problem hiding this comment.
Yeah this is a good question. I guess it depends where duplicates should be handles. E.g. if reprocessing is done and new products are released, should we reprocess and handle duplicates later? Can discuss in our meeting today.
There was a problem hiding this comment.
changed to make-existing-products which by default is set to False
sar_pipeline/aws/metadata/stac.py
Outdated
| burst_id=self.burst_id, | ||
| ) | ||
| else: | ||
| raise ValueError() |
There was a problem hiding this comment.
Is it worth putting an error message here?
There was a problem hiding this comment.
Removed and changed to an if and elif, it will already be covered by
| month of burst acquisition | ||
| day : str | ||
| day of burst acquisition | ||
| """ |
There was a problem hiding this comment.
Can you add a return statement to the doc string?
3 participants
--skip-existing-productsflag to search and skip products if they already exist in s3--skip-upload-to-s3to specify skipping the upload to s3 alonedemtodem_type