Upgrade/improve product exists logic

[abradley60]

Main Changes

• Changing logic to first check if the burst files already exist in S3, and stopping the process early if the do (unless make_existing_products is set)
• This is changed from the previous logic which will always download a scene first
• This can only be implemented when --scene_data_source ASF is set, as the burst_id's for a given scene cannot be queried from the CDSE (It seems the burst_id in the format of t012_025211_iw2 is an ASF thing. Thus, --scene_data_source CDSE will still download the scene before checking if the burst products exist
• The change is important for the production of the RTC_S1_STATIC layers. We can now run a large time period with little overhead to ensure all of the required static layers are created.

Other

• increased ASF timeout to 90s from default of 30s
• Cleaning up error codes and shuffling some stuff out of the main cli

[caitlinadams]

Generally looks good to me -- I've made minor suggestions to do with formatting.

[geoscience-aman]

Hi Alex, thanks for adding the capability to exit early if the static layers exist on S3.

Two suggestions:

1. Even if --scene_data_source CDSE, can't we still use asf_search outside the if else block to check whether the static layers exist on S3? If the user hasn't provided ASF credentials, then I understand we can revert to our current fallback of downloading the scene with CDSE.
2. For exiting early, it is unclear to me how this currently happens. Is it that the burst_id_list will be empty? If this is the case, maybe it's worth adding a more explicit exit statement.

[abradley60]

Hi Alex, thanks for adding the capability to exit early if the static layers exist on S3.

Two suggestions:

1. Even if --scene_data_source CDSE, can't we still use asf_search outside the if else block to check whether the static layers exist on S3? If the user hasn't provided ASF credentials, then I understand we can revert to our current fallback of downloading the scene with CDSE.
2. For exiting early, it is unclear to me how this currently happens. Is it that the burst_id_list will be empty? If this is the case, maybe it's worth adding a more explicit exit statement.

Thanks Aman:

1. I was thinking we would mainly be using the CDSE for ongoing processing when products aren't available on the ASF, hence the burst search wouldn't work. I can add it in though because because if we process older scenes with the CDSE as a source this would be valid logic.
2. I can clear this up in the code and name things a little better, but the exit happens here when we check if the requested layers we want to process already exist in s3. If all requested already exist, the process exists early with the 100 code which is then picked up in the shell script that orchestrates everything:

• sar-pipeline/sar_pipeline/aws/cli.py

  Line 282 in 35ebd0e

  burst_id_list = check_burst_products_exists_in_s3(

• sar-pipeline/sar_pipeline/aws/preparation/burst_utils.py

  Line 168 in 35ebd0e

  def check_burst_products_exists_in_s3(

• sar-pipeline/sar_pipeline/aws/preparation/burst_utils.py

  Line 255 in 35ebd0e

  sys.exit(100)

• sar-pipeline/scripts/run_aws_pipeline.sh

  Line 193 in 35ebd0e

  if [ $exit_code -eq 100 ]; then

[geoscience-aman]

1. I was thinking we would mainly be using the CDSE for ongoing processing when products aren't available on the ASF, hence the burst search wouldn't work.

I guess my point here was that even if we are using the CDSE to download the scene, there's no reason we can't use the ASF to check whether the static layers exist on S3 first, right?

[abradley60]

1. I was thinking we would mainly be using the CDSE for ongoing processing when products aren't available on the ASF, hence the burst search wouldn't work.

I guess my point here was that even if we are using the CDSE to download the scene, there's no reason we can't use the ASF to check whether the static layers exist on S3 first, right?

Yeah you're right it's probably just good practice to do that for all scenario's and download the scene if they can't be found and get the burst list from the SLC. Will update

[caitlinadams]

1. I was thinking we would mainly be using the CDSE for ongoing processing when products aren't available on the ASF, hence the burst search wouldn't work.

I guess my point here was that even if we are using the CDSE to download the scene, there's no reason we can't use the ASF to check whether the static layers exist on S3 first, right?

Yeah you're right it's probably just good practice to do that for all scenario's and download the scene if they can't be found and get the burst list from the SLC. Will update

Would the user have to supply both ASF and CDSE credentials for this to work?

[abradley60]

Would the user have to supply both ASF and CDSE credentials for this to work?

Not for the search, I don't think credentials are needed. Just the download. Will make sure my logic accounts for that

[abradley60]

@caitlinadams and @geoscience-aman I have added the logic to query the ASF for bursts in the scene even if the CDSE is set as the data source. Are you able to review?

[caitlinadams]

Looks good to me, thanks Alex!