Data sources

[abradley60]

Upgrades:

• PR for downloading scenes and orbits from either CDSE or the ASF
• Timing decorator to time key functions
• Fixing docker entry for running in AWS

Discussion:

A note on timing we may want to consider. For the same scene below is the timing difference for ASF vs CDSE. Perhaps we can implement a BEST option for sourcing data? E.g. if scene older than say 5 days get from the ASF, otherwise get from the CDSE. The NCI is also an option and in theory closer to our region. Open to dicussion on this...

Downloding S1A_IW_SLC__1SSH_20220101T124744_20220101T124814_041267_04E7A2_1DAD

INFO:sar_pipeline.utils.general:timing : download_slc_from_asf : 148.38s

vs.

INFO:sar_pipeline.utils.general:timing : download_slc_from_cdse : 401.65s

[geoscience-aman]

Good stuff Alex!

The decorator for timing the functions is really neat and seems useful.

Regarding the broader question of what data source we use, we can:

1. Use ASF for both orbits and SLCs given that the orbit problem has now been fixed in sentineleof. The risk seems low here since we will specify exactly what scene we want so our program will at least fail loudly. If we find this is happening often, we can switch to CDSE or use CDSE as a fallback.
2. Just use CDSE and deal with the longer time for downloading the SLCs.

My preference is the first option, but I will leave it up to you to decide :)

[abradley60]

Thanks Aman!

I agree with point 1. I think the one caveat is for near real time processing. From the checks I have made it seems the ASF can sometimes be well over a day behind the CDSE. So perhaps for back-processing we can rely on the ASF and then move towards the CDSE or even Aus cophub which is also pretty up to date for NRT?

[geoscience-aman]

I agree with point 1. I think the one caveat is for near real time processing. From the checks I have made it seems the ASF can sometimes be well over a day behind the CDSE. So perhaps for back-processing we can rely on the ASF and then move towards the CDSE or even Aus cophub which is also pretty up to date for NRT?

In that case maybe we should just use CDSE then to avoid extra logic and having to maintain two different images.

[abradley60]

I agree with point 1. I think the one caveat is for near real time processing. From the checks I have made it seems the ASF can sometimes be well over a day behind the CDSE. So perhaps for back-processing we can rely on the ASF and then move towards the CDSE or even Aus cophub which is also pretty up to date for NRT?

In that case maybe we should just use CDSE then to avoid extra logic and having to maintain two different images.

Shouldn't need to maintain two images. It's a flag that can be set at runtime --scene_data_source which allows ASF or CDSE . This can just be changed when needed. Or, we could easily specify something like a BEST option which would use the ASF for old scenes, and CDSE for new scenes. Just spitballing here, can see what Cailtin thinks.

[caitlinadams]

I've added some minor suggestions to make the docs clear, and a question.

[caitlinadams]

I agree with point 1. I think the one caveat is for near real time processing. From the checks I have made it seems the ASF can sometimes be well over a day behind the CDSE. So perhaps for back-processing we can rely on the ASF and then move towards the CDSE or even Aus cophub which is also pretty up to date for NRT?

In that case maybe we should just use CDSE then to avoid extra logic and having to maintain two different images.

Shouldn't need to maintain two images. It's a flag that can be set at runtime --scene_data_source which allows ASF or CDSE . This can just be changed when needed. Or, we could easily specify something like a BEST option which would use the ASF for old scenes, and CDSE for new scenes. Just spitballing here, can see what Cailtin thinks.

I agree with Alex. The logic is well set-up that we can have one image and just toggle which flag we want to use. I think we could add something like the BEST option if we really wanted to, but think it could probably wait until later.

[caitlinadams]

Thanks for making the fix! Ready for you to merge