Rz dedup (#458)

* Log-based dedup

* Extraction dedup

* ariaTSsetup handle dedup with input files removed

* ariaTSsetup bug fixes

* Inc connectedComponents in extracted files list

* DEM download dedup

* Deconflict run log with dev

* Bug fix duplicated logs

* Lazy formatting

* Skip re-download mask

* arrres multiple of base pixel size

* Check for dem and mask consistency

* PEP8 updates

* Support for standard pixel sizes 1 or 3 arcsec

* Bug fix for pixel size standardization

* pep8 fix for res determination

* Removal vestigial section which deletes geoemtry files automatically

* Make sure to also crop connected component file

* Support iterative addition of layers

* Add tile_mate dependency

* Error out when previously extracted GUNWs not found.

* Bug fix for DEM and mask dedup

* Write extracted_file_names to pickle file

* Refine SET and tropo dedup to capture all uninterpolated files

* Bug-fix for checking dimensions of existing tropo and SET extracted layers

* Error out if croptounion option changed from one run to the next

* Move croptounion check early in product sorting function

* Add space in exception message

---------

Co-authored-by: rzinke <rzinke@jpl.caltech.edu>
Co-authored-by: Simran S Sangha <sssangha@ucla.edu>

Author: 3 people

environment.yml

@@ -33,7 +33,7 @@ dependencies:
   - requests
   - scipy>1.10.0
   - shapely
-  - tile_mate
+  - tile_mate>=0.0.12
   - xarray
   #for ARIA-tools-docs
   #- jupyterlab

requirements.txt

@@ -23,7 +23,7 @@ rioxarray
 requests
 scipy>1.10.0
 shapely
-tile_mate
+tile_mate>=0.0.12
 xarray
 ## notebook dependencies, keep commented out for better compatiability with other codes
 #jupyterlab

tools/ARIAtools/constants.py

@@ -9,6 +9,10 @@
 Record global variables for use in other scripts
 """
 
+# List of supported pixel sizes (in arcseconds)
+ARIA_PX_SIZES = [0.000277778,
+                 0.000833334]
+
 # Create lists of all supported models and all aria layers
 ARIA_EXTERNAL_CORRECTIONS = ['troposphereHydrostatic',
                              'troposphereWet',
@@ -30,7 +34,6 @@
                'amplitude']
 ARIA_LAYERS += ARIA_EXTERNAL_CORRECTIONS
 ARIA_LAYERS += ARIA_INTERNAL_CORRECTIONS
-ARIA_LAYERS += ARIA_TROPO_MODELS
 
 ARIA_STANDARD_INTF_LAYERS = ['unwrappedPhase', 'coherence']
 ARIA_STANDARD_GEOM_LAYERS = ['incidenceAngle', 'azimuthAngle']
@@ -57,3 +60,4 @@
     'solidEarthTide': 'setStack'
 }
 ARIA_STACK_OUTFILES.update({i: i + 'Stack' for i in ARIA_TROPO_MODELS})
+