Support for COG scenes from CDSE

[johntruckenbrodt]

The Copernicus Data Space Ecosystem (CDSE) distributes scenes as COG with a different naming scheme, e.g.:
S1A_IW_GRDH_1SDV_20151021T063752_20151021T063817_008247_00B9CD_2770_COG.SAFE

[oceanites]

I tried to support this and did some progress:
One issue are the changed names which can be fixed by

• adding r'(?P<cog>_COG)?' to pyroSAR.patterns.safe
• adding r'(?P<cog>-cog)?' to pyroSAR.drivers.SAFE().pattern_ds

---

Here are the full regexes and different names:

Filename regex and names:

^(?P<sensor>S1[ABCD])_(?P<beam>S1|S2|S3|S4|S5|S6|IW|EW|WV|EN|N1|N2|N3|N4|N5|N6|IM)_(?P<product>SLC|GRD|OCN)(?P<resolution>F|H|M|_)_(?P<processingLevel>1|2)(?P<category>S|A)(?P<pols>SH|SV|DH|DV|VV|HH|HV|VH)_(?P<start>[0-9]{8}T[0-9]{6})_(?P<stop>[0-9]{8}T[0-9]{6})_(?P<orbitNumber>[0-9]{6})_(?P<dataTakeID>[0-9A-F]{6})_(?P<productIdentifier>[0-9A-F]{4})(?P<cog>_COG)?\.SAFE$

S1C_IW_GRDH_1SDV_20250522T181408_20250522T181433_002447_0051DB_5E65_COG.SAFE
S1C_IW_GRDH_1SDV_20250522T181408_20250522T181433_002447_0051DB_0875.SAFE

Annotation files inside the Sentinel-1 archive:

^s1[abcd]-(?P<swath>s[1-6]|iw[1-3]?|ew[1-5]?|wv[1-2]|n[1-6])-(?P<product>slc|grd|ocn)-(?P<pol>hh|hv|vv|vh)-(?P<start>[0-9]{8}t[0-9]{6})-(?P<stop>[0-9]{8}t[0-9]{6})-(?:[0-9]{6})-(?:[0-9a-f]{6})-(?P<id>[0-9]{3})(?P<cog>-cog)\.xml$
s1c-iw-grd-vh-20250522t181408-20250522t181433-002447-0051db-002.xml
s1a-iw-grd-vh-20230103t052549-20230103t052614-046615-059639-002-cog.xml

[oceanites]

Another issue I found was that gdal.Open() complains about the COG files in the GRDBorderNoiseRemoval step:

Traceback (most recent call last):
  File "/home/tom/venvs/blossom/lib/python3.13/site-packages/pyroSAR/snap/util.py", line 644, in geocode
    gpt(wf_name, groups=groups, cleanup=cleanup, tmpdir=outname,
    ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        gpt_exceptions=gpt_exceptions, gpt_args=gpt_args,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        removeS1BorderNoiseMethod=removeS1BorderNoiseMethod)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/venvs/blossom/lib/python3.13/site-packages/pyroSAR/snap/auxil.py", line 411, in gpt
    scene.removeGRDBorderNoise(method=removeS1BorderNoiseMethod)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/venvs/blossom/lib/python3.13/site-packages/pyroSAR/drivers.py", line 1719, in removeGRDBorderNoise
    S1.removeGRDBorderNoise(self, method=method)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/venvs/blossom/lib/python3.13/site-packages/pyroSAR/S1/auxil.py", line 758, in removeGRDBorderNoise
    ras_master = gdal.Open(master, GA_Update)
  File "/home/tom/venvs/blossom/lib/python3.13/site-packages/osgeo/gdal.py", line 9239, in Open
    return _gdal.Open(*args)
           ~~~~~~~~~~^^^^^^^
RuntimeError: File /home/tom/scratch/blossom/S1A__IW___D_20230103T052549_Cal_NR_Orb_ML_TF_TC_dB/bnr/S1A_IW_GRDH_1SDV_20230103T052549_20230103T052614_046615_059639_B924_COG.SAFE/measurement/s1a-iw-grd-vv-20230103t052549-20230103t052614-046615-059639-001-cog.tiff has C(loud) O(ptimized) G(eoTIFF) layout. Updating it will generally result in losing part of the optimizations (but will still produce a valid GeoTIFF file). If this is acceptable, open the file with the IGNORE_COG_LAYOUT_BREAK open option set to YES.

This can be fixed by changing the code in question to

ras_master = gdal.OpenEx(master, GA_Update, open_options=["IGNORE_COG_LAYOUT_BREAK=YES"])
ras_slaves = [gdal.OpenEx(x, GA_Update, open_options=["IGNORE_COG_LAYOUT_BREAK=YES"]) for x in scene.findfiles('s1.*tiff') if x != master]

[johntruckenbrodt]

Hi @oceanites,
thanks a lot for looking into this. I am on leave this month and cannot do much myself.
Your changes look good. As long as the SAR processor (SNAP/GAMMA) supports these products, it should indeed mostly be regex pattern adjustments. The border noise issue you're encountering can also be avoided by changing the removal method used. The function geocode has a parameter for it. The default is a custom pyroSAR implementation, which modifies the scene in place. Modifying the GDAL call should be fine.
Would you like to open a PR for this? I will look it through once I am back in July.
Cheers,
John