Add support for fetching source HLS scenes to OPERA metadata export

[cmspeed]

This PR introduces the ability to fetch and append corresponding source HLS (Harmonized Landsat and Sentinel-2) scenes for OPERA products directly into the output opera_products_metadata.xlsx. By passing a new CLI flag, users can now automatically retrieve the download links for R/G/B/NIR, and Fmask bands associated with the corresponding OPERA granules.

Key Changes

• Added --include-hls flag to next_pass.py to optionally trigger the fetching of source HLS scenes.

• When --include-hls is active, the opera_products_metadata.xlsx file will include six new columns:

  • Source HLS Granule ID
  • HLS Download URL (B04/Red)
  • HLS Download URL (B03/Green)
  • HLS Download URL (B02/Blue)
  • HLS Download URL (B8A/B05/NIR)
  • HLS Download URL (Fmask)

• Utilizes the earthaccess library to fetch HLS data links. Implements a two-tiered search strategy to find the corresponding source scenes:

  • Primary: Extracts the HLS granule ID directly from the OPERA product's InputGranules metadata (Standard for DSWx).
  • Fallback: Dynamically searches CMR using the Tile ID, date, and bounding box (Required for DIST products).

• Maps the Near-Infrared (NIR) band based on the specific HLS collection, accounting for naming differences between Sentinel-2 (HLSS30 uses B8A) and Landsat (HLSL30 uses B05).

Test functionality

The following command will return and opera_products_metadata.xlsx containing the corresponding HLS scenes.

next-pass -b 35.5 36 -107 -106 --include-hls

[ehavazli]

Critical Issues - Must Fix Before Merge

1. Breaking API Change - Function Signature

Location: next_pass.py:255

Adding include_hls between compute_tide and products breaks backward compatibility for positional calls.

Fix: Move include_hls after satellites OR make keyword-only:

def run_next_pass(
    functionality: str = "both",
    compute_cloudiness: bool = False,
    compute_tide: bool = False,
    products: List[str] | str | None = None,
    satellites: List[str] | str | None = None,
    *,  # Force keyword-only
    include_hls: bool = False,
):

2. Wrong earthaccess API Parameter

Location: utils/opera_products.py:180

earthaccess 0.17.0 uses granule_name, not granule_ur. Current code fails silently.

Fix:

results = earthaccess.search_data(short_name=collection, granule_name=granule_id)

3. Missing Geometry Guard

Location: utils/opera_products.py:422

Fallback search passes bounding_box=None when geometry missing, may crash.

Fix: Skip fallback if no geometry:

if tile_id and start_time != "N/A" and geom:
    search_bounds = geom.bounds
    hls_results = earthaccess.search_data(...)

4. Only First HLS Granule Processed

Location: utils/opera_products.py:394

next() only processes first HLS input. Multiple source granules ignored.

Fix: Handle all or document why first-only correct:

raw_hls_list = [g for g in input_granules if g.startswith("HLS.")]

5. Missing Tests

No test coverage for:

• HLS extraction from InputGranules
• Fallback CMR search
• Band mapping logic (S30 vs L30)
• Error cases

Lower Priority (Follow-up OK)

• Sequential API calls may be slow for large result sets (consider parallelization)
• No progress indication during HLS fetching
• "N/A" doesn't distinguish API failure from missing data

[cmspeed]

Thanks for spotting these. I have just pushed changes addressing the following:

• Updated the fallback search condition to explicitly check for the presence of geom before attempting to access geom.bounds.
• Added a clarifying comment in the code to address first hls granule selection. Because OPERA HLS-derived products are generated on a strict 1-to-1 basis with the source HLS MGRS tiles, there is only one source scene per product. Therefore, retrieving the first match via next() is the expected behavior.
• Updated the error handling to return "API_ERROR" if the CMR API times out/fails, and "NOT_FOUND" if the query succeeds but returns zero results. "N/A" is now strictly reserved for non-HLS products (like Sentinel-1 datasets) that bypass the search block entirely.
• Added several mock-based unit tests to test_opera_products.py to cover the HLS InputGranules extraction, the S30/L30 band mapping logic, the fallback CMR search, and the API error handling. Also added an integration test (test_hls.py) to ensure the external CMR API contract remains valid over time.

[ehavazli]

Thanks for addressing the issues! The fixes look solid.

A few things to consider for later (not blocking this PR):

The integration tests call the live CMR API which might cause CI flakiness. You could skip them in CI with pytest -m "not integration" or mock the earthaccess calls.

fetch_hls_granule_links returns None on errors but [] when there are no results. Works fine but the mixed return types could trip someone up later.

The tile ID extraction uses length check but doesn't validate the format. Something like r'^T\d{2}[A-Z]{3}$' would catch malformed IDs, though the current approach works for real data.

Looks good to merge.