Support for STAC metadata

[fmigneault]

Description

The STAC format has been proposed to handle the differences of metadata/reference files formats across EO inputs. It should be considered as a supported format to simplify (or offer more leverage over) Application Package creation.

STAC allows handling the following issues (not necessarily exhaustive) :

• different sensors that produce different output files (think for example LiDAR point cloud vs Optical imagery or Radar imagery)
• satellite output of different formats and references (different bands, metadata, etc. for RS1, RS2, S1-S3, etc.)
• specifying a list of interrelated images as an input (think for example of all files within a S2 .SAFE directory) - now we must enforce a ZIP to move them around all together

Concerns

• When a docker container is executed for the AP, the files with common base directory (ex: all under the .SAFE), will have to be mounted under the same internal volume so that any relative mapping by GDAL still work.
• No explicit mime-type (registered in IANA) yet exists for STAC.
  • We need to assume something like the following. This case will need to be handled in weaver.formats.
    • application/x-stac
    • application/stac+json
    • Item: application/geo+json; application=stac; profile=item
    • Catalog: application/json; application=stac; profile=catalog
    • Collection: application/json; application=stac; profile=collection
  • We don't want to assume any generic application/json as this could well be a valid input type that doesn't represent a STAC definition/schema.
  • Alternatively, an application/json could be allowed if the stac_version field is located in it.
• Consider schemas provided under https://schemas.stacspec.org/
  • The corresponding CWL format: https://schemas.stacspec.org/v1.1.0/item-spec/json-schema/item.json could be considered to resolve ExecuteCollectionFormat STAC representations
    

    weaver/weaver/execute.py

    Lines 100 to 106 in 5a4805e

    	class ExecuteCollectionFormat(Constants):
    	STAC = "stac-collection" # type: ExecuteCollectionFormatType_STAC
    	STAC_ITEMS = "stac-items" # type: ExecuteCollectionFormatType_STAC_ITEMS
    	OGC_COVERAGE = "ogc-coverage-collection" # type: ExecuteCollectionFormatType_OGC_COVERAGE
    	OGC_FEATURES = "ogc-features-collection" # type: ExecuteCollectionFormatType_OGC_FEATURES
    	OGC_MAP = "ogc-map-collection" # type: ExecuteCollectionFormatType_OGC_MAP
    	GEOJSON = "geojson-feature-collection" # type: ExecuteCollectionFormatType_GEOJSON

• When the EMS receives AOI/TOI/CollectionID inputs (which in turns produces OpenSearch requests), the results obtained will have to be parsed in order to formulate the appropriate STAC definition and forward them to the ADES.
• https://github.com/opengeospatial/ogcapi-processes extensions (Part 3: Workflows & Chaining) are being considered to add support to collections, including STAC. Alignment with their developments and recommendations of EO Apps Best Practices must be considered.
• Special handling will have to be accomplished to distinguish a JSON input object as STAC collection from other JSON structures used for Complex (file) and/or free-form OAS object with schema ([Feature] Support I/O schema field definition #245).

References

Specifications

• Examples/definition of collection input structure in OGC-API:
  • Workflow examples and use cases ... opengeospatial/ogcapi-processes#279 (comment)
  • Workflow examples and use cases ... opengeospatial/ogcapi-processes#279 (comment)
  • Workflow examples and use cases ... opengeospatial/ogcapi-processes#279 (comment)
• Definition of stac-input in CWL EO Apps Best Practices: https://docs.ogc.org/bp/20-089r1.html#req-app-stac-input
  (requires support of Directory type, with STAC items located in it based on catalog.json file)
  (see specifically 8.4.1. Staging Input and Output EO Products in https://docs.ogc.org/bp/20-089r1.html#toc28 for example CWL and corresponding STAC files targeting the Directory)
• Properties comparison between STAC and OGC API - Records
  Subtle differences in property naming between DCAT and Records opengeospatial/ogcapi-records#158 (comment)
• STAC / Records Harmonization  opengeospatial/ogcapi-records#178
• Consider Language support
  • https://github.com/stac-extensions/language
  • HTTP Accept-Language https://www.rfc-editor.org/rfc/rfc3282#section-3, https://www.rfc-editor.org/rfc/rfc7231#section-5.3.5
  • Update owslib to for Accept-Language header #78

Implementation

• https://crim-ca.slack.com/archives/CPMEDH2NL/p1652371575119009

  pip install pystac-client
  stac-client search http://stac-dev.crim.ca/stac/ -c c604ffb6d610adbb9a6b4787db7b8fd7 \
    -q "variable_id=tx_mean" "driving_model_id=MPI-ESM1-2-HR" "modeling_realm=atmos"

• https://github.com/cedadev/asset-scanner
• https://github.com/matprov/asset-scanner-example/blob/thredds_input_example/item_descriptions/CMIP5.BCCAQv2.txgt_37.allrcps.yaml
• https://github.com/matprov/asset-scanner-example/blob/thredds_input_example/conf/thredds-extract-assets.yaml
• https://github.com/radiantearth/stac-spec
• https://github.com/azavea/pystac