Roadmap for V2 of the ML Model Extension

[rbavery]

Below are acceptance criteria for #2 the new version of the ML Model Extension. After all these items are done I think #2 is ready to merge and we can publish a new version 2 release to https://github.com/stac-extensions/ml-model/issues . I'm working on most of these, let me know if you think other acceptance criteria should be included cc @fmigneault

• Incorporate feedback from @fmariv and others at @earthpulse
  • I had a meeting with @fmariv, they're keen to review the extension, provide feedback and see if we can align it more with general ML use cases. They're working on supporting the STAC extension ecosystem for ML as a part of the EOTDL initiative
  • https://hackmd.io/DBRF1sQCS1WmSqygJNKQJQ?view
  • feel free to comment first and then submit a PR against https://github.com/rbavery/dlm-extension/tree/validate
• address Potential improvements from STAC community meeting #5
  • this will involve segregating flat fields from objects that are not meant to be searched on and are instead used for inference model loading, model input and output processing, and documenting in detail the accelerator and runtime details needed to run the model
• improve output section of the spec for popular detection tasks #4 by creating output objects for common detection tasks. At a minimum, start with single label classification, semantic segmentation, object detection. This will help resolve common ambiguities when interpreting model outputs, like the bbox coordinate ordering for object detection. EDIT this can be left for after the v2 release
• resolve how to reference the ML model extension from other STAC items/collections #3, deciding how to refer to model extension metadata. within it's own STAC item/collection json or are these fields composed with common metadata in STAC json representing a spatiotemporal asset?
• New Machine Learning Model Extension Version 2.0.alpha schema and (de)serialization, validation package #2 (comment) create an example and schema for the final spec
• reorganize and update the stac_model extension validation and serialization package according to the final spec: New Machine Learning Model Extension Version 2.0.alpha schema and (de)serialization, validation package #2 (comment)
• fix JSON schema with MLM fields + support pydantic/pystac objects rbavery/dlm-extension#2
• consider inputs from user feedback (Revisiting the ML Model extension stac-extensions/ml-model#13)
• consider community examples applying DLM (https://github.com/sentinel-hub/stac-ml-example)
• handle complex normalization with clipping #8
• handle multiple model artifacts that are associated with a model #9
• Allow various Bands representations of Model Input Object #12
• Link on best practices to STAC check to close discoverability loop #13
• integrate JSON schema representing new properties/fields (mix of https://github.com/crim-ca/dlm-extension/tree/main/json-schema and ML Model schema definition rbavery/dlm-extension#1)
• hyperparameter definition #14
• Proposal: Extend the Classification schema for integration with Machine Learning Model extension stac-extensions/classification#48
• https://github.com/stac-extensions/example-links/issues/4
• Add more example expression objects stac-extensions/processing#31
• add more processing:expression examples for python, docker and generic URI stac-extensions/processing#33
• consideration of end_datetime: null not supported by STAC Core spec
  (Datasets without time radiantearth/stac-spec#1268)

[devisperessutti]

Added some comments to the HackMD summary file.

Happy to take on some of the tasks above to push this forward.

[rbavery]

Hi @devisperessutti thanks a bunch. I talked with Francis and I think the main things left to do are to update the example with the current spec (which I can do) and to make a schema json that follows the README description of the schema.

We now have an example.json that combines required pystac common metadata with required ML Model Extension metadata. do you want to take a stab at writing the schema json and PR that against stac-extensions/example-links#2 ? I'm happy to review and assist.

I think improving the spec so that output tasks have their own object structure can be a to do for post Version 2 so that we can start contributing smaller PRs after this large refactor.

[devisperessutti]

We now have an example.json that combines required pystac common metadata with required ML Model Extension metadata. do you want to take a stab at writing the schema json and PR that against stac-extensions/example-links#2 ? I'm happy to review and assist.

Sounds good, I'll try to tackle this early next week.

[fmigneault]

Related by not directly impacting the roadmap: stac-extensions/ml-aoi#8

[fmigneault]

@rbavery
With rbavery#2, I now got functioning JSON-schema validation against a few JSON examples using different combinations of STAC extensions. I applied a few modifications to the pydantic/pystac definitions to better align with the schema. There might be some discrepancies here and there, such as minItems not perfectly validated the same way, but the eurosat_resnet example loaded as pystac.Item works against the JSON schema as well. I only got a minor validation of bands property that I need to fix to wrap things up (see "FIXME_$ref": "#/$defs/AnyBandsRef" in the schema).

I would like to get your review of rbavery#2 so that we can integrate it in #2, and realign all branches together.

[fmigneault]

@devisperessutti
I invite you to push updates from your samples in https://github.com/sentinel-hub/stac-ml-example using MLM fields updates from rbavery#2.
Specifically, you can add your samples in the examples/ directory, and then add them to this test (https://github.com/rbavery/dlm-extension/blob/4eb30dab98617dfee4cc6b1e5dbc1d50d4770779/tests/test_schema.py#L8-L18) to validate them against the definitions.
If the test passes, that will allow us to ensure your use cases remain compatible with the extension schema.
Additional feedback/questions welcome.

[fmigneault]

Moved to crim-ca#4