Add schema/metadata migrations

[mferrera]

This is just a sketch. This will requirement some refinement, discussion, and design.

We need to be able to migrate metadata from older versions to newer versions. We should add a new block, i.e. schema_migrations that logs this. We should add a single function that upgrade metadata from version n to n+1. This means every schema version should have a corresponding migration function, if it applies.

Sketch:

$schema: ..
version: "0.9.0"
source: "fmu"

schema_migration:
  original_version: "0.8.0"
  - timestamp: 2025-....
    migration_tool:
      name: fmu-dataio
      version: 2.5.0
    from_version: "0.8.0"
    to_version: "0.9.0"
    changes:
    - field: "data.standard_result"
      action: "renamed"
      previous_name: "data.product"
      required_context: false  # If FMU runtime is required
          
tracklog:
  - ...
     event: created
     ...
  - datetime: 2025..
     user:
       id: system
     ...
     event: "schema_migrated"
     migration_details: # this block not necessary, if in a migration block?
       from_version: "0.8.0"
       to_version: "0.9.0"
       migration_id: <uuid>

In dataio, we will need to establish functions that do these migrations and validated them. Sumo can then use them in ETL pipelines.

from fmu.dataio.migrations import migrate_schema

new_schema = migrate_schema(existing_schema, data_object, "0.9.0")
# Apply all schema migrations between the existing schema version and 0.9.0
# Each version change has its own upgrade function. It may in some cases do nothing

Open questions

What about metadata values only possible to get during FMU experiment runtime?

• We probably need to establish default values in these circumstances with a way to flag when they have been generated by a migration. This is a form of "optionality" that adds some difficulty on the consumer end, but not more difficult than managing a multitude of logic for a multitude of schema versions (i.e., it should be able to handle such fields programmatically no matter the version)

[perolavsvendsen]

Schema migration will happen inside Sumo (possibly have an API endpoint for triggering it?), so the function might sit there - not in fmu-dataio. (Don't think we use/can use fmu-dataio directly in Sumo (core), but perhaps this is a separate service?)

Point is, we might discuss exactly where such a function should exist.

[perolavsvendsen]

I agree that the situation where we have mandatory field containing information that we simply do not have during a migration. I don't know the best way of dealing with that, conceptually, but it feels like this could quickly become data pollution. It is easy to envision situations where defaulting might end up rendering the data somewhat useless after a few generations, if the "new" information is something consumers starts to rely on.

I am reluctant to open this door, as it goes against the whole purpose, but perhaps this is one more argument for allowing the data owner to modify individual metadata fields of stored data (in Sumo). We have talked about this in relations to access earlier. We also see that misconfiguration of the FMU workflow can produce erroneous metadata where changing in-place is the least painful fix (as rerunning is not a viable option always).

This also begs the question if a schema migration should/could (eventually) be a user activity. I.e. the user triggers this via the FMU data management front-end (Sumo front-end). Perhaps we automate when we can, and just ask the user to manually do those migrations that require manual input 🤔

I guess the summary of this is that there might be a larger user experience discussion to be had here.

[mferrera]

This issue may or may not be taken up, but in case we look upon it later...

Offline discussion summary:

• Maybe we could use jsonpatch
• Maybe a translation between JSON Patch -> Elasticsearch updates could work
• Maybe we just publish Elasticsearch updates
• No matter the case, publishing functions to do this is probably not the way to go

[mferrera]

I created a package that can do this in a structured way: https://pyrmute.readthedocs.io/en/latest/

I don't believe that this package should be used here because of the way that it works, but my learning is that it is in general simple to provide an interface for these sorts of migrations (in Python). One useful thing I created in this package is strong model diffing capabilities. That is maybe a better thread to pull on, as it would enable migration code generation which means it does not necessarily need to be locked into written Python code.