[FEATURE] Index schema update strategy

[ajleong623]

Is your feature request related to a problem?

I was informed by @cwperks that when updating index mappings in the future versions, there should be a strategy to how to keep track of schema versions. This way, we could track and document changes to the index mappings overtime.

What solution would you like?

There should be an additional metadata to denote the version of the schema.

What alternatives have you considered?

I have not yet thought of any alternatives.

Do you have any additional context?

Add any other context or screenshots about the feature request here.

[epugh]

Long discussion at OpenSearchCon NA 2025 about how to migrate the indexes that support our Plugins. Would be nice to have a single way of doing it at the Plugin Architecture. Though we could of course do a one off for search-relevance.

I went to https://opensearchconna2025.sched.com/event/25GqD/a-deep-dive-into-opensearch-plugin-architecture-and-introducing-hot-reload-functionality-pranav-garg-aman-kumar-amazon-web-services and talked to Pranav and Aman about the need for this.

[ajleong623]

@epugh I am still not sure about index migration and the connection with plugins. I would like to have some clarification over this.

[epugh]

In my head, thinking about this is our next big thing after we land scheduled evaluation. I looked at some of our index field names, and I think we didn't follow the same pattern as other plugins in OpenSearch. So we need to think about a migration strategy. I'm inspired on how Ruby on Rails has amazing migrations. Likewise in the Java world, we have Liquibase and Flyway. Think about how these would be applied not to a database, but to a search index.

We've done this in the past on various client projects, so I need to dig up some sample code and see if we have any lessons learned that I could share!

[ajleong623]

@epugh Ok, I have read through the resources you have sent. Looks like we need a way to store all the versions (I already suspected this, but we could not store it in an index, so I would have to find another place to store those). It looks like Ruby on Rails found a way to organize all the possible types of operations in a migration and created a set of methods for those. Let me know if you could show me some of the past client projects as other potential solutions.