Backward migrations

[psliwa]

Hello,

Are backward migrations possible in stamina? Using current api I wasn't able to figure out how to define backward migration.

The use case is rollback: when we deploy new version with schema version V5 but then we decide to rollback whole application to version with V3 schema, we would like to migrate data from V5 to V3 in the fly.

Does it make sense? If it does I can try to prepare PR for that.

[agemooij]

Interesting idea!
It's certainly something we have never considered before and I think it would break quite a bit of the conceptual model behind Stamina. Wouldn't it be easier to always roll forward, even if v6 would be functionally equivalent to v3?

How would you change Stamina to support this? What would the code look like?

[psliwa]

In our deployment model rollback is a deploy of old version of application, so there is no way to add just one commit to codebase with migration.

The idea is to define backward migration along with forward migration. When we want to replace version X of application and deploy version Y, then we have to store somewhere what version of each message is in application version X. Having that, in case of rollback we make backup of current database, migrate backup form newer to older version (using backward migrations). Then we can drop tables and restore migrated backup. We want to do this only for snapshots in order to minimalize number of messages.

I haven't think yet about how to implement this, because I don't have confirmation if it has sense from your point of view.

[agemooij]

I think it would depend on the solution and how it impacts the usability of the main model more than anything else. If adding backwards migrations (i.e. an unusual corner case AFAIK) would make Stamina a lot harder to use, I'd vote to keep it out. If, on the other hand, it could be added in a way that makes sense and keeps the main use case as simple as it is now, I would be interested in merging it.

I think the only way (I can see) to do this cleanly would be to keep the main DSL mostly untouched but somehow make sure every transformation is safely bidirectional so it can be applied backwards. And then You would have to separate the idea of the "currently supported" version from the migrations in some way.

In other words: I don't really have an idea for how to do it but if you can come up with one, we would definitely be interested!

[kkafara]

I was reading this discussion and came with an idea similar to @psliwa's one but working in a slightly different way.

Currently both Persister[T, V] and Migrator[R, V] can handle events saved with version X <= V. What if it could handle also V + 1 on 'unpersist' side?

I think it could be useful in few situations. Possible scenario could be:

• app running in production
• schema migration planned (schema changes shouldn't probably be done often so it can be planned)
• first deploy same version of app with only added ability to read newer data V+1 and migrate it one version back to V
• deploy app that actually writes data in newer version

what it can solve:

• rollback to previous version, as @psliwa described in first post, is possible, as it should be able to read data saved by 'failed' deployment - but only one version backwards
• rolling upgrade in case of clustered application that uses shards will be much simpler: now when few nodes are upgraded and shard is migrated to older node it won't recover there because it can't deserialize data. With this change all nodes should be able to read newer data so it doesn't matter where shard will be (re)started as it should be able to recover on every node

[psliwa]

In general @kkafara idea looks IMHO ok, rolling upgrade looks promising. The only concern I have is that it moves complexity from eventual rollback (which is something that doesn't occur so often) to regular development and deployment process. So when you have more persisted types, developing become complicated - you need to prepare new RC with only new migrator and perform two deployments: old production version with new migrators and new production version. Maybe both solutions can be applied?