Redesign "snapshots" to be "operations"

[zachdaniel]

Right now, collaborating with multiple developers using the migration generator can be cumbersome. You have to remove any generated migrations locally, pull, and then regenerate migrations. This is because snapshots essentially are a linear process.

If we rearchitect this to use a single json file with an array of operations that have been generated, and then have the snapshots derived as a projection of those operations, then there will no longer be issues w/ multiple developers working together with the migration generator (e.g on different branches).

[kernel-io]

I just saw this got released recently, it's late here, but part of my brain says this may help
https://github.com/slab/delta-elixir

It sounds about right, just layer each operation on top of each other to get the current 'state' of all the operations that have been applied....

[zachdaniel]

Yeah, that is a great idea :)

[zachdaniel]

We would make operations work on top of old snapshots, so it wouldn't be a breaking change (i.e require removing/doing surgery on snapshots), and I think there may actually be a relatively simple way to do it, which would be simply to store a json patch or delta of some kind to the snapshot object.
So the process for getting a snapshot could be:

1. get the latest snapshot (for backwards compatibility)
2. apply all of the operations in the operations/ folder for that table
3. generate the new snapshot (in memory)
4. diff the old snapshot and new snapshots
5. write the diff as a new list of operations in the operations folder

Generally speaking, interleaving operations should be just fine after that, although there may need to be some small structural changes, e.g instead of keeping lists like [{name: "name"}] we will need to keep an object like {"name": {...}}. This can also be solved in a backwards compatible way by checking for lists when serializing old snapshots and converting to objects, no need to alter the old snapshot files or anything.

[zachdaniel]

We actually want to make these operations part of core, so that any extension can react to these changes.

[zachdaniel]

One small note: each operation should probably get its own file, or each batch.