live movement to migrator movement to migrator movement-aptos to live movement-aptos

[l-monninger]

Summary

As #119 (comment) points out, because movement and movement-aptos migration logic is written using temporary instances of migrator and node, we have potentially different and intermediate versions of the chains. That is, the migration and its checks proceed live movement $\to$ migrator movement $\to$ migrator movement-aptos $\to$ live movement-aptos.

migrator movement $\to$ migrator movement-aptos is what this repository reasons about directly. So, this is covered.

However, practically, we are left with two potential sharp-edges:

1. From the point at which live movement $\to$ migrator movement occurs, we would either have (a) two replicas (perhaps forks) of the chain which proceed independently or else (b) to stop live movement.

Note

On the surface this isn't a big problem because we should be able to forward the blocks processed in between to live movement-aptos and by our criteria have a semantically correct outcome.

2. From the point at which migrator movement-aptos $\to$ live movement-aptos, we would have to have clear idea of how to transition from the movement-aptos network which is in a migrator state to one that is live.

Important

Does migrator movement-aptos $\to$ live movement-aptos mean adding new replicas? More actual participants? Stopping some machines and starting others? Reorganizing p2p broadcasts?

Much of this starts to intersect with the notions of Environments and Contexts presented as abstractions herein. For example, can the closure of an environment be neatly tied to these transitions? Can we complete migrator movement-aptos and launch live movement-aptos from code? Does anything need to happen at all?

To begin this discussion, consider the following questions:

1. What should happen at live movement $\to$ migrator movement?
2. What should happen at migrator movement-aptos $\to$ live movement-aptos?

[l-monninger]

@0xmovses @andygolay @sebtomba @franck44 @apenzk @sausagee

We should probably get clear about what's supposed to happen here.

@mcmillennick

Tagging you too as this starts to invoke a lot of questions about the underlying infrastructure.

[nicholasflintwillow]

I'm finishing up the support needed to get the mainnet performance updates out this week. Next week I'll be starting diving into running an aptos validator node. Last I was able to dive in I saw that they had a docker-compose and helm chart so that will be a good place to start.

[l-monninger]

@mcmillennick

This thread is less concerned with running the validator network--for which the existing helm charts are fine--and instead with the entire process of starting the migration, mirror, and eventually switching over.

[apenzk]

What should happen at live movement $\to$ migrator movement?

Am i correct in that the question is how to exactly handle the transition event?

1. Can we halt live movement? Transition. Continue with migrator movement
migrator ...) this seems a reasonable option. Its reasonable to inform community about an upgrade stop as long it's predictable short.
2. I am lacking context why there are two intermediate networks (migrator movement and .. . Where do I read up on this.

[l-monninger]

Yes, that's the question.

1. I'm not sure how much we can halt. That's what this discussion is for. However, if we want migrate and then check the correctness of the migration--particularly if we are tracking (`tracking` checks and criteria #135)--then we would probably be down for too long.
2. The intermediate networks/chains are what are created herein. See README.md and this example blanket implementation of Migrationish.

[apenzk]

how much is too long? i know some networks have been down hours on significant events. obvs days would not be good.

[l-monninger]

For the snapshot criteria, it is looking like we can evaluate within a couple hours. For tracking, however, it only really makes sense as a cut over if the period of evaluation is longer.

[l-monninger]

@apenzk @mcmillennick

To both of your questions/statements, this repo is currently in place where we can effectively push out a binary (soon container) which when run on nodes that will perform the migration and run checks for the correctness of the migration. However, in so doing, it's obviously creating additional processes which have their own notion of network state--at least temporarily.

[apenzk]

thx. So my understanding for the following is that we intake the same transactions in the same order and hope for the same output at each block height?

To being this discussion, consider the following questions:

• What should happen at live movement → migrator movement?
• What should happen at migrator movement-aptos → live movement-aptos?

[apenzk]

do you think it will yield in fact yield the same output or are there parameters that undoubtedly change the block output

[l-monninger]

Block output by cryptographic summary will be different because the migration features several transactions on migrator movement and migrator movement-aptos which should not be on live movement. However, there are other values which should be equivalent and indeed which we have checked are.

The above is not the part that's missing. It's the dashed lines in the diagram up top which are.

[l-monninger]

#146 grounds this discussion in a potential K8s flow, for those who are interested in trying to start from a more practical basis.

[l-monninger]

@areshand

Would be good to have your comments here.

[areshand]

I think once the migrator Movement-Aptos passes all the checks, we should keep it running as a follower node to catch up on the latest transactions and stay in sync for a while. That gives us a chance to run some additional checks and make sure everything looks healthy.
After that, we should be able to redirect traffic to it and use it as the main validator, right?

[l-monninger]

Yeah, so this is kind of the idea behind the tracking API (see also #135).

After that, we should be able to redirect traffic to it and use it as the main validator, right?

So under this approach, I think there would be two high-level requirements that will require filling in gaps in the implementation and checking with stakeholders:

1. The thing that is migrator movement-aptos MUST also include feature complete replica for live movement-aptos. It can't be a sort of stripped down executor and its configuration should match the target network. And, while we would want to remove the context, processes, etc. that are responsible for enabling checks, it ultimately needs to be live enough to ensure continuity of the network directly.
   1. Do we allow ourselves the ability to restart?
2. There will only be one replica at the start of live movement-aptos.

[areshand]

1. Yes, I imagine we have a load balancer in place that we can use to clone request and send a copy to the migrator-movement-aptos node.
2. I agree that migrator-movement-aptos should be a fully functional node configured to match the target network. It should be able to restart cleanly, assuming transactions are already saved in the DA, so we won’t lose any data on restart.
3. We only need one migrator node in practice, but having multiple follower nodes as backups could help us handle node failures or other unexpected issues. That way, we can quickly hot-swap the validator to another follower node if needed.

[l-monninger]

1. Yes, I imagine we have a load balancer in place that we can use to clone request and send a copy to the migrator-movement-aptos node.

Yeah, I've actually be using an additional container proxy (partly because it's more debuggable). But, that's the idea for "traffic mirroring."

2. I agree that migrator-movement-aptos should be a fully functional node configured to match the target network. It should be able to restart cleanly, assuming transactions are already saved in the DA, so we won’t lose any data on restart.

Okay, so this needs some polishing, as right now we are relying on the localnet and some related abstractions to make starting the multiple services easy.

I do think, if we are knowingly allowing it restart, this can be deferred to somewhat to the K8s jobs and service definitions.

3. We only need one migrator node in practice, but having multiple follower nodes as backups could help us handle node failures or other unexpected issues. That way, we can quickly hot-swap the validator to another follower node if needed.

I'm thinking about it more in terms of availability at the cut over. That is, do we want this to be our production replica set and literally all we are doing is switching over traffic?

[l-monninger]

I will diagram some variations of these approaches later today.