feat: scaffold process-compose

[0xmovses]

Summary

• RFCs: Link to RFC, Link to RFC, or $\emptyset$.
• Categories: any of protocol-units, networks, scripts, util, cicd, or misc.

Update process compose stack to start the new Da Sequencer.

Changelog

• update process compose script. Remove the Celestia and Eth process.
• Update build script
• Update de sequencer main to start the grpc + storage. Mock Celestia.
• Correct some bug.
• Add da sequencer setup and config migration for current follower
• Change the Tx pipeling trask processing in the full node to avoid task paused by sync process and starving because another take all the processing slot.
• Put the block execution in one sync code block to avoid multiple call to spawn_blocking.

These changes have been tested using a network of 2 full nodes and one da-sequencer.

Testing

To start the stack alone:

CELESTIA_LOG_LEVEL=FATAL nix develop --extra-experimental-features nix-command --extra-experimental-features flakes --command bash  -c "just movement-full-node native build.setup --keep-tui"

To run the load test:

CELESTIA_LOG_LEVEL=FATAL nix develop --extra-experimental-features nix-command --extra-experimental-features flakes --command bash  -c "just movement-full-node native build.setup.load --keep-tui"

To run the soak test

CELESTIA_LOG_LEVEL=FATAL nix develop --extra-experimental-features nix-command --extra-experimental-features flakes --command bash  -c "just movement-full-node native build.setup.soak --keep-tui"

Outstanding issues

Celestia is a mock.

[ganymedio]

I noticed this error in the Build CI check:

error[E0271]: expected `impl Future<Output = Result<Pin<Box<dyn Stream<Item = Result<BlockV1, ClientDaSequencerError>> + Send>>, ClientDaSequencerError>>` to be a future that resolves to `Result<(Pin<Box<dyn Stream<Item = Result<BlockV1, ClientDaSequencerError>> + Send>>, UnboundedReceiver<()>), ClientDaSequencerError>`, but it resolves to `Result<Pin<Box<dyn Stream<Item = Result<BlockV1, ClientDaSequencerError>> + Send>>, ClientDaSequencerError>`
   --> protocol-units/execution/maptos/opt-executor/src/background/transaction_pipe.rs:452:3
    |
452 | /         async fn stream_read_from_height(
453 | |             &mut self,
454 | |             _request: movement_da_sequencer_proto::StreamReadFromHeightRequest,
455 | |         ) -> Result<StreamReadBlockFromHeight, ClientDaSequencerError> {
    | |______________________________________________________________________^ expected `Result<(Pin<Box<...>>, ...), ...>`, found `Result<Pin<Box<...>>, ...>`
    |
    = note: expected enum `Result<(Pin<Box<dyn Stream<Item = Result<BlockV1, ClientDaSequencerError>> + Send>>, ...), ...>`
               found enum `Result<Pin<Box<(dyn Stream<Item = Result<movement_da_sequencer_proto::BlockV1, ClientDaSequencerError>> + std::marker::Send + 'static)>>, _>`
    = note: the full type name has been written to '/home/runner/actions-runner/_work/movement/movement/target/debug/deps/maptos_opt_executor-e23d7ac56cee81de.long-type-17787066510390306011.txt'
    = note: consider using `--verbose` to print the full type name to the console
note: required by a bound in `movement_da_sequencer_client::DaSequencerClient::{synthetic#0}`
   --> /home/runner/actions-runner/_work/movement/movement/protocol-units/da-sequencer/client/src/lib.rs:40:3
    |
40  |         Output = Result<(StreamReadBlockFromHeight, UnboundedReceiver<()>), ClientDaSequencerError>,
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `DaSequencerClient::{synthetic#0}`

[ganymedio]

Should this file be gitignored?

[musitdev]

It's mistake. I remove it.

[ganymedio]

Should this file be gitignored?

[musitdev]

the same

[ganymedio]

I keep getting Error: invalid DA height causing the full node to panic:

I remove .movement each time I build.

I'll try starting the repo from zero and see if that fixes it.

[ganymedio]

Now I keep getting Error: readiness check fail in movement-full-node after Error: failed to build Funder in movement-faucet

with CELESTIA_LOG_LEVEL=FATAL nix develop --extra-experimental-features nix-command --extra-experimental-features flakes --command bash -c "just movement-full-node native build.setup.eth-local.celestia-local --keep-tui"

I'll try the load test to see if that works for me.

[musitdev]

The error is strange, I run the cargo check --all-targets on Mac and linux.
Perhaps I've done a last minute change and didn't check. Let me see.

[ganymedio]

Hmm when I run the load test it gets skipped, I tried a couple times

[musitdev]

For the first error you should have started from a previous .movement with a wrong start Da height.
For the second, I don't know. I don't see on the process. Does the da-sequencer is working.
For the last I need all the stack trace. There an issue during block processing.

[mzabaluev]

With latest changes, load and soak tests fail with an error in the setup process:

[setup	] 2025-04-28T16:37:32.763986Z  INFO movement_full_node_setup: Starting Movement Full Node Setup
[setup	] 2025-04-28T16:37:32.764095Z  INFO movement_full_node_setup: Configuration file found, start migration.
[setup	] Error: EOF while parsing a value at line 1 column 0

[mzabaluev]

After a811e23, load test fails with transactions timed out at default settings.

The soak test also reports transactions timed out, and after a while block production stops, though the test continues running.

[ganymedio]

@musitdev not sure whether you want to fix in this PR but with cargo check --all-targets I get a few errors:

error[E0308]: mismatched types
   --> protocol-units/execution/maptos/fin-view/src/fin_view.rs:87:56
    |
87  |         let mut executor = Executor::try_from_config(config, mempool_tx_exec_result_sender).await?;
    |                            -------------------------         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `UnboundedSender<Vec<...>>`, found `Sender<Vec<...>>`
    |                            |
    |                            arguments to this function are incorrect
    |
    = note: expected struct `tokio::sync::mpsc::UnboundedSender<Vec<TxExecutionResult>>`
               found struct `futures::futures_channel::mpsc::Sender<Vec<TxExecutionResult>>`
note: associated function defined here
   --> /Users/andygmove/Downloads/repos/movement/protocol-units/execution/maptos/opt-executor/src/executor/initialization.rs:141:15
    |
141 |     pub async fn try_from_config(
    |                  ^^^^^^^^^^^^^^^

warning: unused variable: `tx_sender`
   --> protocol-units/execution/maptos/dof/src/v1.rs:560:8
    |
560 |         let (tx_sender, _tx_receiver) = mpsc::channel::<Vec<(u64, SignedTransaction)>>(16);
    |              ^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_tx_sender`

error[E0061]: this method takes 2 arguments but 1 argument was supplied
   --> protocol-units/execution/maptos/fin-view/src/fin_view.rs:88:47
    |
88  |         let (context, _transaction_pipe) = executor.background(mempool_commit_tx_receiver)?;
    |                                                     ^^^^^^^^^^---------------------------- an argument of type `futures::futures_channel::mpsc::Sender<MempoolClientRequest>` is missing
    |
note: expected `UnboundedReceiver<Vec<...>>`, found `Receiver<Vec<...>>`
   --> protocol-units/execution/maptos/fin-view/src/fin_view.rs:88:58
    |
88  |         let (context, _transaction_pipe) = executor.background(mempool_commit_tx_receiver)?;
    |                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^
    = note: expected struct `tokio::sync::mpsc::UnboundedReceiver<Vec<TxExecutionResult>>`
               found struct `futures::futures_channel::mpsc::Receiver<Vec<TxExecutionResult>>`
note: method defined here
   --> /Users/andygmove/Downloads/repos/movement/protocol-units/execution/maptos/opt-executor/src/executor/initialization.rs:201:9
    |
201 |     pub fn background(
    |            ^^^^^^^^^^
help: provide the argument
    |
88  |         let (context, _transaction_pipe) = executor.background(/* tokio::sync::mpsc::UnboundedReceiver<Vec<TxExecutionResult>> */, /* futures::futures_channel::mpsc::Sender<MempoolClientRequest> */)?;
    |                                                               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

warning: `maptos-dof-execution` (lib test) generated 5 warnings
error[E0277]: `Result<movement_types::block::BlockCommitment, anyhow::Error>` is not a future
   --> protocol-units/execution/maptos/fin-view/src/fin_view.rs:156:34
    |
156 |             executor.execute_block(block).await?;
    |                                          -^^^^^
    |                                          ||
    |                                          |`Result<movement_types::block::BlockCommitment, anyhow::Error>` is not a future
    |                                          help: remove the `.await`
    |
    = help: the trait `futures::Future` is not implemented for `Result<movement_types::block::BlockCommitment, anyhow::Error>`, which is required by `Result<movement_types::block::BlockCommitment, anyhow::Error>: std::future::IntoFuture`
    = note: Result<movement_types::block::BlockCommitment, anyhow::Error> must be a future or must implement `IntoFuture` to be awaited
    = note: required for `Result<movement_types::block::BlockCommitment, anyhow::Error>` to implement `std::future::IntoFuture`

warning: `movement-signer-test` (test "signer") generated 1 warning
Some errors have detailed explanations: E0061, E0277, E0308.
For more information about an error, try `rustc --explain E0061`.
warning: `maptos-fin-view` (lib test) generated 1 warning
error: could not compile `maptos-fin-view` (lib test) due to 3 previous errors; 1 warning emitted

[mzabaluev]

@andygolay I have fixed the test build.

[ganymedio]

LGTM