How to add tasks dynamically?

[odesenfans]

Hello! Very interesting project.

I have a question on how to use QMDB in a dynamic setting where I don't know my tasks in advance. I see in the examples that you need to provide a task list when starting a new block, and SharedAdsWrap.add_task only takes a task_id as parameter.

What can I do if I don't know my tasks in advance?

Thanks!

[absolute0kelvin]

Yes, you can.

First you create a large enough task_list with RwLock::new(None) and then create a TasksManager with this task_list and a very large last_task_id. Here is some useful code: https://github.com/LayerZero-Labs/qmdb/blob/main/qmdb/examples/v2_demo.rs#L63. You can change it to fit your needs.

Then you can dynamically change the task#i in task_list and after that, use shared_ads.add_task to feed i to QMDB.

When you finish the last task, call TasksManager.set_last_task_id before using shared_ads.add_task to feed last_task_id to QMDB.

[odesenfans]

Out of curiosity, what are the benefits of this approach? At first glance is forces you to potentially pre-allocate an XXL vector, for data that is eventually copied to an entry?

[absolute0kelvin]

To enable real-time response.

Suppose you are a centralized sequencer and you are packing transactions into blocks. Each block can have 100K transactions at most. And the block interval is 2 seconds. Normally, you build a block every 2 seconds and the other fullnodes get the whole block and then execute it.

Now you have another option: you stream the transactions to fullnodes like RSTP and each fullnode execute the received transactions one by one immediately. When you tell the fullnodes "this block stops here", they tell QMDB to end current block by calling TasksManager.set_last_task_id.

Thus, a fullnode can inform the user who eth_subscribe to a transaction ASAP. The total latency include: 1) the RPC node broadcasts a transaction to the sequencer; 2) the sequencer confirms this transaction will be included in current block by broadcasting it through a RSTP-like stream; 3) the RPC node receives the confirmation in the stream and execute the transaction; 4) the RPC node returns the transaction's receipt to its sender. Please note the block interval is not included in the latency.

No mater how long the block interval is, you can get the receipt of the transaction sent by you ASAP.