[Question] RPC Block Lag Issue Analysis and Optimization Request

[giskook]

Issue Description

When Sequencer transaction volume is high, RPC experiences block lag, meaning RPC retrieves new blocks slower than the Sequencer produces them.

Version Information

• Code Version: xlayer-erigon v0.5.1-RC-1 (based on cdk-erigon v2.61.0)
• Hardware Configuration:
  • Sequencer: 16 cores, 64GB RAM
  • Stateless Executor: 5 instances, each with 16 cores and 32GB RAM

Mechanism Analysis

• After the Sequencer generates a block, it must wait for the Stateless Executor to complete block validation before pushing the block to the Data Streamer. RPC retrieves block data from the Data Streamer.
• The result of the Stateless Executor’s validation is only queried in the next cycle, causing RPC to be unable to access the latest block immediately.

Identified Issues

Fast Sequencer processing: Under high load, the Sequencer closes a block in approximately 200ms.
Slow Stateless Executor validation: Each block takes about 4 seconds to be validated, significantly lagging behind the Sequencer’s block production speed, leading to RPC delays.

Optimization Goals

• Enhance Stateless Executor validation efficiency to reduce block validation delays.
• Optimize the data pushing mechanism to enable RPC to access Sequencer-generated blocks faster.

[giskook]

RPC's log

10.36.109.171
[INFO] [02-17|17:39:39.993] RPC Daemon notified of new headers       from=9223780 to=9223781 amount=1 hash=0x7a9655032ff9c40225092a012af2399c826cbff4f33df3f12687d41a4aaaedce header sending=17.82µs log sending=312ns

10.36.109.249
[INFO] [02-17|17:39:39.898] RPC Daemon notified of new headers       from=9223780 to=9223781 amount=1 hash=0x7a9655032ff9c40225092a012af2399c826cbff4f33df3f12687d41a4aaaedce header sending=30.068µs log sending=1.068µs

10.36.111.175
[INFO] [02-17|17:39:39.894] RPC Daemon notified of new headers       from=9223778 to=9223779 amount=1 hash=0x6c3653adfabb38a55c62ac0ecab35319bb96b34613652611bd35206fc1170cd7 header sending=12.468µs log sending=333ns

10.36.110.46
[INFO] [02-17|17:39:39.890] RPC Daemon notified of new headers       from=9223777 to=9223778 amount=1 hash=0x3f9192e056598e3f4a0b84a02a0e4777e48b14ba4e77831f52262ec64f798bf6 header sending=11.493µs log sending=297ns

10.36.111.200
[INFO] [02-17|17:39:39.875] RPC Daemon notified of new headers       from=9223779 to=9223780 amount=1 hash=0xf8c424e0316d3249fb1e11c099a2b5391f31a6639f38452f140705bc750ad37d header sending=12.469µs log sending=335ns

10.36.110.178
[INFO] [02-17|17:39:39.705] RPC Daemon notified of new headers       from=9223777 to=9223778 amount=1 hash=0x3f9192e056598e3f4a0b84a02a0e4777e48b14ba4e77831f52262ec64f798bf6 header sending=16.764µs log sending=415ns

Sequencer's log
block 9223771-9223779 is the busy blocks.

  "content": "[INFO] [02-17|17:39:16.929] [5/13 Execution] Starting block 9223768 (forkid 13)... ",
  "content": "[INFO] [02-17|17:39:19.930] [5/13 Execution] Increment trie hashes started previousRootHeight=9223750 calculatingRootHeight=9223768",
  "content": "[INFO] [02-17|17:39:19.937] [5/13 Execution] Finish block 9223768 with 0 transactions... info-tree-index=54491",
  "content": "[INFO] [02-17|17:39:19.962] [5/13 Execution] Starting verification request batch-number=157632 blocks-range=[9223751;9223768] pending-requests=2",
  "content": "[INFO] [02-17|17:39:20.023] Generating witness timing                batch=157632 blockFrom=9223751 blockTo=9223768 taken=60.187295ms", 
  "content": "[INFO] [02-17|17:39:22.994] [5/13 Execution] Finished verification request batch-number=157632 blocks-range=[9223751;9223768] is-valid=true pending-requests=1",

  "content": "[INFO] [02-17|17:39:19.964] [5/13 Execution] Starting block 9223769 (forkid 13)... ",
  "content": "[INFO] [02-17|17:39:22.966] [5/13 Execution] Increment trie hashes started previousRootHeight=9223750 calculatingRootHeight=9223769",
  "content": "[INFO] [02-17|17:39:22.972] [5/13 Execution] Finish block 9223769 with 0 transactions... info-tree-index=54491",
  "content": "[INFO] [02-17|17:39:22.994] [5/13 Execution] Starting verification request batch-number=157632 blocks-range=[9223751;9223769] pending-requests=2",
  "content": "[INFO] [02-17|17:39:23.056] Generating witness timing                batch=157632 blockFrom=9223751 blockTo=9223769 taken=61.53131ms",
  "content": "[INFO] [02-17|17:39:26.054] [5/13 Execution] Finished verification request batch-number=157632 blocks-range=[9223751;9223769] is-valid=true pending-requests=1",

  "content": "[INFO] [02-17|17:39:22.995] [5/13 Execution] Starting block 9223770 (forkid 13)... ",
  "content": "[INFO] [02-17|17:39:25.997] [5/13 Execution] Increment trie hashes started previousRootHeight=9223750 calculatingRootHeight=9223770",
  "content": "[INFO] [02-17|17:39:26.010] [5/13 Execution] Finish block 9223770 with 1 transactions... info-tree-index=54491",
  "content": "[INFO] [02-17|17:39:26.054] [5/13 Execution] Starting verification request batch-number=157632 blocks-range=[9223751;9223770] pending-requests=2",
  "content": "[INFO] [02-17|17:39:26.123] Generating witness timing                batch=157632 blockFrom=9223751 blockTo=9223770 taken=68.027286ms",
  "content": "[INFO] [02-17|17:39:26.810] [5/13 Execution] Finished verification request batch-number=157632 blocks-range=[9223751;9223770] is-valid=true pending-requests=1",

  "content": "[INFO] [02-17|17:39:26.056] [5/13 Execution] Starting block 9223771 (forkid 13)... ",
  "content": "[INFO] [02-17|17:39:26.603] [5/13 Execution] Increment trie hashes started previousRootHeight=9223750 calculatingRootHeight=9223771",
  "content": "[INFO] [02-17|17:39:26.655] [5/13 Execution] Finish block 9223771 with 30 transactions... info-tree-index=54491",
  "content": "[INFO] [02-17|17:39:26.810] [5/13 Execution] Starting verification request batch-number=157632 blocks-range=[9223751;9223771] pending-requests=2",
  "content": "[INFO] [02-17|17:39:26.812] [7/13 HashState] Promoting plain state   from=9223750 to=9223771",
  "content": "[INFO] [02-17|17:39:26.812] [7/13 HashState] Incremental promotion   from=9223750 to=9223771 codes=true csbucket=AccountChangeSet",
  "content": "[INFO] [02-17|17:39:26.812] [7/13 HashState] Incremental promotion   from=9223750 to=9223771 codes=false csbucket=AccountChangeSet",
  "content": "[INFO] [02-17|17:39:26.812] [7/13 HashState] Incremental promotion   from=9223750 to=9223771 codes=false csbucket=StorageChangeSet",
  "content": "[INFO] [02-17|17:39:26.826] RPC Daemon notified of new headers       from=9223750 to=9223771 amount=21 hash=0xd7d93497223f5e0e755d7762e3b24a728af144e46ba25745867f28c1ddbcc505 header sending=32.001µs log sending=234ns",
  "content": "[INFO] [02-17|17:39:26.880] [5/13 Execution] Increment trie hashes started previousRootHeight=9223771 calculatingRootHeight=9223772",
  "content": "[INFO] [02-17|17:39:27.038] RPC Daemon notified of new headers       from=9223771 to=9223772 amount=1 hash=0xc84d45d94311a6a4b94f0d3a2ef8ad9aa0770f66cc1063a8acf89aca00eb9af8 header sending=8.21µs log sending=243ns",
  "content": "[INFO] [02-17|17:39:27.074] Generating witness timing                batch=157632 blockFrom=9223751 blockTo=9223771 taken=263.559901ms",
  "content": "[INFO] [02-17|17:39:32.789] [5/13 Execution] Finished verification request batch-number=157632 blocks-range=[9223751;9223771] is-valid=true pending-requests=13",

  "content": "[INFO] [02-17|17:39:26.826] [5/13 Execution] Starting block 9223772 (forkid 13)... ",
  "content": "[INFO] [02-17|17:39:26.880] [5/13 Execution] Increment trie hashes started previousRootHeight=9223771 calculatingRootHeight=9223772",
  "content": "[INFO] [02-17|17:39:26.931] [5/13 Execution] Finish block 9223772 with 30 transactions... info-tree-index=54491",
  "content": "[INFO] [02-17|17:39:27.027] [5/13 Execution] Starting verification request batch-number=157633 blocks-range=[9223772;9223772] pending-requests=2",
  "content": "[INFO] [02-17|17:39:27.038] RPC Daemon notified of new headers       from=9223771 to=9223772 amount=1 hash=0xc84d45d94311a6a4b94f0d3a2ef8ad9aa0770f66cc1063a8acf89aca00eb9af8 header sending=8.21µs log sending=243ns",
  "content": "[INFO] [02-17|17:39:27.092] [5/13 Execution] Increment trie hashes started previousRootHeight=9223772 calculatingRootHeight=9223773",
  "content": "[INFO] [02-17|17:39:27.239] RPC Daemon notified of new headers       from=9223772 to=9223773 amount=1 hash=0x7accaea2f424041954a9526a26772f470983e2c5b8f52d3bac6b02124882ca57 header sending=9.116µs log sending=214ns",
  "content": "[INFO] [02-17|17:39:27.262] Generating witness timing                batch=157633 blockFrom=9223772 blockTo=9223772 taken=233.84784ms",
  "content": "[INFO] [02-17|17:39:32.789] [5/13 Execution] Finished verification request batch-number=157633 blocks-range=[9223772;9223772] is-valid=true pending-requests=12",
  
 "content": "[INFO] [02-17|17:39:27.038] [5/13 Execution] Starting block 9223773 (forkid 13)... ",
  "content": "[INFO] [02-17|17:39:27.092] [5/13 Execution] Increment trie hashes started previousRootHeight=9223772 calculatingRootHeight=9223773",
  "content": "[INFO] [02-17|17:39:27.147] [5/13 Execution] Finish block 9223773 with 30 transactions... info-tree-index=54491",
  "content": "[INFO] [02-17|17:39:27.228] [5/13 Execution] Starting verification request batch-number=157634 blocks-range=[9223773;9223773] pending-requests=3",
  "content": "[INFO] [02-17|17:39:27.239] RPC Daemon notified of new headers       from=9223772 to=9223773 amount=1 hash=0x7accaea2f424041954a9526a26772f470983e2c5b8f52d3bac6b02124882ca57 header sending=9.116µs log sending=214ns",
  "content": "[INFO] [02-17|17:39:27.309] [5/13 Execution] Increment trie hashes started previousRootHeight=9223773 calculatingRootHeight=9223774",
  "content": "[INFO] [02-17|17:39:27.484] RPC Daemon notified of new headers       from=9223773 to=9223774 amount=1 hash=0xf7cb1a0c49dd13d5b2a5216cf877b8e021ab683000c281e466e1df52d8e08b50 header sending=11.964µs log sending=256ns",
  "content": "[INFO] [02-17|17:39:27.521] Generating witness timing                batch=157634 blockFrom=9223773 blockTo=9223773 taken=291.990379ms",
  "content": "[INFO] [02-17|17:39:32.789] [5/13 Execution] Finished verification request batch-number=157634 blocks-range=[9223773;9223773] is-valid=true pending-requests=11",
  
  "content": "[INFO] [02-17|17:39:27.239] [5/13 Execution] Starting block 9223774 (forkid 13)... ",
  "content": "[INFO] [02-17|17:39:27.309] [5/13 Execution] Increment trie hashes started previousRootHeight=9223773 calculatingRootHeight=9223774",
  "content": "[INFO] [02-17|17:39:27.375] [5/13 Execution] Finish block 9223774 with 30 transactions... info-tree-index=54491",
  "content": "[INFO] [02-17|17:39:27.472] [5/13 Execution] Starting verification request batch-number=157635 blocks-range=[9223774;9223774] pending-requests=4",
  "content": "[INFO] [02-17|17:39:27.484] RPC Daemon notified of new headers       from=9223773 to=9223774 amount=1 hash=0xf7cb1a0c49dd13d5b2a5216cf877b8e021ab683000c281e466e1df52d8e08b50 header sending=11.964µs log sending=256ns",
  "content": "[INFO] [02-17|17:39:27.559] [5/13 Execution] Increment trie hashes started previousRootHeight=9223774 calculatingRootHeight=9223775",
  "content": "[INFO] [02-17|17:39:27.694] RPC Daemon notified of new headers       from=9223774 to=9223775 amount=1 hash=0xa7809418dd7e92b66ec900e7d0278bc63d92985ac543e50e374e2aa3e94287e7 header sending=8.282µs log sending=233ns",
  "content": "[INFO] [02-17|17:39:27.700] Generating witness timing                batch=157635 blockFrom=9223774 blockTo=9223774 taken=226.442725ms",
  "content": "[INFO] [02-17|17:39:32.789] [5/13 Execution] Finished verification request batch-number=157635 blocks-range=[9223774;9223774] is-valid=true pending-requests=10",

  "content": "[INFO] [02-17|17:39:27.485] [5/13 Execution] Starting block 9223775 (forkid 13)... ",
  "content": "[INFO] [02-17|17:39:27.559] [5/13 Execution] Increment trie hashes started previousRootHeight=9223774 calculatingRootHeight=9223775",
  "content": "[INFO] [02-17|17:39:27.608] [5/13 Execution] Finish block 9223775 with 30 transactions... info-tree-index=54491",
  "content": "[INFO] [02-17|17:39:27.684] [5/13 Execution] Starting verification request batch-number=157636 blocks-range=[9223775;9223775] pending-requests=5",
  "content": "[INFO] [02-17|17:39:27.694] RPC Daemon notified of new headers       from=9223774 to=9223775 amount=1 hash=0xa7809418dd7e92b66ec900e7d0278bc63d92985ac543e50e374e2aa3e94287e7 header sending=8.282µs log sending=233ns",
  "content": "[INFO] [02-17|17:39:27.909] Generating witness timing                batch=157636 blockFrom=9223775 blockTo=9223775 taken=224.482766ms",
  "content": "[INFO] [02-17|17:39:27.956] [5/13 Execution] Increment trie hashes started previousRootHeight=9223775 calculatingRootHeight=9223776",
  "content": "[INFO] [02-17|17:39:28.097] RPC Daemon notified of new headers       from=9223775 to=9223776 amount=1 hash=0xb9aa1bb82dc1ff579aea68caf8ae21b60f6d2cc8c5f3182a78aad3400936cd21 header sending=11.422µs log sending=244ns",
  "content": "[INFO] [02-17|17:39:32.789] [5/13 Execution] Finished verification request batch-number=157636 blocks-range=[9223775;9223775] is-valid=true pending-requests=9",

  "content": "[INFO] [02-17|17:39:27.695] [5/13 Execution] Starting block 9223776 (forkid 13)... ",
  "content": "[INFO] [02-17|17:39:27.956] [5/13 Execution] Increment trie hashes started previousRootHeight=9223775 calculatingRootHeight=9223776",
  "content": "[INFO] [02-17|17:39:28.010] [5/13 Execution] Finish block 9223776 with 30 transactions... info-tree-index=54491",
  "content": "[INFO] [02-17|17:39:28.086] [5/13 Execution] Starting verification request batch-number=157637 blocks-range=[9223776;9223776] pending-requests=6",
  "content": "[INFO] [02-17|17:39:28.097] RPC Daemon notified of new headers       from=9223775 to=9223776 amount=1 hash=0xb9aa1bb82dc1ff579aea68caf8ae21b60f6d2cc8c5f3182a78aad3400936cd21 header sending=11.422µs log sending=244ns",
  "content": "[INFO] [02-17|17:39:28.154] [5/13 Execution] Increment trie hashes started previousRootHeight=9223776 calculatingRootHeight=9223777",
  "content": "[INFO] [02-17|17:39:28.297] RPC Daemon notified of new headers       from=9223776 to=9223777 amount=1 hash=0x56f457f8729b4969e47c46872a6709f8efc1eb412ecb448fc3facc356ec3647c header sending=9.01µs log sending=238ns",
  "content": "[INFO] [02-17|17:39:28.322] Generating witness timing                batch=157637 blockFrom=9223776 blockTo=9223776 taken=235.255212ms",
  "content": "[INFO] [02-17|17:39:35.860] [5/13 Execution] Finished verification request batch-number=157637 blocks-range=[9223776;9223776] is-valid=true pending-requests=9",

  "content": "[INFO] [02-17|17:39:28.097] [5/13 Execution] Starting block 9223777 (forkid 13)... ",
  "content": "[INFO] [02-17|17:39:28.154] [5/13 Execution] Increment trie hashes started previousRootHeight=9223776 calculatingRootHeight=9223777",
  "content": "[INFO] [02-17|17:39:28.209] [5/13 Execution] Finish block 9223777 with 30 transactions... info-tree-index=54491",
  "content": "[INFO] [02-17|17:39:28.289] [5/13 Execution] Starting verification request batch-number=157638 blocks-range=[9223777;9223777] pending-requests=7",
  "content": "[INFO] [02-17|17:39:28.297] RPC Daemon notified of new headers       from=9223776 to=9223777 amount=1 hash=0x56f457f8729b4969e47c46872a6709f8efc1eb412ecb448fc3facc356ec3647c header sending=9.01µs log sending=238ns",
  "content": "[INFO] [02-17|17:39:28.458] [5/13 Execution] Increment trie hashes started previousRootHeight=9223777 calculatingRootHeight=9223778",
  "content": "[INFO] [02-17|17:39:28.530] Generating witness timing                batch=157638 blockFrom=9223777 blockTo=9223777 taken=240.772759ms",
  "content": "[INFO] [02-17|17:39:28.571] RPC Daemon notified of new headers       from=9223777 to=9223778 amount=1 hash=0x3f9192e056598e3f4a0b84a02a0e4777e48b14ba4e77831f52262ec64f798bf6 header sending=8.929µs log sending=236ns",
  "content": "[INFO] [02-17|17:39:35.860] [5/13 Execution] Finished verification request batch-number=157638 blocks-range=[9223777;9223777] is-valid=true pending-requests=8",

  "content": "[INFO] [02-17|17:39:28.298] [5/13 Execution] Starting block 9223778 (forkid 13)... ",
  "content": "[INFO] [02-17|17:39:28.458] [5/13 Execution] Increment trie hashes started previousRootHeight=9223777 calculatingRootHeight=9223778",
  "content": "[INFO] [02-17|17:39:28.499] [5/13 Execution] Finish block 9223778 with 30 transactions... info-tree-index=54491",
  "content": "[INFO] [02-17|17:39:28.562] [5/13 Execution] Starting verification request batch-number=157639 blocks-range=[9223778;9223778] pending-requests=8",
  "content": "[INFO] [02-17|17:39:28.571] RPC Daemon notified of new headers       from=9223777 to=9223778 amount=1 hash=0x3f9192e056598e3f4a0b84a02a0e4777e48b14ba4e77831f52262ec64f798bf6 header sending=8.929µs log sending=236ns",
  "content": "[INFO] [02-17|17:39:28.727] [5/13 Execution] Increment trie hashes started previousRootHeight=9223778 calculatingRootHeight=9223779",
  "content": "[INFO] [02-17|17:39:28.774] Generating witness timing                batch=157639 blockFrom=9223778 blockTo=9223778 taken=211.893413ms",
  "content": "[INFO] [02-17|17:39:28.859] RPC Daemon notified of new headers       from=9223778 to=9223779 amount=1 hash=0x6c3653adfabb38a55c62ac0ecab35319bb96b34613652611bd35206fc1170cd7 header sending=11.216µs log sending=242ns",
  "content": "[INFO] [02-17|17:39:35.860] [5/13 Execution] Finished verification request batch-number=157639 blocks-range=[9223778;9223778] is-valid=true pending-requests=7",

  "content": "[INFO] [02-17|17:39:28.573] [5/13 Execution] Starting block 9223779 (forkid 13)... ",
  "content": "[INFO] [02-17|17:39:28.727] [5/13 Execution] Increment trie hashes started previousRootHeight=9223778 calculatingRootHeight=9223779",
  "content": "[INFO] [02-17|17:39:28.779] [5/13 Execution] Finish block 9223779 with 30 transactions... info-tree-index=54491",
  "content": "[INFO] [02-17|17:39:28.851] [5/13 Execution] Starting verification request batch-number=157640 blocks-range=[9223779;9223779] pending-requests=9",
  "content": "[INFO] [02-17|17:39:28.859] RPC Daemon notified of new headers       from=9223778 to=9223779 amount=1 hash=0x6c3653adfabb38a55c62ac0ecab35319bb96b34613652611bd35206fc1170cd7 header sending=11.216µs log sending=242ns",
  "content": "[INFO] [02-17|17:39:29.067] Generating witness timing                batch=157640 blockFrom=9223779 blockTo=9223779 taken=215.640447ms",
  "content": "[INFO] [02-17|17:39:29.275] [5/13 Execution] Increment trie hashes started previousRootHeight=9223779 calculatingRootHeight=9223780",
  "content": "[INFO] [02-17|17:39:29.381] RPC Daemon notified of new headers       from=9223779 to=9223780 amount=1 hash=0xf8c424e0316d3249fb1e11c099a2b5391f31a6639f38452f140705bc750ad37d header sending=281.037µs log sending=214ns",
  "content": "[INFO] [02-17|17:39:35.860] [5/13 Execution] Finished verification request batch-number=157640 blocks-range=[9223779;9223779] is-valid=true pending-requests=6",

  "content": "[INFO] [02-17|17:39:28.859] [5/13 Execution] Starting block 9223780 (forkid 13)... ",
  "content": "[INFO] [02-17|17:39:29.275] [5/13 Execution] Increment trie hashes started previousRootHeight=9223779 calculatingRootHeight=9223780",
  "content": "[INFO] [02-17|17:39:29.307] [5/13 Execution] Finish block 9223780 with 15 transactions... info-tree-index=54491",
  "content": "[INFO] [02-17|17:39:29.372] [5/13 Execution] Starting verification request batch-number=157641 blocks-range=[9223780;9223780] pending-requests=10",
  "content": "[INFO] [02-17|17:39:29.381] RPC Daemon notified of new headers       from=9223779 to=9223780 amount=1 hash=0xf8c424e0316d3249fb1e11c099a2b5391f31a6639f38452f140705bc750ad37d header sending=281.037µs log sending=214ns",
  "content": "[INFO] [02-17|17:39:29.524] Generating witness timing                batch=157641 blockFrom=9223780 blockTo=9223780 taken=150.831838ms",
  "content": "[INFO] [02-17|17:39:29.531] [5/13 Execution] Increment trie hashes started previousRootHeight=9223780 calculatingRootHeight=9223781",
  "content": "[INFO] [02-17|17:39:29.596] RPC Daemon notified of new headers       from=9223780 to=9223781 amount=1 hash=0x7a9655032ff9c40225092a012af2399c826cbff4f33df3f12687d41a4aaaedce header sending=7.381µs log sending=239ns",
  "content": "[INFO] [02-17|17:39:35.860] [5/13 Execution] Finished verification request batch-number=157641 blocks-range=[9223780;9223780] is-valid=true pending-requests=5",

  "content": "[INFO] [02-17|17:39:29.382] [5/13 Execution] Starting block 9223781 (forkid 13)... ",
  "content": "[INFO] [02-17|17:39:29.531] [5/13 Execution] Increment trie hashes started previousRootHeight=9223780 calculatingRootHeight=9223781",
  "content": "[INFO] [02-17|17:39:29.554] [5/13 Execution] Finish block 9223781 with 3 transactions... info-tree-index=54491",
  "content": "[INFO] [02-17|17:39:29.588] [5/13 Execution] Starting verification request batch-number=157642 blocks-range=[9223781;9223781] pending-requests=11",
  "content": "[INFO] [02-17|17:39:29.596] RPC Daemon notified of new headers       from=9223780 to=9223781 amount=1 hash=0x7a9655032ff9c40225092a012af2399c826cbff4f33df3f12687d41a4aaaedce header sending=7.381µs log sending=239ns",
  "content": "[INFO] [02-17|17:39:29.622] [5/13 Execution] Increment trie hashes started previousRootHeight=9223781 calculatingRootHeight=9223782",
  "content": "[INFO] [02-17|17:39:29.672] Generating witness timing                batch=157642 blockFrom=9223781 blockTo=9223781 taken=84.496061ms",
  "content": "[INFO] [02-17|17:39:29.685] RPC Daemon notified of new headers       from=9223781 to=9223782 amount=1 hash=0x2e1fb822a756abf116748a274a4ac40397c89140c68b745b56e2b6b15b6fa0a9 header sending=7.715µs log sending=228ns",
  "content": "[INFO] [02-17|17:39:35.860] [5/13 Execution] Finished verification request batch-number=157642 blocks-range=[9223781;9223781] is-valid=true pending-requests=4",
  
 "content": "[INFO] [02-17|17:39:29.596] [5/13 Execution] Starting block 9223782 (forkid 13)... ",
  "content": "[INFO] [02-17|17:39:29.622] [5/13 Execution] Increment trie hashes started previousRootHeight=9223781 calculatingRootHeight=9223782",
  "content": "[INFO] [02-17|17:39:29.645] [5/13 Execution] Finish block 9223782 with 3 transactions... info-tree-index=54491",
  "content": "[INFO] [02-17|17:39:29.677] [5/13 Execution] Starting verification request batch-number=157643 blocks-range=[9223782;9223782] pending-requests=12",
  "content": "[INFO] [02-17|17:39:29.685] RPC Daemon notified of new headers       from=9223781 to=9223782 amount=1 hash=0x2e1fb822a756abf116748a274a4ac40397c89140c68b745b56e2b6b15b6fa0a9 header sending=7.715µs log sending=228ns",
  "content": "[INFO] [02-17|17:39:29.712] [5/13 Execution] Increment trie hashes started previousRootHeight=9223782 calculatingRootHeight=9223783",
  "content": "[INFO] [02-17|17:39:29.759] Generating witness timing                batch=157643 blockFrom=9223782 blockTo=9223782 taken=81.004551ms",
  "content": "[INFO] [02-17|17:39:29.761] RPC Daemon notified of new headers       from=9223782 to=9223783 amount=1 hash=0x1ec99f14a5af7ef88ca2e0bab784d9a3a8222c62da8fef8758261ea14f603962 header sending=9.25µs log sending=233ns",
  "content": "[INFO] [02-17|17:39:38.914] [5/13 Execution] Finished verification request batch-number=157643 blocks-range=[9223782;9223782] is-valid=true pending-requests=4",

  "content": "[INFO] [02-17|17:39:29.685] [5/13 Execution] Starting block 9223783 (forkid 13)... ",
  "content": "[INFO] [02-17|17:39:29.712] [5/13 Execution] Increment trie hashes started previousRootHeight=9223782 calculatingRootHeight=9223783",
  "content": "[INFO] [02-17|17:39:29.727] [5/13 Execution] Finish block 9223783 with 1 transactions... info-tree-index=54491",
  "content": "[INFO] [02-17|17:39:29.754] [5/13 Execution] Starting verification request batch-number=157644 blocks-range=[9223783;9223783] pending-requests=13",
  "content": "[INFO] [02-17|17:39:29.761] RPC Daemon notified of new headers       from=9223782 to=9223783 amount=1 hash=0x1ec99f14a5af7ef88ca2e0bab784d9a3a8222c62da8fef8758261ea14f603962 header sending=9.25µs log sending=233ns",
  "content": "[INFO] [02-17|17:39:29.853] Generating witness timing                batch=157644 blockFrom=9223783 blockTo=9223783 taken=99.185012ms",
  "content": "[INFO] [02-17|17:39:32.766] [5/13 Execution] Increment trie hashes started previousRootHeight=9223783 calculatingRootHeight=9223784",
  "content": "[INFO] [02-17|17:39:32.823] RPC Daemon notified of new headers       from=9223783 to=9223784 amount=1 hash=0xda9a584dd9fe949b3f71cfe9ab772a4c7818018e1d34fdf768058b2a9ddf8e9e header sending=9.779µs log sending=235ns",
  "content": "[INFO] [02-17|17:39:38.914] [5/13 Execution] Finished verification request batch-number=157644 blocks-range=[9223783;9223783] is-valid=true pending-requests=3",

batch 157632 https://www.oklink.com/zh-hans/xlayer/batch/157632

 "content": "[INFO] [02-17|17:39:27.075] Sending request to grpc server           grpcUrl=okx-defi-xlayer-slexe-pro-104798-2.okx-defi-xlayer-slexe-pro:50071 ourRoot=0x9cb34afbd5940eb6832582822ee8e180c512050c1c2d4982caeb92f0edf40d9c oldRoot=0x5248742d54e2ed219089265fb4ddebfeaff2cc9265b93e17257e2bdf24f0b01b batch=157632 witness-size=\"291 kB\" data-stream-size=\"61 kB\" blocks-count=21 correlation=d3bb6984-a7f5-4b26-9db1-74116f8c10a0",
 "content": "[INFO] [02-17|17:39:31.161] executor result                          match=true grpcUrl=okx-defi-xlayer-slexe-pro-104798-2.okx-defi-xlayer-slexe-pro:50071 batch=157632 blocks-count=21 our-counters=\"map[A:420207 B:1431420 D:40294 K:1504 M:44130 P:243946 S:31551635 SHA:150]\" exec-counters=\"map[A:151923 B:703085 D:39162 K:1449 M:26020 P:211277 S:17135298 SHA:150]\" exec-root=0x9cb34afbd5940eb6832582822ee8e180c512050c1c2d4982caeb92f0edf40d9c our-root=0x9cb34afbd5940eb6832582822ee8e180c512050c1c2d4982caeb92f0edf40d9c exec-old-root=0x5248742d54e2ed219089265fb4ddebfeaff2cc9265b93e17257e2bdf24f0b01b our-old-root=0x5248742d54e2ed219089265fb4ddebfeaff2cc9265b93e17257e2bdf24f0b01b correlation=d3bb6984-a7f5-4b26-9db1-74116f8c10a0",

batch 157633
https://www.oklink.com/zh-hans/xlayer/batch/157633
https://www.oklink.com/xlayer/block/9223772

  "content": "[INFO] [02-17|17:39:27.262] Sending request to grpc server           grpcUrl=okx-defi-xlayer-slexe-pro-104798-3.okx-defi-xlayer-slexe-pro:50071 ourRoot=0xa751d3e70e840dc7967d6d4327827f50844b941fd9e9f5e81221b20a4b297b5e oldRoot=0x9cb34afbd5940eb6832582822ee8e180c512050c1c2d4982caeb92f0edf40d9c batch=157633 witness-size=\"250 kB\" data-stream-size=\"56 kB\" blocks-count=1 correlation=c93f3e6d-1d1c-4f61-b967-eb053cf6ede6",
  "content": "[INFO] [02-17|17:39:31.401] executor result                          match=true grpcUrl=okx-defi-xlayer-slexe-pro-104798-3.okx-defi-xlayer-slexe-pro:50071 batch=157633 blocks-count=1 our-counters=\"map[A:418231 B:1427066 D:40050 K:1446 M:44070 P:235952 S:31475972 SHA:150]\" exec-counters=\"map[A:150519 B:700175 D:38940 K:1415 M:26004 P:202468 S:17090421 SHA:150]\" exec-root=0xa751d3e70e840dc7967d6d4327827f50844b941fd9e9f5e81221b20a4b297b5e our-root=0xa751d3e70e840dc7967d6d4327827f50844b941fd9e9f5e81221b20a4b297b5e exec-old-root=0x9cb34afbd5940eb6832582822ee8e180c512050c1c2d4982caeb92f0edf40d9c our-old-root=0x9cb34afbd5940eb6832582822ee8e180c512050c1c2d4982caeb92f0edf40d9c correlation=c93f3e6d-1d1c-4f61-b967-eb053cf6ede6",

[giskook]

After discussing with @praetoriansentry, There's one way to speed up, we can set up another data streamer for the RPC, so that, we can speedup the height.