Skip to content

Conversation

@Tristan-Wilson
Copy link
Member

When running multiple sequencers in HA mode, all nodes run an ExpressLaneTracker that polls for auction resolution events. This means the "AuctionResolved" and "New express lane auction round" log messages appear on every sequencer node, making it difficult for operators to distinguish which node is active vs standby.

Add isActiveSequencer field to these log messages so operators can identify whether the logging node is the active sequencer (sequencing transactions) or an inactive one (validating and forwarding, ready for failover).

The active status is determined by checking if the sequencer's forwarder is nil - active sequencers don't have a forwarder, while inactive ones forward to the active sequencer.

When running multiple sequencers in HA mode, all nodes run an
ExpressLaneTracker that polls for auction resolution events. This means
the "AuctionResolved" and "New express lane auction round" log messages
appear on every sequencer node, making it difficult for operators to
distinguish which node is active vs standby.

Add isActiveSequencer field to these log messages so operators can
identify whether the logging node is the active sequencer (sequencing
transactions) or an inactive one (validating and forwarding, ready for
failover).

The active status is determined by checking if the sequencer's forwarder
is nil - active sequencers don't have a forwarder, while inactive ones
forward to the active sequencer.
@github-actions
Copy link

github-actions bot commented Dec 12, 2025

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
2185 2 2183 0
View the top 2 failed tests by shortest run time
TestAuctioneerRecoversBidsOnRestart
Stack Traces | 17.090s run time
... [CONTENT TRUNCATED: Keeping last 20 lines]
ERROR[12-12|12:18:40.772] Context closed, autonomous auctioneer shutting down
INFO [12-12|12:18:40.772] Context done while waiting redis streams to be ready, failed to start
--- FAIL: TestAuctioneerRecoversBidsOnRestart (17.09s)
INFO [12-12|12:18:40.774] Submitted transaction                    hash=0xd8c1a676c779129fd780d2bd8d6609c99d24b12f3b2ffe08a2bcaa80ab064813 from=0xD4570383A3aE2e9fE20148B43085E76a73cE2Fa2 nonce=0  recipient=0x5886857945f2d2aB8A8d5e3B445b1AE344cb12BA value=0
INFO [12-12|12:18:40.781] Submitted transaction                    hash=0xd7867a972a53e3124419c3fd2041b981ca9bcfc851d5c2dd8cb2e883d963654e from=0x3e8A23a46d28E626617d801C55EB1f5EAA667460 nonce=0  recipient=0xfd10FEaF051e3101DDa78f1528429755Fb82dE07 value=0
INFO [12-12|12:18:41.711] Starting work on payload                 id=0x03ec285da568c66c
INFO [12-12|12:18:41.711] Updated payload                          id=0x03ec285da568c66c                      number=18 hash=4b8e72..83adeb txs=1 withdrawals=0 gas=46478     fees=4.6478e-08  root=7e8ebb..526e4d elapsed="290.932µs"
INFO [12-12|12:18:41.711] Stopping work on payload                 id=0x03ec285da568c66c                      reason=delivery
INFO [12-12|12:18:41.711] Imported new potential chain segment     number=18 hash=4b8e72..83adeb blocks=1 txs=1 mgas=0.046 elapsed="455.33µs"  mgasps=102.075  triediffs=45.05KiB triedirty=0.00B
INFO [12-12|12:18:41.712] Chain head was updated                   number=18 hash=4b8e72..83adeb root=7e8ebb..526e4d elapsed="41.267µs"
INFO [12-12|12:18:41.712] Starting work on payload                 id=0x03a3d1507aff9bb7
INFO [12-12|12:18:41.712] Updated payload                          id=0x03a3d1507aff9bb7                      number=18 hash=d516fe..993d0a txs=0 withdrawals=0 gas=0         fees=0           root=7a0b4f..0d6c45 elapsed="43.702µs"
INFO [12-12|12:18:41.712] Starting work on payload                 id=0x030db1ee916ad4ba
INFO [12-12|12:18:41.712] Stopping work on payload                 id=0x03a3d1507aff9bb7                      reason=delivery
INFO [12-12|12:18:41.712] Imported new potential chain segment     number=18 hash=d516fe..993d0a blocks=1 txs=0 mgas=0.000 elapsed="102.211µs" mgasps=0.000    triediffs=38.10KiB triedirty=0.00B
INFO [12-12|12:18:41.712] Chain head was updated                   number=18 hash=d516fe..993d0a root=7a0b4f..0d6c45 elapsed="19.737µs"
INFO [12-12|12:18:41.712] Updated payload                          id=0x030db1ee916ad4ba                      number=18 hash=5980f8..1e1fd0 txs=1 withdrawals=0 gas=46490     fees=4.649e-08   root=d7c348..c48182 elapsed="271.938µs"
INFO [12-12|12:18:41.713] Stopping work on payload                 id=0x030db1ee916ad4ba                      reason=delivery
INFO [12-12|12:18:41.713] Imported new potential chain segment     number=18 hash=5980f8..1e1fd0 blocks=1 txs=1 mgas=0.046 elapsed="419.754µs" mgasps=110.755  triediffs=43.44KiB triedirty=0.00B
INFO [12-12|12:18:41.713] Chain head was updated                   number=18 hash=5980f8..1e1fd0 root=d7c348..c48182 elapsed="29.465µs"
TestSequencerPriceAdjustsFrom25Gwei
Stack Traces | 55.210s run time
... [CONTENT TRUNCATED: Keeping last 20 lines]
�[90mposted new batch 65�[0;0m
�[34;1mbytes 0 185�[0;0m
�[34;1mInitial values�[0;0m
�[34;1m    L1 base fee 1903561437�[0;0m
�[34;1m    L1 estimate 10000000000�[0;0m
�[90mposted new batch 61�[0;0m
    fees_test.go:224: �[31;1m [L1 gas price estimate should tend toward the basefee] �[0;0m
ERROR[12-12|12:29:26.687] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=50 messageCount=134 messagesPerBatch=1 postedMessages=1 unpostedMessages=84  batchBacklogEstimate=84
ERROR[12-12|12:29:26.744] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=51 messageCount=135 messagesPerBatch=1 postedMessages=1 unpostedMessages=84  batchBacklogEstimate=84
ERROR[12-12|12:29:26.755] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=43 messageCount=126 messagesPerBatch=1 postedMessages=1 unpostedMessages=83  batchBacklogEstimate=83
ERROR[12-12|12:29:26.781] Dangling trie nodes after full cleanup
ERROR[12-12|12:29:26.790] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=52 messageCount=136 messagesPerBatch=1 postedMessages=1 unpostedMessages=84  batchBacklogEstimate=84
ERROR[12-12|12:29:26.808] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=44 messageCount=129 messagesPerBatch=1 postedMessages=1 unpostedMessages=85  batchBacklogEstimate=85
ERROR[12-12|12:29:26.852] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=45 messageCount=129 messagesPerBatch=1 postedMessages=1 unpostedMessages=84  batchBacklogEstimate=84
ERROR[12-12|12:29:26.869] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=53 messageCount=136 messagesPerBatch=1 postedMessages=1 unpostedMessages=83  batchBacklogEstimate=83
ERROR[12-12|12:29:26.997] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=46 messageCount=130 messagesPerBatch=1 postedMessages=1 unpostedMessages=84  batchBacklogEstimate=84
ERROR[12-12|12:29:27.003] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=54 messageCount=138 messagesPerBatch=1 postedMessages=1 unpostedMessages=84  batchBacklogEstimate=84
--- FAIL: TestSequencerPriceAdjustsFrom25Gwei (55.21s)
ERROR[12-12|12:29:27.054] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=55 messageCount=139 messagesPerBatch=1 postedMessages=1 unpostedMessages=84  batchBacklogEstimate=84
ERROR[12-12|12:29:27.079] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=47 messageCount=131 messagesPerBatch=1 postedMessages=1 unpostedMessages=84  batchBacklogEstimate=84

📣 Thoughts on this report? Let Codecov know! | Powered by Codecov

var isActiveFunc func() bool
if n.Sequencer != nil {
isActiveFunc = func() bool {
_, forwarder := n.Sequencer.GetPauseAndForwarder()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to double-check, should the pause be handled here in the same way as in Sequencer.getForwarder?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants