Skip to content

Fix flaky pathdb test failures caused by MEL integration#4576

Open
joshuacolvin0 wants to merge 3 commits intomasterfrom
fix-pathdb-flaky-tests
Open

Fix flaky pathdb test failures caused by MEL integration#4576
joshuacolvin0 wants to merge 3 commits intomasterfrom
fix-pathdb-flaky-tests

Conversation

@joshuacolvin0
Copy link
Copy Markdown
Member

Fix nil InboxTracker panic in TestBatchPosterL1SurplusMatchesBatchGasFlaky
by checking MessageExtractor first, matching the pattern used elsewhere.

Fix TestMessageExtractionLayer_TxStreamerHandleReorg timeout by replacing
log message detection with direct balance verification. InitTestLog sets
a global logger, so parallel tests clobber each other's log handlers,
causing the test to miss expected log messages and time out.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 27, 2026

Codecov Report

❌ Patch coverage is 22.58065% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 34.52%. Comparing base (5b41808) to head (d6b48be).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4576      +/-   ##
==========================================
- Coverage   34.52%   34.52%   -0.01%     
==========================================
  Files         498      498              
  Lines       59092    59100       +8     
==========================================
+ Hits        20400    20402       +2     
- Misses      35067    35071       +4     
- Partials     3625     3627       +2     

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 27, 2026

❌ 10 Tests Failed:

Tests completed Failed Passed Skipped
4700 10 4690 0
View the top 3 failed tests by shortest run time
TestPruningDBSizeReduction
Stack Traces | 0.000s run time
=== RUN   TestPruningDBSizeReduction
--- FAIL: TestPruningDBSizeReduction (0.00s)
TestParentChainEthConfigForkTransition
Stack Traces | 3.670s run time
... [CONTENT TRUNCATED: Keeping last 20 lines]
INFO [03-30|17:10:29.038] Starting work on payload                 id=0x038d6dd3a3aee288
INFO [03-30|17:10:29.038] Ethereum protocol stopped
INFO [03-30|17:10:29.038] Transaction pool stopped
INFO [03-30|17:10:29.038] Persisting dirty state                   head=32  root=525ab0..830e2c layers=32
INFO [03-30|17:10:29.038] Updated payload                          id=0x03a2bda03ff0daeb number=11   hash=054161..da23b7 txs=1  withdrawals=0 gas=730,968    fees=7.30968e-07    root=2eed45..efe30f elapsed="674.117µs"
INFO [03-30|17:10:29.039] DataPoster sent transaction              nonce=4    hash=27e51d..c62983 feeCap=50,024,391,870 tipCap=5,000,000,000 blobFeeCap=<nil> gas=155,134
INFO [03-30|17:10:29.039] BatchPoster: batch sent                  sequenceNumber=5   from=10   to=13   prevDelayed=2   currentDelayed=5   totalSegments=3  numBlobs=0
INFO [03-30|17:10:29.038] Chain head was updated                   number=17   hash=c1ab74..7b1f74 root=928fa7..7823e5 elapsed="269.983µs"
INFO [03-30|17:10:29.039] Stopping work on payload                 id=0x03a2bda03ff0daeb reason=delivery
INFO [03-30|17:10:29.039] Updated payload                          id=0x038d6dd3a3aee288 number=85   hash=4d2ab8..79460b txs=1  withdrawals=0 gas=142,833    fees=0.000714165    root=768688..130f83 elapsed=1.457ms
INFO [03-30|17:10:29.040] Imported new potential chain segment     number=11   hash=054161..da23b7 blocks=1  txs=1  mgas=0.731  elapsed=1.120ms      mgasps=652.090  triediffs=35.10KiB   triedirty=0.00B
INFO [03-30|17:10:29.040] Submitted transaction                    hash=0xe93909223674146e6c24255fb8e38e28bb6edcf17d35469bb99fed048b0f72e1 from=0xaF24Ca6c2831f4d4F629418b50C227DF0885613A nonce=42   recipient=0xaF24Ca6c2831f4d4F629418b50C227DF0885613A value=1,000,000,000,000
INFO [03-30|17:10:29.040] Chain head was updated                   number=11   hash=054161..da23b7 root=2eed45..efe30f elapsed="217.224µs"
INFO [03-30|17:10:29.040] Stopping work on payload                 id=0x038d6dd3a3aee288 reason=delivery
INFO [03-30|17:10:29.041] Persisted dirty state to disk            size=149.21KiB elapsed=2.301ms
INFO [03-30|17:10:29.041] Blockchain stopped
INFO [03-30|17:10:29.041] Starting work on payload                 id=0x0361c260aff9f27e
INFO [03-30|17:10:29.041] Imported new potential chain segment     number=85   hash=4d2ab8..79460b blocks=1  txs=1  mgas=0.143  elapsed=2.008ms      mgasps=71.122   triediffs=385.52KiB  triedirty=0.00B
INFO [03-30|17:10:29.042] Updated payload                          id=0x0361c260aff9f27e number=12   hash=be3e36..35762e txs=0  withdrawals=0 gas=0          fees=0              root=45da5a..ad2485 elapsed="290.261µs"
--- FAIL: TestParentChainEthConfigForkTransition (3.67s)
TestInboxReaderBlobFailureWithDelayedMessage
Stack Traces | 11.080s run time
... [CONTENT TRUNCATED: Keeping last 20 lines]
WARN [03-30|17:11:50.696] P2P server will be useless, neither dialing nor listening
INFO [03-30|17:11:50.699] Submitted transaction                    hash=0x9231f5ebd1342d999cc4da78fef150cd877849578be5578bb3c44729b11c46cf from=0x57Ff0F473737a1c161bfF9efDF016F7991585088 nonce=25  recipient=0x4C4Ac1257945782bf52A724e75687d9010C770DE value=0
WARN [03-30|17:11:50.699] Getting file info                        dir= error="stat : no such file or directory"
--- FAIL: TestInboxReaderBlobFailureWithDelayedMessage (11.08s)
INFO [03-30|17:11:51.102] Chain head was updated                   number=6   hash=f80e66..3c8fee root=e7af0d..d419f9 elapsed="294.128µs"
INFO [03-30|17:11:51.101] Submitted transaction                    hash=0xb056e310f7f88ab7036660633eb2baca7132f685ff79b82e06d8909d9b3e8fda from=0x26E554a8acF9003b83495c7f45F06edCB803d4e3 nonce=128 recipient=0x0C709F340F0BB2e361229e345B7e26999d0969Ab value=340,282,366,920,938,463,463,374,607,431,768,211,456
INFO [03-30|17:11:51.103] Stopping work on payload                 id=0x033c042a37169def reason=delivery
WARN [03-30|17:11:51.103] Sequencer ReadFromTxQueueTimeout is higher than MaxBlockSpeed ReadFromTxQueueTimeout=1s MaxBlockSpeed=10ms
WARN [03-30|17:11:51.104] Sanitizing invalid node buffer size      provided=1024.00MiB updated=256.00MiB
INFO [03-30|17:11:51.104] Load database journal from disk
INFO [03-30|17:11:51.104] Imported new potential chain segment     number=471 hash=c67956..10df25 blocks=1  txs=1   mgas=0.021  elapsed=17.571ms     mgasps=1.195    triediffs=653.64KiB triedirty=233.01KiB
INFO [03-30|17:11:51.105] Chain head was updated                   number=471 hash=c67956..10df25 root=4ce292..d70f6d elapsed="76.152µs"
INFO [03-30|17:11:51.105] State snapshot generator is not found
INFO [03-30|17:11:51.105] Starting snapshot generation             root=56e81f..63b421 accounts=0 slots=0 storage=0.00B dangling=0 elapsed="3.577µs"
INFO [03-30|17:11:51.105] Initialized path database                triecache=600.00MiB statecache=400.00MiB buffer=256.00MiB state-history="last 345600 blocks"
INFO [03-30|17:11:51.105] Deploying challenge factory contracts
INFO [03-30|17:11:51.106] Deploying osp0
INFO [03-30|17:11:51.106] Submitted transaction                    hash=0xdcf0d300cf9c8eda4f9b139d5b35335eabeb62afd2424b07018042d3f36c69c2 from=0xaF24Ca6c2831f4d4F629418b50C227DF0885613A nonce=82  recipient=0xaF24Ca6c2831f4d4F629418b50C227DF0885613A value=1,000,000,000,000
INFO [03-30|17:11:51.106] addresss table import complete
INFO [03-30|17:11:51.106] retryables import complete

📣 Thoughts on this report? Let Codecov know! | Powered by Codecov

Fixes flaky test failures caused by the Message Extraction Layer (MEL)
integration by introducing a BatchDataReader interface that abstracts
over MEL vs InboxTracker, and adding proper delayed message rollback
on accumulator mismatch in addMessages.

Key changes:
- Add BatchDataReader interface and BatchDataSource() to eliminate
  duplicated MEL-vs-InboxTracker dispatch across system tests
- Roll back delayed messages when AddSequencerBatches detects a
  mismatch, preventing orphaned DB entries
- Wrap both rollback and original mismatch errors in the error chain
- Replace unbounded polling loops in tests with deadline-guarded waits
- Fix pre-existing diagnostic bug in meaningless_reorg_test.go

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@eljobe
Copy link
Copy Markdown
Member

eljobe commented Mar 30, 2026

Assigned back to @joshuacolvin0 to clean up the lint issues and investigate the still-failing test.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
# Conflicts:
#	system_tests/common_test.go
@ganeshvanahalli
Copy link
Copy Markdown
Contributor

Awesome, thanks for this PR @joshuacolvin0, I would like to hold this PR off a little bit, if its not urgent, I'm working currently to replace txStreamer taking in both inboxtracker and messageExtractor to just taking in an unified interface similar to your BatchDataReader interface.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants