Skip to content

Toggle rpc endpoint used on errors (3.8.0 backport) #3837

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Oct 15, 2025
Merged

Toggle rpc endpoint used on errors (3.8.0 backport) #3837

merged 2 commits into from
Oct 15, 2025

Conversation

gligneul
Copy link
Contributor

Backport of #3834

eljobe added 2 commits October 14, 2025 20:51
@eljobe @gligneul
Toggle rpc endpoint used on errors
ba2dedc 
When there is a failure fetching blobs from `blob_sidecars` switch to
`blobs` (and vice-versa.)

This makes sense because a node can start syncing before the fusaka fork, and we
can automatically switch to the new `blobs` endpoint when the legacy endpoint
fails (because it was strongly deprecated and removed in 2 clients.)
@eljobe @gligneul
Also, add a dangerous flag to skip blob proof verification
e1bfed1
KolbyML
KolbyML approved these changes Oct 14, 2025
View reviewed changes
Copy link
Member

@KolbyML KolbyML left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit: looks good

@eljobe eljobe self-assigned this Oct 14, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 15, 2025 •
edited
Loading

❌ 3 Tests Failed:

Tests completed Failed Passed Skipped
2139 3 2136 0
View the top 3 failed tests by shortest run time 
TestVersion30
Stack Traces | 7.200s run time 
... [CONTENT TRUNCATED: Keeping last 20 lines]
�[38;5;48;1myay!! we validated block 27 in 63.98ms�[0;0m
�[38;5;48;1myay!! we validated block 28 in 76.57ms�[0;0m
    precompile_inclusion_test.go:90: goroutine 522201 [running]:
        runtime/debug.Stack()
        	/opt/hostedtoolcache/go/1.25.3/x64/src/runtime/debug/stack.go:26 +0x5e
        github.com/offchainlabs/nitro/util/testhelpers.RequireImpl({0x40813d0, 0xc06c6b01c0}, {0x403f600, 0xc1aa455e00}, {0x0, 0x0, 0x0})
        	/home/runner/work/nitro/nitro/util/testhelpers/testhelpers.go:29 +0x55
        github.com/offchainlabs/nitro/system_tests.Require(0xc06c6b01c0, {0x403f600, 0xc1aa455e00}, {0x0, 0x0, 0x0})
        	/home/runner/work/nitro/nitro/system_tests/common_test.go:1723 +0x5d
        github.com/offchainlabs/nitro/system_tests.testPrecompiles(0xc06c6b01c0, 0x1e, {0xc0c2b07db0, 0x6, 0x4f5b89?})
        	/home/runner/work/nitro/nitro/system_tests/precompile_inclusion_test.go:90 +0x371
        github.com/offchainlabs/nitro/system_tests.TestVersion30(0xc06c6b01c0?)
        	/home/runner/work/nitro/nitro/system_tests/precompile_inclusion_test.go:67 +0x798
        testing.tRunner(0xc06c6b01c0, 0x3cc9a00)
        	/opt/hostedtoolcache/go/1.25.3/x64/src/testing/testing.go:1934 +0xea
        created by testing.(*T).Run in goroutine 1
        	/opt/hostedtoolcache/go/1.25.3/x64/src/testing/testing.go:1997 +0x465
        
    precompile_inclusion_test.go:90: �[31;1m [] execution aborted (timeout = 5s) �[0;0m
--- FAIL: TestVersion30 (7.20s)

TestBlockHashFeedMismatch
Stack Traces | 22.740s run time 
... [CONTENT TRUNCATED: Keeping last 20 lines]
DEBUG[10-15|02:50:47.356] Journaled pathdb diff layer              root=8b1711..236b84 parent=04d46a..660f46 id=15                 block=14
DEBUG[10-15|02:50:47.356] Journaled pathdb diff layer              root=7db149..9324db parent=8b1711..236b84 id=16                 block=15
DEBUG[10-15|02:50:47.356] Journaled pathdb diff layer              root=9d9c90..890da7 parent=7db149..9324db id=17                 block=16
DEBUG[10-15|02:50:47.356] Journaled pathdb diff layer              root=0f85b6..5507f9 parent=9d9c90..890da7 id=18                 block=17
DEBUG[10-15|02:50:47.356] Journaled pathdb diff layer              root=0c0178..ea94e5 parent=0f85b6..5507f9 id=19                 block=18
DEBUG[10-15|02:50:47.356] Journaled pathdb diff layer              root=d090a7..3f8fcf parent=0c0178..ea94e5 id=20                 block=19
DEBUG[10-15|02:50:47.356] Journaled pathdb diff layer              root=c5bc31..e44fbe parent=d090a7..3f8fcf id=21                 block=20
DEBUG[10-15|02:50:47.356] Journaled pathdb diff layer              root=be1140..3185ec parent=c5bc31..e44fbe id=22                 block=21
DEBUG[10-15|02:50:47.356] Journaled pathdb diff layer              root=8e0201..049d91 parent=be1140..3185ec id=23                 block=22
DEBUG[10-15|02:50:47.356] Journaled pathdb diff layer              root=8ab8fc..096c93 parent=8e0201..049d91 id=24                 block=23
DEBUG[10-15|02:50:47.356] Journaled pathdb diff layer              root=88adfe..b97759 parent=8ab8fc..096c93 id=25                 block=24
DEBUG[10-15|02:50:47.357] Journaled pathdb diff layer              root=fca1ef..e3ffd8 parent=88adfe..b97759 id=26                 block=25
DEBUG[10-15|02:50:47.357] Journaled pathdb diff layer              root=f0c21e..8e087f parent=fca1ef..e3ffd8 id=27                 block=26
DEBUG[10-15|02:50:47.357] Journaled pathdb diff layer              root=ce5a5e..beddda parent=f0c21e..8e087f id=28                 block=27
DEBUG[10-15|02:50:47.357] Journaled pathdb diff layer              root=6f3d72..76b158 parent=ce5a5e..beddda id=29                 block=28
DEBUG[10-15|02:50:47.357] Journaled pathdb diff layer              root=005040..c0b80d parent=6f3d72..76b158 id=30                 block=29
INFO [10-15|02:50:47.357] Persisted dirty state to disk            size=138.75KiB elapsed=2.891ms
INFO [10-15|02:50:47.358] Blockchain stopped
TRACE[10-15|02:50:47.358] P2P networking is spinning down
--- FAIL: TestBlockHashFeedMismatch (22.74s)

TestEthSyncing
Stack Traces | 30.770s run time 
... [CONTENT TRUNCATED: Keeping last 20 lines]
ERROR[10-15|03:00:02.664] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=81  messageCount=161 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:02.690] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=82  messageCount=162 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:02.714] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=83  messageCount=163 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:02.737] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=84  messageCount=164 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:02.764] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=85  messageCount=165 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:02.783] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=86  messageCount=166 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:02.809] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=87  messageCount=167 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:02.831] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=88  messageCount=168 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:02.850] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=89  messageCount=169 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:02.862] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=90  messageCount=170 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:02.884] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=91  messageCount=171 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:02.901] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=92  messageCount=172 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:02.914] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=93  messageCount=173 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:02.932] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=94  messageCount=174 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:02.946] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=95  messageCount=175 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:03.126] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=96  messageCount=176 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:03.163] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=97  messageCount=177 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:03.196] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=98  messageCount=178 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:03.222] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=99  messageCount=179 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80
ERROR[10-15|03:00:03.251] a large batch posting backlog exists     recentlyHitL1Bounds=false currentPosition=100 messageCount=180 messagesPerBatch=1 postedMessages=1 unpostedMessages=80 batchBacklogEstimate=80

📣 Thoughts on this report? Let Codecov know! | Powered by Codecov

@eljobe eljobe merged commit 2bfbe14 into v3.8.x-release Oct 15, 2025
29 of 30 checks passed
@eljobe eljobe deleted the gligneul/blob-fix-3.8 branch October 15, 2025 05:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eljobe eljobe eljobe approved these changes

@KolbyML KolbyML KolbyML approved these changes

@hkalodner hkalodner Awaiting requested review from hkalodner

Assignees

@eljobe eljobe

Labels

design-approved

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@gligneul @eljobe @KolbyML