kona: Check L2 reorg due to L1 reorg

[pcw109550]

Found while attempting to run monorepo acceptance tests targeting kona-node(version op-rs/kona@fe6dfcf)

L1 origin selection is done at https://github.com/op-rs/kona/blob/fe6dfcf771059109f1d75043d5ecbbfa3b6ca1a5/crates/node/service/src/actors/sequencer/actor.rs#L214 while block building at kona.

Kona seems to fail detecting L1 reorg while unsafe block building, and also not resetting(reorging). This case was supposed to be covered at kona's acceptance test at l2_reorg_after_l1_reorgs_test.go, TestL2ReorgAfterL1Reorg test but due to #18675 the test preset was running op-node sequencer instead of kona-node sequencer so this L1 reorg case was not covered.

TestL2ReorgAfterL1Reorg disables derivation by turning off the batcher. Even in this case, L2 reorg must occur.

Manually created temp test at https://github.com/ethereum-optimism/optimism/tree/pcw109550/temp-kona-unsafe-head-l1-reorg. We may compare the logs between targeting op-node or kona-node.

op-node

op-node correctly triggers L2 reorg due to L1 reorg, correcting L1 origin of unsafe heads.

op-node:

cd op-acceptance-tests/tests/sync/unsafe_reorg
go test ./... -run ^TestKona_ReorgRecovery$   -count=1 -v   > t-op-node.log

Example logs:

    sync_test.go:46:            INFO [12-22|19:39:25.137] Target L2 Block to reorg                 l2=c4867a..e28da7:12 l1_origin=a40ad0..b5badb:4
    sync_test.go:52:            INFO [12-22|19:39:57.167] Triggering L1 reorg                      l1=a40ad0..b5badb:4
    sync_test.go:62:            INFO [12-22|19:39:57.174] Triggered L1 reorg                       l1=3793a3..d60e63:4
    sync_test.go:89:            INFO [12-22|19:40:17.178] seq wowow                                bn=88d71d..fe758c:0 l1=210dff..303438:0
    sync_test.go:89:            INFO [12-22|19:40:17.178] seq wowow                                bn=3345a4..45bef4:1 l1=210dff..303438:0
    sync_test.go:89:            INFO [12-22|19:40:17.179] seq wowow                                bn=39bff2..ef0823:2 l1=210dff..303438:0
    sync_test.go:89:            INFO [12-22|19:40:17.180] seq wowow                                bn=f8c2db..7a19dd:3 l1=210dff..303438:0
    sync_test.go:89:            INFO [12-22|19:40:17.180] seq wowow                                bn=524a88..9e405b:4 l1=210dff..303438:0
    sync_test.go:89:            INFO [12-22|19:40:17.181] seq wowow                                bn=8f92d8..13c014:5 l1=210dff..303438:0
    sync_test.go:89:            INFO [12-22|19:40:17.182] seq wowow                                bn=698a4a..bb2b3e:6 l1=210dff..303438:0
    sync_test.go:89:            INFO [12-22|19:40:17.182] seq wowow                                bn=dc22f3..1d42da:7 l1=9f74d0..821f45:1
    sync_test.go:89:            INFO [12-22|19:40:17.183] seq wowow                                bn=7ac768..dcff83:8 l1=26f71e..6fbbc5:2
    sync_test.go:89:            INFO [12-22|19:40:17.184] seq wowow                                bn=cf7626..408ed5:9 l1=7d972b..d8f257:3
    sync_test.go:89:            INFO [12-22|19:40:17.184] seq wowow                                bn=c3a3fc..061536:10 l1=7d972b..d8f257:3
    sync_test.go:89:            INFO [12-22|19:40:17.185] seq wowow                                bn=6ae593..f33173:11 l1=7d972b..d8f257:3
    sync_test.go:89:            INFO [12-22|19:40:17.185] seq wowow                                bn=5e389f..b13379:12 l1=3793a3..d60e63:4
    sync_test.go:89:            INFO [12-22|19:40:17.186] seq wowow                                bn=09de5d..d64aee:13 l1=3793a3..d60e63:4
    sync_test.go:89:            INFO [12-22|19:40:17.186] seq wowow                                bn=5fc4d7..009dfa:14 l1=3793a3..d60e63:4
    sync_test.go:89:            INFO [12-22|19:40:17.187] seq wowow                                bn=45975c..a7b541:15 l1=857df6..50d4d0:5
    sync_test.go:89:            INFO [12-22|19:40:17.187] seq wowow                                bn=580222..603512:16 l1=857df6..50d4d0:5
    sync_test.go:95:            INFO [12-22|19:40:17.198] l1 wowow                                 bn=210dff..303438:0
    sync_test.go:95:            INFO [12-22|19:40:17.198] l1 wowow                                 bn=9f74d0..821f45:1
    sync_test.go:95:            INFO [12-22|19:40:17.198] l1 wowow                                 bn=26f71e..6fbbc5:2
    sync_test.go:95:            INFO [12-22|19:40:17.199] l1 wowow                                 bn=7d972b..d8f257:3
    sync_test.go:95:            INFO [12-22|19:40:17.199] l1 wowow                                 bn=3793a3..d60e63:4
    sync_test.go:95:            INFO [12-22|19:40:17.199] l1 wowow                                 bn=857df6..50d4d0:5

kona-node

op-node fails to detec L1 reorg, never correcting L1 origin of unsafe heads.

kona-node

cd op-acceptance-tests/tests/sync/unsafe_reorg
# build kona-node
DEVSTACK_L2CL_KIND=kona RUST_BINARY_PATH_KONA_NODE=/tmp/kona-node go test ./... -run ^TestKona_ReorgRecovery$   -count=1 -v   > t-kona-node.log
grep -E "wow|Target L2 Block to reorg|Triggering L1 reorg|Triggered L1 reorg|Triggered L2 reorg" t-kona-node.log

Example logs:

    sync_test.go:46:            INFO [12-22|19:51:38.227] Target L2 Block to reorg                 l2=485c4f..b8c75b:12 l1_origin=8f07ca..784b1e:4
    sync_test.go:52:            INFO [12-22|19:52:10.309] Triggering L1 reorg                      l1=8f07ca..784b1e:4
    sync_test.go:62:            INFO [12-22|19:52:10.314] Triggered L1 reorg                       l1=88a6d1..2c629d:4
    sync_test.go:89:            INFO [12-22|19:52:30.317] seq wowow                                bn=566965..e73137:0 l1=788267..1cda25:0
    sync_test.go:89:            INFO [12-22|19:52:30.317] seq wowow                                bn=37200c..ec8191:1 l1=788267..1cda25:0
    sync_test.go:89:            INFO [12-22|19:52:30.317] seq wowow                                bn=c5cd36..356326:2 l1=788267..1cda25:0
    sync_test.go:89:            INFO [12-22|19:52:30.318] seq wowow                                bn=f69467..3f6983:3 l1=788267..1cda25:0
    sync_test.go:89:            INFO [12-22|19:52:30.318] seq wowow                                bn=ca91df..41bb5c:4 l1=788267..1cda25:0
    sync_test.go:89:            INFO [12-22|19:52:30.318] seq wowow                                bn=6b604e..309cef:5 l1=788267..1cda25:0
    sync_test.go:89:            INFO [12-22|19:52:30.318] seq wowow                                bn=993038..c3f10e:6 l1=788267..1cda25:0
    sync_test.go:89:            INFO [12-22|19:52:30.319] seq wowow                                bn=fd1c74..c77f7b:7 l1=788267..1cda25:0
    sync_test.go:89:            INFO [12-22|19:52:30.319] seq wowow                                bn=b62ee2..2b6538:8 l1=fbb460..efd863:1
    sync_test.go:89:            INFO [12-22|19:52:30.319] seq wowow                                bn=0943fc..ed02d3:9 l1=a52333..b7862b:2
    sync_test.go:89:            INFO [12-22|19:52:30.319] seq wowow                                bn=353776..5dc355:10 l1=79c6dc..d102d4:3
    sync_test.go:89:            INFO [12-22|19:52:30.320] seq wowow                                bn=bc0d9f..a30718:11 l1=79c6dc..d102d4:3
    sync_test.go:89:            INFO [12-22|19:52:30.320] seq wowow                                bn=485c4f..b8c75b:12 l1=8f07ca..784b1e:4
    sync_test.go:89:            INFO [12-22|19:52:30.320] seq wowow                                bn=a3d6f1..6bbf9d:13 l1=8f07ca..784b1e:4
    sync_test.go:89:            INFO [12-22|19:52:30.320] seq wowow                                bn=55f15a..c4fd4b:14 l1=8f07ca..784b1e:4
    sync_test.go:89:            INFO [12-22|19:52:30.321] seq wowow                                bn=53f4ec..fe06f6:15 l1=9c33a7..2c079c:5
    sync_test.go:95:            INFO [12-22|19:52:30.327] l1 wowow                                 bn=788267..1cda25:0
    sync_test.go:95:            INFO [12-22|19:52:30.327] l1 wowow                                 bn=fbb460..efd863:1
    sync_test.go:95:            INFO [12-22|19:52:30.327] l1 wowow                                 bn=a52333..b7862b:2
    sync_test.go:95:            INFO [12-22|19:52:30.328] l1 wowow                                 bn=79c6dc..d102d4:3
    sync_test.go:95:            INFO [12-22|19:52:30.328] l1 wowow                                 bn=88a6d1..2c629d:4

[pcw109550]

Root Cause Analysis:

Issue 1: L1 Origin Selector caching

https://github.com/op-rs/kona/blob/fe6dfcf771059109f1d75043d5ecbbfa3b6ca1a5/crates/node/service/src/actors/sequencer/origin_selector.rs#L154

    /// Attempts to fetch the next L1 origin block.
    async fn try_fetch_next_origin(&mut self) -> Result<(), L1OriginSelectorError> {
        // If there is no next L1 origin set, attempt to find it. If it's not yet available, leave
        // it unset.
        if let Some(current) = self.current.as_ref() {
            // If the next L1 origin is already set, do nothing.
            if self.next.is_some() {
                return Ok(()); // [!] Never refreshes cached value!
            }

During an L1 reorg, the cached self.next became stale (referenced the old fork), but was never re-validated.

Issue 2: Missing L1 Canonical Validation at L1 Origin

At https://github.com/op-rs/kona/blob/fe6dfcf771059109f1d75043d5ecbbfa3b6ca1a5/crates/node/service/src/actors/sequencer/origin_selector.rs#L57, there is no check of L1 Origin is valid or orphaned.

Comparing with op-node:

optimism/op-node/rollup/sequencing/origin_selector.go

Lines 226 to 243 in 1294a28

	func (los *L1OriginSelector) findL1OriginOfNextL2Block(
	l2Head eth.L2BlockRef,
	currentL1Origin eth.L1BlockRef, nextL1Origin eth.L1BlockRef,
	matchAutoDerivation bool) (eth.L1BlockRef, error) {
	if (currentL1Origin == eth.L1BlockRef{}) {
	// This would indicate a programming error, since the currentL1Origin
	// should _always_ be available.
	// The first value (for block 1) is specified in rollup.Config.Genesis.L1
	// and it is then only updated to non-empty values.
	panic("origin-selector: currentL1Origin is empty")
	}
	if l2Head.L1Origin.Hash != currentL1Origin.Hash {
	return currentL1Origin, ErrInvalidL1Origin
	}
	if (nextL1Origin != eth.L1BlockRef{} && nextL1Origin.ParentHash != currentL1Origin.Hash) {
	return nextL1Origin, ErrNextL1OriginOrphaned
	}

op-node properly handles L1 inconsistency, and returning ErrInvalidL1Origin or ErrNextL1OriginOrphaned. This op-node l1 origin selector improvement was done at #18589.

Issue 3: No proper reset during block building

At op-node, when L1 inconsistency detected during block building, at l1 origin selector, the sequencer actor emits a reset event:

optimism/op-node/rollup/sequencing/sequencer.go

Lines 508 to 514 in 1294a28

	case errors.Is(err, ErrInvalidL1Origin), errors.Is(err, ErrNextL1OriginOrphaned):
	d.metrics.RecordSequencerInconsistentL1Origin(l2Head.L1Origin, l1Origin.ID())
	d.emitter.Emit(d.ctx, rollup.ResetEvent{
	Err: fmt.Errorf("cannot build new L2 block with L1 origin %s (parent L1 %s) on current L2 head %s with L1 origin %s",
	l1Origin, l1Origin.ParentHash, l2Head, l2Head.L1Origin),
	})
	return

This reset event will propagate to op-node origin selector actor:

optimism/op-node/rollup/sequencing/origin_selector.go

Lines 205 to 211 in 1294a28

	func (los *L1OriginSelector) reset() {
	los.mu.Lock()
	defer los.mu.Unlock()
	los.currentOrigin = eth.L1BlockRef{}
	los.nextOrigin = eth.L1BlockRef{}
	}

and wipe out the current l1 cache.

kona node may implement a similar l1 info invalidation logic like op-node when l1 inconsistency detected.

Possible Fixes

1. Strengthen kona's l1 origin selector referring op-node's.
2. Add unit test for origin_selector.rs covering L1 inconsistency cases.
3. Revise reset logic at kona when l1 inconsistency is detected. Currently op-node emits a reset event, but kona does not. Kona may emit something like Err(ResetError::ReorgDetected(block.hash, next_l1_origin.parent_hash).into());, but this error is only emitted during derivation.
4. Enable monorepo's acceptance tests more targeting kona. Currently only jovian https://github.com/op-rs/kona/blob/fe6dfcf771059109f1d75043d5ecbbfa3b6ca1a5/tests/justfile#L24.