PoC to allow indexing unsealed finalized execution results

[peterargue]

Currently, EVM Gateways always use event streaming for the latest events, which are always for sealed blocks. There is work starting soon to add support for streaming data from unsealed finalized blocks, but that is still several months away. In the mean time, we'd like to create a gateway PoC that demonstrates indexing EVM execution data soon after execution (soft-finality).

It turns out this is possible today using the polling version of the Access API, specifically the GetEventsForBlockIDs. The general flow would be:

1. Subscribe to new finalized blocks using the SubscribeBlockHeaders* endpoints.
2. For each new finalized block, call GetEventsForBlockIDs using the block's ID for each of the EVM event types
3. Index the events normally

Notes:

• It's possible for an unsealed execution result to be incorrect and abandoned (execution fork). This is caused by a bug and is very rare, but there needs to be a way to handle it when it happens.
• There currently isn't an easy way for the gateway to know if it got an incorrect result. This would require changes to the Access API, which we're not likely going to do for a PoC. Since there is a proper solution coming to ANs soon, we will accept this limitation and rely on operators to detect and handle these forks manually.
• There is already a way to manually rollback a gateway to a past block that we can use. We will need to ensure this also works for the new state index as well.

[bluesign]

Why not add this to AN with SubscribeEventsByBlockHeight with a flag like sealed=false etc?

or maybe just a POC AN proxy, encapsulating this logic, instead of making it part of EVM GW, just emulating SubscribeEventsByBlockHeight with finalized blocks. ( an relaying anything else to AN )

[peterargue]

Why not add this to AN with SubscribeEventsByBlockHeight with a flag like sealed=false etc?

or maybe just a POC AN proxy, encapsulating this logic, instead of making it part of EVM GW, just emulating SubscribeEventsByBlockHeight with finalized blocks. ( an relaying anything else to AN )

That's a good point. A field like sealed=false is probably what we'd end up adding anyway. I'll look into this and post back

[m-Peter]

@peterargue I started working on a PoC for the flow you described, regarding the available APIs (#728).
I got it working, but I was receiving some strange errors for testnet & mainnet.

For testnet, I was consistently getting this error:

{
    "level": "fatal",
    "version": "development",
    "component": "bootstrap-ingestion",
    "engine": "*ingestion.Engine",
    "error": "failure in event subscription with: recoverable: disconnected: error receiving : rpc error: code = DeadlineExceeded desc = stream encountered an error: context deadline exceeded",
    "time": "2025-01-21T12:33:45+02:00",
    "message": "engine failed to run"
}

every ~550 blocks or so. Started indexing at height 211176670 on spork access-001.devnet51.nodes.onflow.org:9000 .

For mainnet, I was consistently getting this error:

{
    "level": "fatal",
    "version": "development",
    "component": "bootstrap-ingestion",
    "engine": "*ingestion.Engine",
    "error": "failure in event subscription with: recoverable: disconnected: error receiving : rpc error: code = Internal desc = unexpected EOF",
    "time": "2025-01-21T13:10:54+02:00",
    "message": "engine failed to run"
}

every ~350 blocks or so. Started indexing at height 85981135 on spork access-001.mainnet25.nodes.onflow.org:9000.

To work around this, I added a retry upon receiving the above 2 errors (see here: https://github.com/onflow/flow-evm-gateway/pull/728/files#diff-a827adfab89b0bf3476547f91d16457d696bd3a32446a9010275de2d78062a5aR219-R239).
Would you have any idea as to why those errors appear in the first place? 🤔

[peterargue]

I looked into this. The issue was with our side-car proxy config. It was setup to timout connections for this grpc endpoint after 120s. That was originally added when all of the endpoints were polling. We'll relax it now that the endpoint also supports streaming