Burst of EVM Transactions fail with "transaction with the same nonce already submitted" error

[vishalchangrani]

Instructions

DL reported on 25th July that all PYUSD0 transfers on Flow EVM were failing.

Problem

DFNS shared this explanation:

We constructed and broadcasted a transaction for EOA 0x66Bda5aEa2c549a35eB56e796c4ced3ef25C6296 with nonce 21712. And everything went through fine. Here are the bytes for the fully signed transaction. 0x02f901978202eb8254d08503bbf075808503bbf075828301785c94f707b3efca33b0f0cec24f805d8937dd1017f64880b9012474fa41210000000000000000000000000000000000000000000000000000000000000060c3be9b9a0d5d009a29d9b8ddf3fc8d8f9852f9db1b38d4119f74c8f25dee1f93cdb7cb536e1367a6458d9ba8ce473e8e3918df915de5bb32de9a645ba5dbf12c000000000000000000000000000000000000000000000000000000000000009899af3eea856556646c98c8b9b2548fe81524075000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000044a9059cbb000000000000000000000000b69bc8c48fa472020120e43a82b62b90c6a351b800000000000000000000000000000000000000000000000000000000000382700000000000000000c001a05451f222665e340ec532967b2895703e28a1eb89b2c48bd7714a55bfd78bc11ba0124c0d349a7442d7820d4e48333028469aedddb33a528cd5921a8c8c292ac4b2
After a while, we don’t see this transaction included on chain, so our rebroadcaster tried to broadcast the same signed transaction again. And that’s when we got the new error back from the eve gateway node. transaction with the same nonce already submitted.
This part makes sense. And correct me if I’m wrong. That means the original transaction we broadcasted is no longer in the gateway’s mempool, it’s been packaged up in a Cadence transaction and sent to main chain, but hasn’t been executed yet.
But what’s unclear to me is why is it taking so long and still not executed. The first rebroadcast is about 2-3 minutes after the transaction was initially broadcasted. I would expect this should be picked up already.

The bug in our system is that we don’t handle this new error correctly. Our current logic is that we treat this error as a terminating error, so we marked the signed transaction as failed and released the nonce back to be reused by another transaction.
Multiple new transactions were constructed after (about a dozen or so), and after attempting to broadcast each one, we kept getting back the error transaction with the same nonce already submitted, so each of those is marked as failed in our system as well.
Again the error makes sense, the original is still stuck on the Cadence side, so any subsequent attempts will keep failing. Question again is what’s causing that transaction to not be executable?

Eventually, after a while, (based on our logs, took about 10 minutes), we finally constructed a new transaction with the same nonce, was able to successfully broadcast again. Here’s the full bytes for the final transaction that succeeded. 0x02f901978202eb8254d08503bbf075808503bbf075828301785c9469eb4e8a33cef058c2b34234b15315c0572aa9bc80b9012474fa41210000000000000000000000000000000000000000000000000000000000000060ff3ef57d5c3213a4bfc0c237959f6ebcc74d1d6c8c917a37497c7eb33da5810d547c2cfc5bd9ea2b42e262aca44112f362d722880c723af7cb615ab0e86a325f000000000000000000000000000000000000000000000000000000000000009899af3eea856556646c98c8b9b2548fe81524075000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000044a9059cbb000000000000000000000000cd0dbe1de00057e56444c478c3c89f4604ca158b00000000000000000000000000000000000000000000000000000000009896800000000000000000c080a07de55acad5866af89946649cf6d7393e4f035f05975abc16c376b2899f58c21ba0237ce130545f651aad4ca42824d1d9bc6355e503a334639ec9008be66d19ba69, and here’s the transaction from the explorer. https://evm.flow.com/tx/0xbb2eee17f9618aa6e759e40ac2bf637105ba4ca210271791fb347a73dae22496.
What this tells us is that after 10 minutes, Cadence dropped the original from execution. But why was the original transaction not executable?
Comparing the later one that succeeded, the fee amount is the same, I don’t see any clue as to why 1 transaction works but the other one doesn’t.
Not clear on how we should handle such a situation. While the original transaction is in Cadence, trying speeding up the transaction doesn’t work, we get the same error back. Rebroadcasting the same transaction doesn’t work. Reusing the nonce for a new transaction doesn’t work. all other transactions from the same EOA with higher nonces are stuck.

Steps to Reproduce

Acceptance Criteria

Context

[vishalchangrani]

TL;DR: With the new flow-tx-id mempool logging shipped in PR #984, we caught a fresh DFNS wedge in the act — two Cadence wrappers for consecutive EVM nonces (25881, 25882) landed in the same Flow block, executed out of order, the higher-nonce wrapper reverted with nonce too high, and the gateway mempool locked the EOA for exactly the 60s idle-eviction window because it never learned the wrapper had reverted.

The incident, verifiable

• DFNS reported ErrInFlightNonce at 2026-07-29T03:10:16.635Z.
• Wedged EVM EOA: 0xdEA93da411CFB1f7d676BF81A686ED36C1d358B3.
• Stuck nonce: 25882.

The mechanism, with on-chain evidence

Both of the following Cadence transactions landed in the same Flow block 159604703:

Cadence tx 0bdb52a62e8b10aee36eb21c20c36db6572f089c95a48e3e0b3e87f7bd38a8de (wrapping EVM nonce 25881):

• Submitted at 2026-07-29T03:09:09Z
• Proposal Key: Index 219, Sequence 1969
• Status: SEALED at block 159604703
• Verify: flow transactions get 0bdb52a62e8b10aee36eb21c20c36db6572f089c95a48e3e0b3e87f7bd38a8de --network mainnet

Cadence tx b946d5c3ea1a80bc165d26824c7504b28dd84c930f182cd048fef52d5c2ca16d (wrapping EVM nonce 25882):

• Submitted at 2026-07-29T03:09:11Z (2s after the 25881 wrapper)
• Proposal Key: Index 221, Sequence 1970
• Status: FAILED at block 159604703 with the following runtime error:

[Error Code: 1101] cadence runtime error: Execution failed:
error: assertion failed: evm_error=nonce too high:
  address 0xdEA93da411CFB1f7d676BF81A686ED36C1d358B3,
  tx: 25882 state: 25881; evm_error_code=202
  --> b946d5c3ea1a80bc165d26824c7504b28dd84c930f182cd048fef52d5c2ca16d:17:12

• Verify: flow transactions get b946d5c3ea1a80bc165d26824c7504b28dd84c930f182cd048fef52d5c2ca16d --network mainnet

Within block 159604703, the 25882 wrapper was executed BEFORE the 25881 wrapper. When EVM processed the 25882 wrapper, on-chain state was still at 25881, so EVM rejected it as "nonce too high". The assertion in run.cdc failed → Cadence tx REVERTED → no EVM.TransactionExecuted event was emitted for nonce 25882.

The EVM tx hash carried inside the failed wrapper is 0x21f46e616b25350a0e1c00a659bf716a89203d1d9c086cfab052ffeb4cf0aaf6. Confirm it never appeared on chain: eth_getTransactionByHash returns null for it.

Why this wedges the gateway

The mempool sets lastConsecutivelySubmitted = 25882 the moment SendTransaction returns nil (success from the AN's perspective — before execution). It is event-driven off EVM.TransactionExecuted. Because the 25882 wrapper reverted before that event fires, the mempool never learns the EVM tx didn't execute. Its in-memory state now believes nonce 25882 was handled, and it rejects every DFNS retry with ErrInFlightNonce — because classify() at services/requester/tx_mempool.go:333 short-circuits on highestSent().atLeast(nonce) before ever reading chain state.

How recovery actually happened

At 03:10:16 — 65 seconds after the failed wrapper — DFNS's next attempt at nonce 25882 was accepted. This precisely matches idleQueueRetention = time.Minute (services/requester/tx_mempool.go:194): after ~60s with no successful Add, the empty EOA queue was evicted, lastConsecutivelySubmitted was cleared, and the new signed tx (a different signed payload with the same nonce, EVM hash 0x4a54bdbd0dfca5e1939987712fa22162de9cf0dcf85ccde0a027c9707d58629d) was accepted. Its wrapping Cadence tx 3e646046aaaf3a6274d5b5978eee99d0bb038ad529a27072240ca385e69976d9 sealed successfully at block 159604789 (~7s after submission — normal timing). The retry EVM tx is on chain: eth_getTransactionByHash(0x4a54bdbd…) returns a valid receipt.

Why two Cadence txs 2s apart landed in the same block

Flow's 0.8s block time is main-chain block time, not collection cadence. Transactions flow through collector clusters first; each cluster produces collections at its own ~1–2s cadence. Cadence txs are cluster-routed by the payer address, so both wrappers (same payer, foundation COA 0xfa9fba4cf025057f) went to the same cluster and were batched into a single collection, then a single main-chain block. TxSubmissionSpacing = 1200ms in the mempool guarantees send-time spacing but not block-inclusion spacing — the comment at services/requester/tx_mempool.go:454-459 claims "two Flow transactions land in different blocks and cannot be reordered by Collection Nodes", and that claim is incorrect when both txs share a cluster.

Once two Cadence txs land in the same collection, execution order within the resulting block is determined by the cluster (not by AN receive time). The two wrappers also used different Flow proposal keys (indices 219 and 221), so Flow's sequence-number ordering (which is enforced per key, not across keys) provided no cross-tx ordering guarantee.

Hypotheses this rules out

For anyone following along:

• Not a silent drop after AN ACK — the wrapper was included on-chain, it just reverted at execution.
• Not a single-KMS-key issue — the COA has many account keys matching the KMS signer public key (indices 219, 221, 262 all used within a few minutes).
• Not the 10-min reference-block expiry recovering the tx — the 10-minute figure from the original DFNS report was almost certainly a gateway restart or session rotation, coincidentally similar to the reference-block validity window. Today's wedge recovered at exactly the 60s idle-eviction boundary.

The fix

The reconciliation-loop direction we already had is correct, and the new evidence sharpens exactly what it needs to do. Rather than only checking GetNextNonce, the loop should poll GetTransactionResult(flowTxID) for each EOA's most recently-submitted wrapper — the flow-tx-id we now log makes this a single AN call. If the wrapper is SEALED with a non-zero ErrorCode (i.e. reverted), roll lastConsecutivelySubmitted back to the on-chain nextNonce and log the Cadence error. That catches this exact scenario within one sealing window (~6–8s) instead of 60s.

Two related preventive avenues (open questions, not proposed for immediate implementation):

• Sticky per-EVM-EOA proposal key assignment on the gateway COA, so consecutive wrappers use the same Flow key and sequence-number ordering guarantees execution order.
• Fix the misleading comment at tx_mempool.go:454-459 regardless — the "two Flow transactions land in different blocks" claim is not true and led us down the wrong hypothesis path initially.

Related work

• PR feat(requester): log wrapping Cadence tx ID on mempool submissions #984 — flow-tx-id logging (already merged/pending review, base mpeter/poc-index-finalized-block-results).
• Tag v1.5.3-with-sf-with-flow-tx-log deployed on the foundation gateway evm-001.mainnet0.nodes.onflow.org:8000.
• Reconciliation loop (PR Implement API functionality to support transactions #2) will be authored on top of the same soft-finality base branch.

[vishalchangrani]

Operational status

Foundation gateway (evm-001.mainnet0.nodes.onflow.org) has been rolled back to v1.5.0-batch-fix-with-sf-and-reduce-spend as a short-term mitigation. That build predates the nonce-aware TxMemPool (PR #971) and uses BatchTxPool with the always-enqueue fix from PR #965. Consecutive-nonce EVM transactions from the same EOA are guaranteed to be batched into a single Cadence tx via EVM.batchRun and executed sequentially in one execution frame, so the intra-block reordering that caused the wedge described earlier in this thread cannot occur. As a side effect ErrInFlightNonce does not exist in this build's code path either.

Traffic routing:

• DFNS / Hifi traffic is being served exclusively through the foundation gateway on the rolled-back build.
• The public endpoint (mainnet.evm.nodes.onflow.org) continues to run the latest version (v1.5.3-with-sf-with-flow-tx-log) with the nonce-aware mempool.

Tradeoff of the revert: every EVM tx now pays a blanket ~2.5s batching latency on the foundation gateway. Acceptable as a bridge because DFNS's rebroadcaster tolerates seconds-scale delays.

Durable fix in progress

PR #985 — feat(requester): auto-heal stuck nonce markers via reconciliation loop — adds a background loop to the nonce-aware mempool that polls each active EOA's most recent wrapping Cadence tx via GetTransactionResult(flowTxID) and clears the stuck marker within one Flow sealing window (~7s) when the wrapper is sealed-with-error or unsealed past a configurable threshold. That eliminates the wedge without requiring the batch-mode revert and preserves the mempool's fast-path benefits.

Once PR #985 is reviewed, testnet-validated, and cut into a new soft-finality tag, the foundation gateway will be upgraded back to the nonce-aware mempool with reconciliation and DFNS traffic will move back to the normal path.

[m-Peter]

Flow's 0.8s block time is main-chain block time, not collection cadence. Transactions flow through collector clusters first; each cluster produces collections at its own ~1–2s cadence. Cadence txs are cluster-routed by the payer address, so both wrappers (same payer, foundation COA 0xfa9fba4cf025057f) went to the same cluster and were batched into a single collection, then a single main-chain block. TxSubmissionSpacing = 1200ms in the mempool guarantees send-time spacing but not block-inclusion spacing

That's an interesting explanation. The currently deployed tag: v1.5.0-batch-fix-with-sf-and-reduce-spend, has demonstrated a zero transaction error rate, throughout the time it has been used. Its implementation is a mix of nonce-aware TxMemPool, which always enqueues, and submits every 2.5 seconds. It makes me think that the safest spacing we can achieve between 2 transaction submission, is around 2.5 seconds.

[vishalchangrani]

Here's the incident replayed end-to-end with the actual numbers — EOA 0xdEA9...58B3, nonces 25881/25882, 2026-07-29 03:09 UTC — first as it happened, then with the fix.

What happened (pre-fix)

T+0s — On-chain EVM nonce is 25881. Client sends tx nonce 25881 → gateway wraps it in Cadence tx X, submits. Mempool: lastConsecutivelySubmitted = 25881, lastFlowTxID = X, lastSubmittedAt = T0.

T+~2s — Client sends nonce 25882 → spacing satisfied, wrapped in Cadence tx Y, submitted. Mempool: lastConsecutivelySubmitted = 25882, lastFlowTxID = Y. Both marks happen on successful send, not execution.

T+~7s — X and Y land in the same Flow block, assembled in the wrong order (Y before X). Execution:

• Y runs first: on-chain nonce is still 25881, tx carries 25882 → run.cdc nonce assertion trips → Y seals with evm_error=nonce too high. No EVM.TransactionExecuted event — the event stream the gateway ingests is blind to it.
• X runs: 25881 executes fine → on-chain nonce advances to 25882. Gateway's index sees 25881 land and reports frontier 25882.

Net on-chain effect: 25881 executed, 25882 dropped. Net mempool belief: "everything up to 25882 sent and fine."

T+~2-3min — DFNS's rebroadcaster resends nonce 25882 → classify: highestSent() = 25882 ≥ 25882 → ErrInFlightNonce. DFNS marks it failed, releases the nonce, builds ~a dozen new txs at 25882 → all rejected the same way. Each rejection also
refreshes q.lastActivity, so the one recovery mechanism that existed — idle-queue eviction after 1 minute of no activity — kept being postponed by the retries themselves. Higher nonces queue behind the wedge.

T+~10min — A gap in DFNS's retry cadence finally lets the queue idle-evict; a fresh queue re-reads the index (25882), the next attempt is accepted and lands. That's the recovery the issue describes.

With the reconcile loop (PR #985)

Same story through T+7s — the fix doesn't prevent the reorder; Y still reverts. The difference starts there:

T+0…7s — The loop ticks every TxReconcileInterval (1s). Each tick snapshots the EOA (highestSent set, lastFlowTxID = Y ≠ 0) and polls GetTransactionResult(Y) outside the lock. While Y is Pending/Finalized/Executed: reverted=false,
stale=false → skip. No state touched.

T+~8s — Y seals. Next tick: Status = Sealed, Error = "evm_error=nonce too high" → reverted=true. Under the lock it re-verifies lastFlowTxID still equals Y and !inFlight(), then:

• resetSubmissionState() — clears lastConsecutivelySubmitted (the false 25882) and submitting
• lastFlowTxID = 0
• WARN log (reason=wrapper-reverted, flow_tx_id=Y) + TxPoolReconcileReset metric

T+~8s+ — DFNS's next rebroadcast of 25882: classify now finds highestSent() unset → reads the frontier from the index → 25882 (X already executed and indexed) → nonceNextExpected → fast-path resend as wrapper Z. Z seals, 25882 executes,
markers re-establish truthfully (lastConsecutivelySubmitted = 25882, lastFlowTxID = Z).

Wedge duration: ~one sealing window (~6-8s) + up to one 1s tick, versus ~10 minutes of retry-dependent limbo. And the backup path covers the other failure shape: if Y had instead never sealed at all (silent AN drop), the same reset fires
once now − lastSubmittedAt > TxReconcileStaleAfter (30s) — with Finding 1 from the review noted: as merged, that stale branch also resets healthy EOAs after 30s idle, which the docs say shouldn't happen.

One honest footnote: the fix is recovery, not prevention. Intra-block reordering can still burn a wrapper; what changed is that the gateway now learns about the revert from the wrapper's Flow tx status — the one channel that does record it —
instead of wedging on its own optimistic marker.