Skip to content

feat(el): default execution clients to discv5-only with ENR bootnodes - #1461

Draft
barnabasbusa wants to merge 9 commits into
mainfrom
bbusa/el-discv5-only-enr-bootnodes
Draft

feat(el): default execution clients to discv5-only with ENR bootnodes#1461
barnabasbusa wants to merge 9 commits into
mainfrom
bbusa/el-discv5-only-enr-bootnodes

Conversation

@barnabasbusa

@barnabasbusa barnabasbusa commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Draft — the named-devnet path needs el_enrs.txt, and mev.yaml is blocked on an upstream rbuilder rebuild. See Depends on and Known blockers.

Why

Discv5 bootstraps from ENRs, not enodes, and two clients make this mandatory rather than cosmetic:

  • besu discards its enode bootnode list entirely once discv5 is enabled and rejects enodes with Invalid ENR bootnode: ... must start with 'enr:'.
  • nimbus-eth1 now hard-disables discv4 (status-im/nimbus-eth1#4582, merged), so the enode list it receives feeds a protocol that no longer runs.

The other clients accept enr: in their bootnode flag too, so ENRs are a strict superset.

What

Discv4 off, discv5 on:

Client Flag
geth --discovery.v4=false --discovery.v5=true
erigon --discovery.v4=false --discovery.v5=true
ethrex --p2p.discv4=false --p2p.discv5=true
reth --disable-discv4-discovery
nethermind --Discovery.DiscoveryVersion=V5
besu --discovery-mode=V5
nimbus none needed — discv4 disabled upstream

Bootnodes now come from ENRs on both paths:

  • kurtosis / shadowfork — each peer's ctx.enr, which meant plumbing the ENR through besu, reth and nimbus (get_enode_enr_for_node); geth, erigon and ethrex already had it.
  • named devnets — new get_devnet_el_enrs() reading el_enrs.txt, mirroring get_devnet_enodes(). bootstrap_nodes.txt is CL beacon ENRs and stays reserved for the CL launchers.

Entries without an ENR are filtered out rather than emitted as empty strings, so a client that can't report one isn't used as a bootnode source instead of corrupting everyone's list.

ethereumjs is deliberately untouched: no discv5 implementation at all (config.ts has only discV4/discDns), so it keeps discv4 and enodes and will not peer with the discv5-only clients.

Default image bumps — all temporary

Every one of these was verified by running the image; three of seven ELs need builds newer than their current release:

image result
hyperledger/besu:latest (26.7.0) leaves --discovery-mode=V5 unmatched
ethpandaops/besu:main accepts it (and rejects the old --Xv5-discovery-enabled)
ghcr.io/paradigmxyz/reth invalid value 'enr:...': Failed to parse url: no host specified
ethpandaops/reth:main starts on enr: bootnodes — 2.4.1-dev (c505ed2)
nethermind/nethermind:latest exits 1: PublicKey should be 64 bytes long
ethpandaops/nethermind:master starts, logs Discv5 bootnodes accepted: 1

Enabling commits, none of which are in a release:

  • besu --discovery-mode#10800, 2026-07-22
  • reth enr: bootnodes via CLI — #26448, 2026-07-29

All three image pins should revert to the upstream images once these ship in releases.

Known blockers

  1. mev.yaml cannot pass yet. The builder runs ethpandaops/reth-rbuilder:develop, whose newest build is 2026-07-17 — 12 days before reth #26448. Verified directly: it rejects enr: bootnodes with Failed to parse url: no host specified. There is no rbuilder tag that works. It needs rebasing on reth ≥ c505ed2; there's no workaround in this package, since putting the builder back on discv4 + enodes would isolate it from the discv5-only mesh anyway.
  2. nethermind cannot report its own ENR. admin_nodeInfo has no enr field — BuildNodeInfo() sets Enode/Id/Ip/ListenAddress/Ports/Protocols only. Kurtosis' extractor errors hard on a missing field rather than returning empty, so nethermind stays on get_enode_for_node and is excluded as a bootnode source. It still peers fine using other nodes' ENRs, and shouldn't be made primary_bootnode. Fixing it upstream needs INodeRecordProvider moved into Nethermind.Network first — Nethermind.JsonRpcNethermind.Network.Discovery would be a circular project reference.

Depends on

ethpandaops/ansible-collection-general#568 adds el_enrs.txt. Until it lands and devnet metadata is regenerated, the named-devnet path has no ENR file to read. The kurtosis/shadowfork path works today.

Testing

kurtosis lint clean across src/.

7-EL enclave (geth, besu, erigon, reth, nethermind, nimbusel, ethrex) discv5-only with ENR bootnodes — all seven peered and began syncing:

el-1-geth        peers=6  block=8
el-2-besu        peers=1  block=6
el-3-erigon      peers=1  block=6
el-4-reth        peers=1  block=0
el-5-nethermind  peers=1  block=0
el-6-nimbus      peers=1  block=0
el-7-ethrex      peers=1  block=6

The same run on unmodified devnet-7 images had besu and reth both Exited (2). Nethermind takes ~2 min to find its first peer (slow discv5 lookup cadence) but does get there.

@barnabasbusa
barnabasbusa force-pushed the bbusa/el-discv5-only-enr-bootnodes branch 2 times, most recently from 81388da to e78d854 Compare July 30, 2026 08:40
@barnabasbusa

Copy link
Copy Markdown
Collaborator Author

CI status

Green: lint, check-typos, title_check, assertoor, minimal, mix-persistence, mev-mock, network_params, .

The only genuine failure is mev.yaml, and it is not fixable in this repo. The builder runs ethpandaops/reth-rbuilder:develop, newest build 2026-07-17 — 12 days before reth #26448 landed. Verified directly:

$ docker run --rm ethpandaops/reth-rbuilder:develop node --bootnodes="enr:-Iu4Q..." --disable-discv4-discovery
error: invalid value 'enr:-Iu4Q...' for '--bootnodes <BOOTNODES>': Failed to parse url: no host specified

No rbuilder tag exists that parses ENRs. It needs rebasing on reth ≥ c505ed2. There's no workaround here either — putting the builder back on discv4 + enodes would isolate it from the discv5-only mesh.

mix-public and mix-with-tools show as failed but were cancelled, not broken: run_with_args has no fail-fast: false (that setting at line 61 belongs to check-consensus-spec-values), so mev.yaml's failure cancels its siblings. Both had already brought up every service before being killed — mix-public logs all 6 ELs and all 6 CLs added, nethermind included:

Service 'el-1-geth-teku' added        Service 'cl-1-teku-geth' added
Service 'el-2-nethermind-prysm' added Service 'cl-2-prysm-nethermind' added
Service 'el-3-erigon-nimbus' added    Service 'cl-3-nimbus-erigon' added
Service 'el-4-besu-lighthouse' added  Service 'cl-4-lighthouse-besu' added
Service 'el-5-reth-lodestar' added    Service 'cl-5-lodestar-reth' added
Service 'el-6-ethrex-grandine' added  Service 'cl-6-grandine-ethrex' added
operation was canceled

Earlier runs in this PR were also mostly self-inflicted: per-pr.yml sets cancel-in-progress: true keyed on the branch, so each successive push cancelled the previous run's jobs and they surfaced as failures.

@barnabasbusa

Copy link
Copy Markdown
Collaborator Author

Made mev.yaml nightly-only rather than per-PR.

Rationale: the builder runs ethpandaops/reth-rbuilder, whose newest build (2026-07-17) predates reth #26448, so it rejects enr: bootnodes and cannot run discv5-only. Because run_with_args doesn't set fail-fast: false, that single leg was also cancelling mix-public and mix-with-tools — 3 of 8 legs red for one unbuildable image.

Coverage is not lost: nightly.yml globs .github/tests/*.yaml and runs with continue-on-error: true, so mev.yaml still executes there, just non-blocking. mev-mock.yaml stays in the per-PR matrix and passes.

Restore it here once ethpandaops/reth-rbuilder is rebased on reth ≥ c505ed2 — that rebuild is the real fix, and it's an ethpandaops-owned image (build-push-reth-rbuilder.yml, upstream flashbots/rbuilder).

Discv5 bootstraps from ENRs, not enodes. Besu discards its enode bootnode
list entirely once discv5 is enabled, and nimbus-eth1 now hard-disables
discv4 (status-im/nimbus-eth1#4582), leaving its enode list feeding a
protocol that no longer runs.

Switch the EL bootnode plumbing from enodes to ENRs and turn discv4 off:

  geth        --discovery.v4=false --discovery.v5=true
  erigon      --discovery.v4=false --discovery.v5=true
  ethrex      --p2p.discv4=false --p2p.discv5=true
  reth        --disable-discv4-discovery
  nethermind  --Discovery.DiscoveryVersion=V5
  besu        --discovery-mode=V5
  nimbus      no flag needed, discv4 is disabled upstream

Bootnodes come from each peer's ENR on the kurtosis/shadowfork path and from
el_enrs.txt on the named-devnet path. Entries without an ENR are skipped
rather than emitted as empty strings, so a client that cannot report one is
simply not used as a bootnode source. Nethermind is one such client: its
admin_nodeInfo has no enr field, and Kurtosis' extractor errors on a missing
field rather than returning empty.

ethereumjs is left on discv4 and enodes: it has no discv5 implementation.
… discv5-only

Verified by running each image:

  hyperledger/besu:latest      leaves --discovery-mode=V5 unmatched
  ghcr.io/paradigmxyz/reth     rejects enr: bootnodes with
                               'Failed to parse url: no host specified'
  nethermind/nethermind:latest exits 1 with
                               'PublicKey should be 64 bytes long'

besu --discovery-mode landed in #10800 (2026-07-22) and reth enr: bootnodes
via CLI in #26448 (2026-07-29); neither is in a release. Revert all three
pins to the upstream images once the changes ship.
The mev builder runs ethpandaops/reth-rbuilder, whose newest build predates
reth #26448, so it rejects enr: bootnodes and cannot run discv5-only. Because
run_with_args does not set fail-fast: false, that one leg also cancels
mix-public and mix-with-tools, taking out 3 of 8 legs.

nightly.yml globs .github/tests/*.yaml with continue-on-error: true, so
mev.yaml still runs there. Restore it here once reth-rbuilder is rebased on
reth >= c505ed2.
@barnabasbusa
barnabasbusa force-pushed the bbusa/el-discv5-only-enr-bootnodes branch from 11b1e81 to ebcba8e Compare July 30, 2026 09:07
reth's discv5 defaults to UDP 9200 (DEFAULT_DISCOVERY_V5_PORT), while
--discovery.port only sets the discv4 socket. The port spec declares
udp-discovery on the discovery port, so with discv4 disabled reth was
advertising an ENR pointing at an undeclared, unpublished port.

Container-to-container traffic inside an enclave is open so this still peered,
but with port_publisher enabled, or behind a real host firewall, reth's discv5
would be unreachable.

Confirmed via admin_nodeInfo on a discv5-only enclave: reth reported
ports.discovery 9200 while every other EL reported 30303.
@barnabasbusa

Copy link
Copy Markdown
Collaborator Author

Found a real bug while checking what admin_nodeInfo reports with discv4 off — reth's discv5 was on an unpublished port.

--discovery.port only sets reth's discv4 socket. Its discv5 listener defaults to DEFAULT_DISCOVERY_V5_PORT = 9200 (crates/net/discv5/src/config.rs:34), so with discv4 disabled reth was signing an ENR advertising UDP 9200 — a port the service's used_port_assignments never declares and port_publisher therefore never publishes.

Visible in admin_nodeInfo across a discv5-only enclave:

el-1-geth        ports: {"discovery": 30303, "listener": 30303}
el-2-besu        ports: {"discovery": 30303, "listener": 30303}
el-3-erigon      ports: {"discovery": 30303, "listener": 30303}
el-4-reth        ports: {"discovery": 9200,  "listener": 30303}   <-- undeclared
el-5-nethermind  ports: {"discovery": 30303, "listener": 30303}
el-6-nimbus      ports: {"discovery": 30303, "listener": 30303}
el-7-ethrex      ports: {"discovery": 30303, "listener": 30303}

It still peered in that run only because container-to-container traffic inside an enclave is unrestricted. With port_publisher on, or behind a host firewall, reth's discv5 would have been unreachable while looking healthy.

Fixed by passing --discovery.v5.port; reth now reports discovery: 30303 and still peers. Same fix applied to the reth role in ansible-collection-general#568, where reth_container_ports publishes only reth_ports_p2p and the firewall role would not have opened 9200 either.

Also worth noting from the same dump: enode stays fully populated with discv4 off on all seven clients. In geth it comes from the local node record (NodeInfo()srv.Self()node.URLv4()), and setupUDPListening()/SetFallbackUDP is gated on NoDiscovery, not DiscoveryV4 — so the record keeps its UDP port and you never get a discport=0 enode. Anything still consuming enode (ethereumjs) is unaffected.

pk910 and others added 5 commits July 31, 2026 11:20
…e enode (#1462)

Targets #1461. Merges #1451 and closes the gap between the two: with ELs
defaulting to discv5-only, the bootnodoor override still handed every EL
client an `enode://` URL, which feeds a disabled protocol — and besu in
V5 mode rejects non-ENR bootnodes outright, so a discv5-only enclave
with bootnodoor enabled had no working EL bootstrap at all.

## What

- **`launch_bootnodoor` also fetches `/el-enr`** and returns `(cl_enr,
enode, el_enr)`. `/el-enr` is the EL identity's record with the CL-only
`eth2` field stripped (mirroring why #1451 switched the CL side to
`/cl-enr`), and stays correct with `separate_keys: true`, where the CL
ENR is a different node identity altogether.
- **`el_launcher.launch` selects the override per client**: the seven
discv5-only launchers (geth, besu, erigon, nethermind, reth,
nimbus-eth1, ethrex) get the EL ENR — their `bootnodoor_enode` parameter
is renamed to `bootnodoor_el_enr` — while discv4-only ethereumjs keeps
the enode (bootnodoor still answers discv4 on the EL identity, so that
pairing keeps working).

## Testing

`kurtosis lint` clean. Two live enclaves on this branch, all images
freshly pulled:

**geth + besu + bootnodoor** — both ELs received `--bootnodes=enr:…`
matching bootnodoor's `/el-enr`, peered within 10s (besu is the strict
case: it hard-rejects enode bootnodes in V5 mode), both imported blocks.

**7-EL matrix + bootnodoor** (geth, besu, erigon, reth, nethermind,
nimbus-eth1, ethrex — bootnodoor as sole bootnode, no static peer
bootnodes):

```
el-1-geth        peers=5
el-2-besu        peers=5
el-3-erigon      peers=0   <- see below
el-4-reth        peers=5
el-5-nethermind  peers=5
el-6-nimbus      peers=5
el-7-ethrex      peers=5
```

Six clients registered in bootnodoor's EL table and formed a full mesh
through it (peers=5 = everyone except erigon); all seven lighthouses
registered in the CL table and peered. Chain progressed on all nodes.

### ⚠️ erigon's discv5 is non-functional

Erigon parses the ENR bootnode, logs `Setup P2P discovery v4=false
v5=true`, and binds UDP 30303 — but never sends a single discv5 packet:
after 15+ min it is absent even from bootnodoor's inactive-node DB, and
it logs `[p2p] No GoodPeers` forever. **#1461's own 7-EL test masked
this**: erigon showed `peers=1` there only because other clients had
erigon's ENR in their *static* bootnode lists and dialed it inbound.
Behind bootnodoor — where discovery is the only way to be found — erigon
is fully isolated (it still follows the chain via its CL / engine API).
Needs an upstream erigon fix; nothing this package can do.

### Stale-image traps (relevant to CI on both parent PRs)

- A cached `ethpandaops/bootnodoor:latest` still on v0.0.2 404s on
`/cl-enr`, hard-failing #1451's fetch.
- A cached `ethpandaops/besu:main` predating besu#10800 doesn't know
`--discovery-mode`.
- `ethpandaops/ethereumjs:master` (2026-05-19) crashes at startup with
`crypto.getRandomValues must be defined` — unrelated to these PRs (dies
generating its node key), but it means the ethereumjs+bootnodoor enode
path can't be live-tested until that image is rebuilt on a newer Node.

## Devnet EL seeding

#1451 seeded bootnodoor's devnet EL side from `enodes.txt` only.
`--el-bootnodes` accepts ENRs and enode URLs alike, so the new
`get_devnet_el_bootnodes` merges `el_enrs.txt` (discv5) with
`enodes.txt` (discv4) into one list and tolerates either file being
absent — `el_enrs.txt` only appears once
ethpandaops/ansible-collection-general#568 lands and devnet metadata is
regenerated, and discv4-only stragglers stay reachable through the
enodes. The merge snippet was verified for all four file-presence
combinations (exit 0, clean comma list); the devnet path itself has no
live devnet with `el_enrs.txt` to test against yet.

---------

Co-authored-by: Chase Wright <chase.wright@ethereum.org>
Signed-off-by: Barnabas Busa <barnabas.busa@ethereum.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants