Summary
gean holds ~3.9 GB RSS on a devnet node where ethlambda holds 451 MB, on the same host, same chain, same leanVM revision (a5909d1…). Roughly two thirds of the gap is glibc malloc arenas: gean has 35 fully-resident 64 MiB arenas, ethlambda has 1.
Not a blocker — the node is healthy and keeping perfect time — but it caps how many nodes fit on a host, and the devnet box is already swapping.
Measurements
Devnet, 15 nodes (4 gean / 6 ethlambda / 5 lantern), 4 committees, gean the aggregators, ~14 hours in at slot ~12,400.
/proc/1/smaps_rollup:
|
gean_0 |
ethlambda_0 |
| Rss |
3,945,352 kB (3.94 GB) |
451,308 kB (451 MB) |
| anonymous |
3,924,312 kB |
435,288 kB |
| file-backed |
5,263 kB |
2,669 kB |
| Swap |
541,960 kB |
— |
Count of fully-resident ~64 MiB mappings in /proc/1/smaps:
gean_0 35 arenas of ~64MB
ethlambda_0 1 arena of ~64MB
99.5% of gean's RSS is anonymous. Of that, only 153 MB is the Go heap (go_memstats_heap_inuse_bytes); lean_node_rss_bytes - go_memstats_heap_inuse_bytes is ~4.07 GB. So this is native-side memory, not Go.
It is not storage: lean_table_bytes gives states at 16.2 MB and the whole database at ~1.26 GB, read by point lookup, with Pebble's block cache at the 8 MB default.
Memory also oscillates hard — over 6 hours, min 3.32–3.63 GB, max 6.19–7.61 GB per node. That is the prover's transient scratch being taken and returned, and it is why single snapshots are misleading. The floor rose over 10 hours by +59 MB to +932 MB depending on the node, which is the part that looks like accumulation.
Mechanism
gean proves on the system allocator by default (--prover-arena off, cmd/gean/flags.go:66) so each proof's multi-GB scratch is returned rather than ratcheting RSS. glibc accepts it back but keeps it in per-thread arenas rather than returning it to the OS, and creates up to 8 × ncores of them — 128 on this 16-core host.
Arenas outlive the threads that created them. go_threads is only 24–27 on these nodes, so the 35 arenas came from earlier bursts of concurrent native calls, not from current thread count. (Checking go_threads first was misleading for exactly this reason.)
ethlambda runs the same leanVM prover with the same allocator choice, but keeps its heavy crypto on a bounded tokio::task::spawn_blocking pool, so it never fans out that way — hence 1 arena.
Worth noting the prover permit is not the difference: crates/common/crypto/src/lib.rs:64 serialises proving because leanVM panics on concurrent proofs, and gean's ProvingGate does the same. Both clients serialise proving. The divergence is elsewhere in how native work is spread across threads.
Candidate fix
One line, no code change. It is a dial rather than a binary — 4 or 8 would still recover most of the memory with more allocator concurrency.
The contention cost looks negligible for this workload: gean's crypto operations run 150–500 ms each while an allocation costs on the order of a microsecond, so lock time is a tiny fraction of job time. That reasoning covers gean's Go code but not the Rust crypto library's internal allocation pattern, which is unverified — if it does many small allocations, the estimate is optimistic.
Expected result, and what it will not fix
35 arenas → 2 should recover roughly 2.1 GB, taking gean from ~3.9 GB to maybe 1.5–1.8 GB.
It will not close the gap to ethlambda's 451 MB. The arenas explain ~2.24 GB of a 3.49 GB difference; about 1.25 GB remains unaccounted for — some is the Go heap (153 MB), some is partially-resident arenas that a 60–70 MB filter misses, and some may be something not yet found. That needs its own pass over the smaller mappings in smaps.
How to test
Change it on one gean node only and restart it, leaving the other three as controls on the same network doing the same work. Then compare:
docker exec gean_N cat /proc/1/smaps_rollup — Rss and Anonymous- the arena count one-liner above
lean_dispatch_event_duration_seconds and the attestation-verification timer, to catch any allocation slowdownmin_over_time(lean_node_rss_bytes[2h]) over a day, to see whether the floor still creeps
If memory drops on that node and timings are unchanged, it is confirmed.
Notes
Summary
gean holds ~3.9 GB RSS on a devnet node where ethlambda holds 451 MB, on the same host, same chain, same leanVM revision (
a5909d1…). Roughly two thirds of the gap is glibc malloc arenas: gean has 35 fully-resident 64 MiB arenas, ethlambda has 1.Not a blocker — the node is healthy and keeping perfect time — but it caps how many nodes fit on a host, and the devnet box is already swapping.
Measurements
Devnet, 15 nodes (4 gean / 6 ethlambda / 5 lantern), 4 committees, gean the aggregators, ~14 hours in at slot ~12,400.
/proc/1/smaps_rollup:Count of fully-resident ~64 MiB mappings in
/proc/1/smaps:99.5% of gean's RSS is anonymous. Of that, only 153 MB is the Go heap (
go_memstats_heap_inuse_bytes);lean_node_rss_bytes - go_memstats_heap_inuse_bytesis ~4.07 GB. So this is native-side memory, not Go.It is not storage:
lean_table_bytesgives states at 16.2 MB and the whole database at ~1.26 GB, read by point lookup, with Pebble's block cache at the 8 MB default.Memory also oscillates hard — over 6 hours, min 3.32–3.63 GB, max 6.19–7.61 GB per node. That is the prover's transient scratch being taken and returned, and it is why single snapshots are misleading. The floor rose over 10 hours by +59 MB to +932 MB depending on the node, which is the part that looks like accumulation.
Mechanism
gean proves on the system allocator by default (
--prover-arenaoff,cmd/gean/flags.go:66) so each proof's multi-GB scratch is returned rather than ratcheting RSS. glibc accepts it back but keeps it in per-thread arenas rather than returning it to the OS, and creates up to8 × ncoresof them — 128 on this 16-core host.Arenas outlive the threads that created them.
go_threadsis only 24–27 on these nodes, so the 35 arenas came from earlier bursts of concurrent native calls, not from current thread count. (Checkinggo_threadsfirst was misleading for exactly this reason.)ethlambda runs the same leanVM prover with the same allocator choice, but keeps its heavy crypto on a bounded
tokio::task::spawn_blockingpool, so it never fans out that way — hence 1 arena.Worth noting the prover permit is not the difference:
crates/common/crypto/src/lib.rs:64serialises proving because leanVM panics on concurrent proofs, and gean'sProvingGatedoes the same. Both clients serialise proving. The divergence is elsewhere in how native work is spread across threads.Candidate fix
ENV MALLOC_ARENA_MAX=2One line, no code change. It is a dial rather than a binary — 4 or 8 would still recover most of the memory with more allocator concurrency.
The contention cost looks negligible for this workload: gean's crypto operations run 150–500 ms each while an allocation costs on the order of a microsecond, so lock time is a tiny fraction of job time. That reasoning covers gean's Go code but not the Rust crypto library's internal allocation pattern, which is unverified — if it does many small allocations, the estimate is optimistic.
Expected result, and what it will not fix
35 arenas → 2 should recover roughly 2.1 GB, taking gean from ~3.9 GB to maybe 1.5–1.8 GB.
It will not close the gap to ethlambda's 451 MB. The arenas explain ~2.24 GB of a 3.49 GB difference; about 1.25 GB remains unaccounted for — some is the Go heap (153 MB), some is partially-resident arenas that a 60–70 MB filter misses, and some may be something not yet found. That needs its own pass over the smaller mappings in
smaps.How to test
Change it on one gean node only and restart it, leaving the other three as controls on the same network doing the same work. Then compare:
docker exec gean_N cat /proc/1/smaps_rollup— Rss and Anonymouslean_dispatch_event_duration_secondsand the attestation-verification timer, to catch any allocation slowdownmin_over_time(lean_node_rss_bytes[2h])over a day, to see whether the floor still creepsIf memory drops on that node and timings are unchanged, it is confirmed.
Notes