perf(docker): prove on jemalloc instead of glibc malloc

[dimka90]

One Dockerfile change: install libjemalloc2 and LD_PRELOAD it, so the prover's allocations go through jemalloc rather than glibc malloc.

Refs #424, and supersedes the MALLOC_ARENA_MAX=2 approach that was withdrawn from #427 before merge.

Why

The prover allocates its scratch through the system allocator and frees it after each proof. glibc takes it back but does not hand it to the OS. Most of it lands in the main heap, which can only shrink from the top — one live allocation above a freed block pins everything below it for the life of the process. jemalloc allocates only via mmap and releases pages on a decay timer.

Measurement

One node on jemalloc, three on glibc. Same image, same chain, same slot, same host — the allocator is the only difference.

node	allocator	RSS at slot 3,440	brk heap
gean_0	jemalloc	448 MB	0
gean_1	glibc	1,150 MB	264
gean_2	glibc	1,135 MB	302
gean_3	glibc	1,126 MB	694

60% reduction, sustained ~1,900 slots after the swap — not an early-run artifact.

The shape is the stronger evidence. The jemalloc node oscillates as memory is taken and returned; the glibc nodes are pinned flat because theirs cannot come back:

gean_0 (jemalloc)  351 → 410 → 357 → 455 → 543 → 454 → 472 → 715 → 532 → 435 → 448 MB
gean_1 (glibc)     973 → 1000 → 973 → 1008 → 1011 → 1030 → 1018 → 975 → … → 1150 MB

gean_0's peak (715 MB) stays below every control's floor (973 MB).

No measured cost

node	signing rate/s	agg verify	CPU cores	tick
gean_0 (jemalloc)	0.204	132.7 ms	0.605	0.800 s
gean_1	0.193	135.4 ms	0.666	0.800 s
gean_2	0.196	140.2 ms	0.977	0.800 s
gean_3	0.193	158.5 ms	0.892	0.800 s

Highest signing rate, fastest aggregated-signature verification, lowest CPU, identical tick. Same justified and finalized slots as the other three, in sync, zero restarts.

Why not MALLOC_ARENA_MAX=2

It worked (~69% on its own measurement) but:

• it bounds retention and cannot return what the main heap already holds — brk was still 264–694 MB on the capped nodes
• it costs allocator concurrency, and that cost grows with the validator set
• it is implicated in fix(node): a wedged native XMSS merge kills the proposal worker and the proving gate, with the node still reporting healthy #430: a thread wedged in vm_mmap_pgoff while holding one of only two arenas took a whole node down ~2.5 hours later, unkillable, requiring a host reboot

jemalloc addresses the class of problem instead of capping a symptom.

Notes on the change

Unqualified soname is deliberate. ld.so resolves libjemalloc.so.2 per architecture, and this image builds for arm64 too; a hardcoded x86_64 path would silently degrade there. Verified that ldconfig -p resolves the bare soname, and that libjemalloc2 is the providing package.

The ldconfig check fails the build rather than letting a missing library become a runtime warning that's easy to miss — a silent failure here would look exactly like "jemalloc didn't help".

Validation status — please read

The mechanism is validated on devnet, but via LD_PRELOAD with the host's library bind-mounted into the running container, not via this Dockerfile. The library and the mechanism are identical; what has not been exercised end-to-end is the image build path.

So on the first devnet run with an image built from this branch, confirm it actually took:

docker exec gean_0 grep -c jemalloc /proc/1/maps    # must be > 0
docker exec gean_0 sh -c "awk '/\[heap\]/{f=1} f&&/^Rss:/{print \$2; exit}' /proc/1/smaps"   # expect 0

If the first is 0, the preload didn't load and the memory result will not reproduce.

[dimka90]

Final reading, slot 5,705

The run continued 2,265 slots past the figures in the description. The result strengthens, and one nuance is worth recording.

node	allocator	RSS floor (2h)	peak (2h)	brk
gean_0	jemalloc	341 MB	4,749 MB	0
gean_1	glibc	960 MB	2,697 MB	144
gean_2	glibc	1,053 MB	4,841 MB	282
gean_3	glibc	1,096 MB	3,142 MB	718

65-69% lower floor, using min_over_time over 2h rather than a point sample, since RSS oscillates with prover activity.

Peaks are not lower, and that matters for how this is described. gean_0 peaked at 4,749 MB and gean_2 on glibc peaked at 4,841 MB — transient peaks are comparable between allocators. jemalloc does not constrain how much the prover takes; it changes whether the memory comes back afterwards. The floor is the figure that determines how many nodes fit on a host over hours, so "floor 341 MB vs 960-1,096 MB, peaks comparable" is the accurate description rather than a single headline number.

Still no cost at this slot:

node	signing/s	agg verify	CPU cores	tick
gean_0 (jemalloc)	0.232	127.6 ms	0.651	0.800 s
gean_1	0.234	127.4 ms	0.833	0.800 s
gean_2	0.236	145.8 ms	0.965	0.800 s
gean_3	0.232	154.6 ms	0.927	0.800 s

Identical signing rate, tied-fastest aggregate verification, lowest CPU, identical tick. Zero restarts across all four for the life of the run; justified and finalized tracked the other nodes throughout.

[dimka90]

Dockerfile path verified end-to-end

The description flagged that the mechanism had only been validated via LD_PRELOAD with a bind-mounted library, not through an image built from this branch. That gap is now closed.

An image built from 86f26cd is running on devnet with all four gean nodes as aggregators:

image:  ghcr.io/geanlabs/gean:jemalloc
rev:    86f26cd3d68f235d8aaebada48f596a982bdcd16

gean_0  jemalloc_maps=5  brk=0 MB
gean_1  jemalloc_maps=5  brk=0 MB
gean_2  jemalloc_maps=5  brk=0 MB
gean_3  jemalloc_maps=5  brk=0 MB

The unqualified soname resolves and the preload loads — grep -c jemalloc /proc/1/maps returns 5 on every node, and brk is 0, confirming the main heap is bypassed. The ldconfig build-time check passed during the build, so the libjemalloc2 package name is correct for ubuntu:24.04.

This run also puts jemalloc under the aggregator role for the first time — the earlier test had gean not aggregating. That is the configuration where the prover allocates hardest, and it is directly comparable to the original baseline (gean aggregating on glibc, RSS floors 2,691-3,429 MB), so the matched-slot comparison will be like-for-like on role as well as slot.

Results to follow once the chain has depth.