Skip to content

feat(proposervm): dedup inner block bytes when the inner VM retains accepted blocks#5629

Draft
containerman17 wants to merge 6 commits into
masterfrom
containerman17/proposervm-dedup-capability
Draft

feat(proposervm): dedup inner block bytes when the inner VM retains accepted blocks#5629
containerman17 wants to merge 6 commits into
masterfrom
containerman17/proposervm-dedup-capability

Conversation

@containerman17

@containerman17 containerman17 commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Why this should be merged

The proposervm persists each accepted block's full container, including the inner block bytes that the inner VM already persists in its own database, so every subnet block is stored twice. Measured on a 751k-block mainnet L1 (FIFA): the duplicate inner bytes are 1.36 GiB, 19.7% of that subnet's total storage (proposervm store 1.755 GiB, of which 1.385 GiB is inner bytes). This removes the second copy with no configuration.

How this works

Inner VMs declare a capability in their InitializeResponse: a new additive proto3 field retains_accepted_blocks = 6 meaning the VM durably retains every accepted block's bytes, retrievable via GetBlock, forever. No RPCChainVMProtocol bump is required (additive field, defaults false). The capability surfaces through a small optional interface (block.RetainsAcceptedBlocksVM), is set by rpcchainvm.VMServer from the served VM, exposed by VMClient, and forwarded through tracedvm. subnet-evm and coreth (the C-chain) implement it returning true: in both VMs accepted bodies are never deleted (state pruning, offline pruning included, only touches the state trie; only rejected blocks are removed). On the C-chain the method is declared by the atomic VM wrapper, since that is the VM the proposervm wraps and it embeds the inner VM as an interface, so a declaration on the inner evm.VM alone would not surface; coreth runs in-process, so the Go interface rather than the proto field is what engages there.

The proposervm checks the capability after the inner VM initializes and, when set, stores accepted blocks in a new state codec version (v1): the block bytes with the inner bytes stripped, plus the inner block ID. Reads reconstruct the exact original container by fetching the inner block from the inner VM. Both codec versions are always readable, so databases with mixed records (normal after an upgrade) work; existing v0 records are not rewritten.

Safety, in depth:

  • Write time: every stripped block is restored and byte-compared against the original before being written; any mismatch falls back to the legacy full v0 record. Because this check is pure in-memory container mechanics, a mismatch has no legitimate data-dependent cause and can only be a strip/restore code bug for some container shape, so the fallback is loud: an error-level log with the block ID and the cause, plus the dedup_blocks_fallback metric (alongside dedup_blocks_stored). Hard-failing the write instead was rejected: a shape-dependent bug would halt every upgraded node on the same block, and a storage feature must not be able to halt consensus.
  • Read time: the reconstructed block's ID is asserted to match the store key, so a wrong inner block can never be served.
  • VMs without the capability (P-chain, X-chain) keep the v0 path byte-for-byte unchanged.

How this was tested

  • Unit tests: strip/restore round-trips for all four stateless block types (signed, Granite, unsigned, option), dedup storage path, plus the existing state suites against the new constructor signatures. Tests that exercise the dedup write path assert dedup_blocks_fallback stays 0, and a dedicated test drives the fallback branch (a block whose bytes cannot round-trip) asserting the counter fires and the record is stored in the legacy full format.
  • End-to-end on a mainnet L1 (TixChain, 2834 blocks), synced from genesis with no subnet config and no chain config: the capability engages on its own (inner VM retains accepted blocks; storing blocks without inner bytes), dedup_blocks_stored 2834, dedup_blocks_fallback 0, all records codec v1, clean restart, historical RPC reads fine.
  • Size: proposervm block store 7.14 MiB (master baseline node, same chain, same tip) vs 1.51 MiB deduplicated, 79% smaller, consistent with the FIFA measurement above.
  • Bulk serving (the GetAncestors bootstrap-serve path adds an inner GetBlock hop per block): a p2p test peer walked the full 2835-block chain against both nodes. Master-built node: 0.11 s cold, 0.05 s warm. Dedup node: 1.15 s cold (about 0.4 ms per block for the inner hop, until the proposervm block cache warms), 0.05 s warm. Both nodes served byte-identical containers (7,187,477 bytes total).
  • C-chain: unit tests (graft/coreth/plugin/evm/atomic/vm, vms/proposervm). The proposervm side is chain-agnostic, the same write/read path exercised end to end above; the C-chain declaration itself is a constant method guarded by a compile-time interface assertion, and the write-time round-trip fallback applies there like everywhere else.
  • Downgrade: pointing a master-built binary at the deduplicated data dir fails chain creation with, verbatim: error while creating new snowman vm failed to repair accepted chain by height: failed to get last accepted block: unknown codec version. The node stays up, nothing is written, and re-upgrading works. avalanchego currently has no database version stamp at startup (the v1.4.5 in the db path is only a folder name), so making future downgrades fail with a friendlier message would be new node-level machinery; left as future work.

Need to be documented in RELEASES.md?

Yes: modifies the proposervm database format for chains whose VM declares retains_accepted_blocks (subnet-evm and the C-chain); downgrade to earlier versions is not supported after running this code.

…ccepted blocks

The proposervm stores each accepted block's full container, including the
inner block bytes that the inner VM (e.g. subnet-evm) already persists in
its own database. On FIFA mainnet (751k blocks) this duplicate copy is
1.36 GiB, 19.7% of the subnet's storage.

Inner VMs now declare a capability in their InitializeResponse
(retains_accepted_blocks, additive proto3 field): the VM durably retains
every accepted block's bytes, retrievable via GetBlock, forever. When the
capability is set, the proposervm stores accepted blocks without their
inner bytes (state codec v1: stripped block + inner block ID) and
reconstructs the exact original bytes on read via the inner VM's GetBlock.
subnet-evm declares it; no configuration is involved anywhere.

Safety: every stripped block is round-tripped at write time and falls back
to full v0 storage on any mismatch (dedup_blocks_fallback metric), and the
reconstructed block ID is asserted against the store key on read. Both
codec versions are always readable, so mixed databases from upgrades work.
Old binaries cannot read v1 records: downgrade after running this code
fails chain creation with "unknown codec version".

Verified on a mainnet L1 (TixChain, 2834 blocks) with no subnet or chain
config: capability engages on its own, 2834/2834 blocks stored
deduplicated with 0 fallbacks, block store 7.14 MiB -> 1.51 MiB (79%
smaller), clean restarts, and a peer-style GetAncestors walk of the whole
chain returns byte-identical containers to a master-built node.
@containerman17 containerman17 force-pushed the containerman17/proposervm-dedup-capability branch from 9893762 to cde9d7b Compare July 4, 2026 05:42
coreth never deletes accepted block bodies: state pruning (offline
pruning included) only touches the state trie, and only rejected blocks
are removed from the chain database. It therefore satisfies the
RetainsAcceptedBlocks contract the same way subnet-evm does, with the
identical state-sync caveat.

The method is declared on the atomic VM wrapper because that is the VM
the proposervm wraps: atomicvm.VM embeds the extension.InnerVM
interface, which does not include RetainsAcceptedBlocks, so declaring it
on evm.VM alone would not surface. coreth runs in-process, so the
optional Go interface (not the plugin proto capability) is the path that
engages.
A write-time round-trip mismatch has no legitimate data-dependent cause
(the check is pure in-memory container mechanics, the inner VM is not
involved), so it can only be a strip/restore code bug for some container
shape. Counting it was too quiet: nodes would silently keep double
storing. Hard-failing PutBlock was considered and rejected because a
shape-dependent bug would halt every upgraded node on the same block,
and a storage feature must not be able to halt consensus.

So the fallback now degrades loudly: tryStripBlock reports the cause
(strip error, restore error, byte mismatch, or marshal error) and
PutBlock logs it at error level with the block ID, next to the existing
dedup_blocks_fallback counter. The logger is plumbed from the chain
context through state.NewMetered and is nil-safe like the metric fields.

Tests now explode on an unexpected fallback: the dedup round-trip test
asserts dedup_blocks_fallback stays 0, and a new test drives the
fallback branch with a block whose bytes cannot round-trip, asserting
the counter fires and the record falls back to the legacy full format.
…ner block is missing

An unclean shutdown between the proposervm accepting a block and the inner
VM durably accepting it leaves the proposervm's last accepted above the
inner VM's. That state is exactly what repairAcceptedChainByHeight exists
to roll back, but with inner-block deduplication its first step, reading
the last accepted block to learn its height, needs the missing inner block
and failed, so chain creation failed permanently.

Since the proposervm commits strictly before the inner VM, a deduplicated
tip whose inner block is unavailable can only be above the inner VM's last
accepted height. Treat that read failure as proof and proceed with the
existing rollback without reading the tip. The same path also covers a
node killed during state sync (the state summary accepts the outer block
before the inner VM has synced) and an inner chain database that was
deleted out from under the proposervm.
…capability' into containerman17/proposervm-dedup-capability
containerman17 added a commit that referenced this pull request Jul 10, 2026
# Conflicts:
#	vms/proposervm/state/block_state.go
#	vms/proposervm/state/block_state_test.go
#	vms/proposervm/state/state.go
#	vms/proposervm/state/state_test.go
#	vms/proposervm/vm.go
containerman17 added a commit that referenced this pull request Jul 11, 2026
# Conflicts:
#	vms/proposervm/state/block_state.go
#	vms/proposervm/state/block_state_test.go
#	vms/proposervm/state/state.go
#	vms/proposervm/state/state_test.go
#	vms/proposervm/vm.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant