Merkle update compression

[Mustang98]

Optimized compression of the MERKLE_UPDATE part of the block based on the efficient state usage. Leads to up to 50% decrease in the block size.

[github-actions]

@codex review

[chatgpt-codex-connector]

To use Codex here, create a Codex account and connect to github.

[github-actions]

@codex review

[chatgpt-codex-connector]

To use Codex here, create a Codex account and connect to github.

[github-actions]

@codex review

[chatgpt-codex-connector]

To use Codex here, create a Codex account and connect to github.

[github-actions]

@codex review

[chatgpt-codex-connector]

To use Codex here, create a Codex account and connect to github.

[github-actions]

• validator/full-node-serializer.cpp:196-222 and 299-317 – tonNode_blockBroadcastCompressedV2 and tonNode_dataFullCompressedV2 now carry the proof out-of-band (f.proof_) but the deserializers no longer enforce any size limit on it. Previously the max_decompressed_size cap applied to the combined proof+data inside the compressed blob; now an attacker can ship an arbitrarily large proof_ and we accept/move it without bounds, which can exhaust memory or blow up logging while still passing the size check on data_compressed_/block_compressed_.

• validator/net/download-block-new.cpp:229-264 – when a peer sends dataFullCompressedV2 that requires state to decompress, we parse the unverified proof and immediately call wait_state_by_prev_blocks, which can trigger expensive state downloads/merges for attacker-chosen prev block IDs before any proof or signature validation. A malicious peer could spam crafted proofs to force repeated state fetches/merges and tie up resources.

[github-actions]

Findings

• extract_prev_blocks_from_proof only handles full BlockProof (block::gen::BlockProof::Record) and fails on proof links, but shard block broadcasts/dataFull still carry proof links. All stateful-decompression paths call this helper (e.g. FullNodeShardImpl::obtain_state_for_decompression, FullNodeCustomOverlay::obtain_state_for_decompression, FullNodeFastSyncOverlay::obtain_state_for_decompression, DownloadBlockNew::got_data). For any shard block using the new state-based compressed V2 format, extract_prev_blocks_from_proof will return “invalid block proof” and the broadcast/download is dropped/aborted, making the new compression unusable for shard blocks. Ref: validator/full-node-serializer.cpp:133-159, validator/full-node-shard.cpp:864-882, validator/full-node-custom-overlays.cpp:68-86, validator/full-node-fast-sync-overlays.cpp:62-82, validator/net/download-block-new.cpp:229-264.

Notes

• No other blocking issues spotted.

[github-actions]

Two critical issues spotted in the new MERKLE_UPDATE compression handling:

• validator/full-node-serializer.cpp still serializes broadcasts and full blocks in the old V2 layout (proof+data packed together and LZ4’d into compressed_; see serialize_block_broadcast around lines 20-67 and serialize_block_full around lines 120-156), but the TL schema now declares separate proof and data_compressed/block_compressed fields (tl/generate/scheme/ton_api.tl:433-466), and the new deserializers expect only one BOC root plus an external proof (validator/full-node-serializer.cpp:196-215, 250-277). Outbound messages won’t match the new schema, and the code will reject peers still speaking the old V2 (two-root) format. This is a protocol/compatibility blocker; serializers (and generated TL files) need to be updated to the new field layout and single-root compression.

• In the block download path, shard state is now fetched before any cryptographic validation: validator/net/download-block-new.cpp:225-260 extracts prev blocks from the untrusted proof in a tonNode_dataFullCompressedV2 and calls wait_state_by_prev_blocks prior to verifying the proof/signatures. A malicious peer can respond with bogus dataFullCompressedV2 messages to force expensive state loads/merges and only fail later during decompression/validation. The state fetch should be gated on proof/signature verification (or at least minimal validation) to avoid this DoS vector.

No tests were added for these paths. If you want, I can draft the serializer changes for the new TL layout and add a guard in the download path to validate before fetching state.

[github-actions]

• validator/full-node-serializer.cpp:133-152 assumes the broadcast proof is a BlockProof, but shard broadcasts/data responses carry a proof link. tlb::unpack_cell will fail for shard proofs, so extract_prev_blocks_from_proof returns an error and the new state-preload path drops shard blockBroadcastCompressedV2/dataFullCompressedV2 messages that require state (fast-sync/custom/public overlays and DownloadBlockNew). Shard blocks compressed with the new MU algorithm would never decompress. Needs to handle proof links too.
• validator/full-node-serializer.cpp:188-191 builds a BlockBroadcast with static_cast<UnixTime>(f.catchain_seqno_); BlockBroadcast.catchain_seqno is CatchainSeqno (uint32). This braced init will narrowly convert a double and should not compile (and would store the wrong type if it did). Should construct with CatchainSeqno.

[github-actions]

• validator/full-node-serializer.cpp:133 extract_prev_blocks_from_proof unpacks only BlockProof. Shard block broadcasts/dataFull carry a BlockProofLink (ValidateBroadcast uses create_proof_link for non-mc), so unpacking fails → wait_state_by_prev_blocks never runs and every V2 shard broadcast/data-full that requires state is dropped. The new Merkle-update compression therefore cannot be used for shard blocks. Handle proof links in this helper (or select the correct parser based on shard) before requesting state.