feat: validated chunk cache

[viviveevee]

Today a partial encoded chunk cannot have its header signature verified when it arrives, because resolving the chunk producer requires the chunk's parent block, which frequently hasn't been processed yet (chunk parts routinely beat their block through the network). The pipeline copes with a two-stage validation: validate_chunk_header_preliminary at arrival checks only shard_id/protocol_version and skips the signature, then validate_chunk_header_full checks the signature later once the parent lands.

This PR verifies the signature by carrying the grandparent block hash (the "anchor", offset 2) and the chunk's own epoch_id in a new chunk-header version, so the producer can be resolved and the signature verified at arrival without the parent — mirroring the existing state-witness V2 design (PartialEncodedStateWitnessV2 already carries prev_prev_block_hash and verifies via resolve_anchored_producer / verify_anchored_chunk_key).

All of this gates on the existing EarlyKickout protocol feature (v152, nightly).

What changed

New chunk header version (V7)

• ShardChunkHeaderInnerV7 = ShardChunkHeaderInnerV5 fields plus prev_prev_block_hash: CryptoHash and epoch_id: EpochId (core/primitives/src/sharding/shard_chunk_header_inner.rs).
• New enum variant V7(...) = 6, all accessor match arms, and two new Option-returning accessors prev_prev_block_hash() / epoch_id() (None for V1–V6). Outer ShardChunkHeader gets matching accessors.
• Constructed under EarlyKickout in ShardChunkHeaderV3::new; grandparent_hash + epoch_id are threaded through ShardChunkWithEncoding::new (5 call sites; genesis/tests pass defaults). validate_version gains the V7 arm.

Arrival-time verification (chain/chunks/src/shards_manager_actor.rs)

• validate_chunk_header_preliminary: for V7 headers, resolve the producer from the header's anchor + epoch_id, run the verify_anchored_chunk_key cross-check, and verify the signature at arrival — before caching or forwarding parts. If the anchor is unprocessed, the chunk is dropped (not cached, not forwarded), making the effective front horizon 2 while MAX_HEIGHTS_AHEAD stays 5.
• The header's epoch_id is not trusted — it is an index disciplined by the signature: a forged epoch_id resolves the wrong producer, whose key the sender doesn't hold, so the signature fails and the chunk is dropped. As hardening, the header's epoch_id is additionally bounded to the plausible epochs for its height (possible_epochs_of_height_around_tip), matching the witness path, to prevent a validator assigned in another epoch from racing out the real chunk via height/shard dedup near a boundary.
• validate_chunk_header_full is left as-is: it already re-resolves from the now-processed parent and is a genuine no-op re-check for V7 (and remains the authoritative backstop).

Shared anchored-producer helpers

• resolve_anchored_producer, verify_anchored_chunk_key, and the resolve_and_verify_anchored_producer wrapper moved from chain/client/src/stateless_validation/validate.rs (witness-only) to chain/chain/src/signature_verification.rs so chunks and witnesses share one implementation. They take a msg_label for caller-named errors; the ANCHORED_CHUNK_PRODUCER_LOOKUP_TOTAL metric moved to chain metrics.

RPC / view surface (core/primitives/src/views.rs)

• ChunkHeaderView gains prev_prev_block_hash and epoch_id as Option + #[serde(default)] fields (byte-identical for pre-V7). Forward From populates them from the new accessors; the reverse From gets a V7 arm keyed on their presence, placed before the V5 arm (V5 and V7 share the same field-presence tuple — without it the reverse impl would silently downgrade V7 to V5 with a different chunk_hash).
• OpenAPI (openapi.json, openrpc.json) and the protocol-schema snapshot regenerated.

Tests

• test_bad_signature_chunk_rejected_at_arrival — a pushed V7 chunk with an invalid producer signature is rejected in preliminary validation and never cached.
• test_v7_mismatched_height_rejected_at_arrival — a validly-signed V7 chunk whose height ≠ parent.height + 1 is rejected by the anchored key cross-check.
• test_cached_bad_chunk_evicted_by_full_validation — a chunk with a misstated epoch_id deposited directly into the cache is evicted by full validation (the backstop).
• early_kickout_boundary (test-loop, nightly) — a live 3-producer/3-shard chain upgrades across the EarlyKickout boundary; asserts liveness across the seam, no missing chunks in steady state (a verification gap would drop valid V7 chunks), and each block's header version matches its epoch's protocol version.
• Plus verify_anchored_chunk_key edge-case coverage (genesis / genesis+1, skipped heights, parent-present vs parent-absent).

Notes for reviewers

• Nightly-only (EarlyKickout = 152); no stable behavior changes.
• The reduced effective pre-fetch horizon (2 vs 5) under the feature is a deliberate consequence of drop-on-unresolvable-anchor; its stability impact is to be measured before enabling on anything but nightly.
• Chunks and witnesses now share get_chunk_producer_info_anchored + verify_anchored_chunk_key; keep them in lockstep with the kickout aggregator.

[codecov]

Codecov Report

❌ Patch coverage is 91.77820% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.25%. Comparing base (b8a041c) to head (57a9f0c).
⚠️ Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
core/primitives/src/views.rs	83.51%	14 Missing and 1 partial ⚠️
chain/chunks/src/shards_manager_actor.rs	89.38%	7 Missing and 5 partials ⚠️
chain/chain/src/signature_verification.rs	92.85%	4 Missing and 3 partials ⚠️
.../client/src/stateless_validation/validate_tests.rs	94.93%	2 Missing and 2 partials ⚠️
core/primitives/src/sharding.rs	95.12%	2 Missing ⚠️
chain/client/src/chunk_producer.rs	75.00%	0 Missing and 1 partial ⚠️
chain/jsonrpc/openapi/src/main.rs	0.00%	1 Missing ⚠️
...rimitives/src/sharding/shard_chunk_header_inner.rs	96.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #16070      +/-   ##
==========================================
+ Coverage   73.19%   73.25%   +0.05%     
==========================================
  Files         856      856              
  Lines      188394   188682     +288     
  Branches   188394   188682     +288     
==========================================
+ Hits       137904   138212     +308     
+ Misses      46070    46050      -20     
  Partials     4420     4420              

Flag	Coverage Δ
pytests-nightly	1.22% <0.00%> (-0.01%)	⬇️
unittests	69.91% <87.11%> (+0.05%)	⬆️
unittests-nightly	69.93% <90.97%> (+0.07%)	⬆️
unittests-spice	65.53% <89.05%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.