[consensus/marshal] Expand Digest Fetching by patrick-ogrady · Pull Request #3722 · commonwarexyz/monorepo

patrick-ogrady · 2026-05-05T22:08:56Z

Related: #3718

cloudflare-workers-and-pages · 2026-05-05T22:09:03Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	commonware-mcp	`1a93809`	May 15 2026, 07:06 AM

patrick-ogrady · 2026-05-05T22:10:03Z

    }

+    /// Enqueue a resolver fetch for a block commitment.
+    async fn enqueue_block_fetch(


remove this thin wrapper

patrick-ogrady · 2026-05-05T22:10:41Z

+                .fetch(ResolverKey::request(Request::Notarized { round }))
+                .await;
+        } else if let (Some(height), BlockSubscriptionKey::Commitment(commitment)) = (height, key) {
+            if height <= self.last_processed_height {


May need this to verify block (read parent)?

patrick-ogrady · 2026-05-05T22:12:10Z

    metadata: Metadata<R, u8, (Epoch, Epoch)>,

+    /// Blocks fetched by digest during ancestry verification, indexed by height.
+    digest_blocks: prunable::Archive<TwoCap, R, <V::Block as Digestible>::Digest, V::StoredBlock>,


put in caches

github-actions · 2026-05-05T22:12:56Z

Benchmark results

Tip

✅ PASSED: No benchmark exceeded the regression threshold.

Benchmark comparison table

Benchmark	Baseline (main)	Current	Delta	Threshold	Status
`qmdb::merkleize/variant=any::unordered::fixed::mmr keys=10000 ch=false sync=false`	1.528 ms	1.532 ms	+0.24%	10.00%	✅ PASS
`qmdb::merkleize/variant=current::ordered::fixed::mmb chunk=256 keys=10000 ch=true sync=false`	2.856 ms	2.830 ms	-0.91%	10.00%	✅ PASS

Baseline commit(s): ced07a033227

patrick-ogrady · 2026-05-05T22:17:42Z

        subscription.await.ok().map(V::into_inner)
    }
+
+    async fn fetch_parent(self, block: Self::Block) -> Option<Self::Block> {


make it clearer what actually fetches when subscribing.

cloudflare-workers-and-pages · 2026-05-05T22:37:25Z

Deploying monorepo with Cloudflare Pages

Latest commit:	`1a93809`
Status:	✅ Deploy successful!
Preview URL:	https://752626d6.monorepo-eu0.pages.dev
Branch Preview URL:	https://expand-digest-fetching.monorepo-eu0.pages.dev

View logs

# Conflicts: # consensus/src/marshal/coding/marshaled.rs # consensus/src/marshal/core/cache.rs # consensus/src/marshal/standard/deferred.rs # consensus/src/marshal/standard/inline.rs

patrick-ogrady · 2026-05-07T06:31:12Z

-        let block_rx = self.marshal.subscribe_by_digest(Some(round), digest).await;
+        let block_rx = self
+            .marshal
+            .subscribe_by_commitment(digest, CommitmentRequest::FetchByRound { round })


the one big downside of this approach is that we can fetch the same object twice (won't dedup because of hint)

If we refactor resolver to allow a multiple "retain keys" to a single "fetch key", we could minimize duplicate work.

patrick-ogrady · 2026-05-07T17:44:16Z

@@ -510,7 +515,10 @@ where
        }



For data on the pending tip, we actually want to specify not to fetch the block (and just wait for it).

This would make it possible for us to starve the resolver capacity (because we won't prune that pending request until finalization).

patrick-ogrady · 2026-05-07T17:53:36Z

        Finalization<S, V::Commitment>,
    >,
+    /// Verified blocks stored by height.
+    verified_blocks_by_height:


we should be able to get rid of the verified by view (if we can track digest by height we don't need that)

maybe call this "certified_blocks_by_height"?

patrick-ogrady · 2026-05-07T18:28:42Z

+    /// Verified blocks stored by height.
+    verified_blocks_by_height:
+        prunable::Archive<TwoCap, R, <V::Block as Digestible>::Digest, V::StoredBlock>,
 }


We should ensure we don't fetch beyond the finalized floor here? if so, we should at least be sure we dedup requests?

patrick-ogrady · 2026-05-07T18:28:57Z

+    verified_blocks_by_height:
+        prunable::Archive<TwoCap, R, <V::Block as Digestible>::Digest, V::StoredBlock>,
 }



I'm not sure we ever need to fetch by notarized anymore?

codecov · 2026-05-08T09:26:30Z

Codecov Report

❌ Patch coverage is 91.40518% with 126 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.79%. Comparing base (6725867) to head (ab54361).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
consensus/src/marshal/core/actor.rs	88.42%	13 Missing and 9 partials ⚠️
consensus/src/marshal/standard/mod.rs	94.50%	19 Missing and 1 partial ⚠️
resolver/src/lib.rs	61.36%	17 Missing ⚠️
consensus/src/marshal/resolver/handler.rs	89.72%	15 Missing ⚠️
consensus/src/simplex/actors/resolver/state.rs	62.50%	12 Missing ⚠️
consensus/src/marshal/core/cache.rs	89.36%	7 Missing and 3 partials ⚠️
consensus/src/marshal/mocks/verifying.rs	69.69%	9 Missing and 1 partial ⚠️
consensus/src/marshal/coding/marshaled.rs	80.95%	7 Missing and 1 partial ⚠️
consensus/src/marshal/core/mailbox.rs	82.05%	7 Missing ⚠️
consensus/src/marshal/standard/validation.rs	76.92%	2 Missing and 1 partial ⚠️
... and 1 more

@@            Coverage Diff             @@
##             main    #3722      +/-   ##
==========================================
- Coverage   95.82%   95.79%   -0.04%     
==========================================
  Files         466      467       +1     
  Lines      185569   187086    +1517     
  Branches     4427     4467      +40     
==========================================
+ Hits       177825   179214    +1389     
- Misses       6324     6430     +106     
- Partials     1420     1442      +22

Files with missing lines	Coverage Δ
consensus/src/lib.rs	`0.00% <ø> (ø)`
consensus/src/marshal/ancestry.rs	`95.00% <100.00%> (+0.35%)`	⬆️
consensus/src/marshal/coding/mod.rs	`98.66% <100.00%> (+0.10%)`	⬆️
consensus/src/marshal/mocks/harness.rs	`98.75% <100.00%> (-0.01%)`	⬇️
consensus/src/marshal/mod.rs	`54.54% <ø> (ø)`
consensus/src/marshal/resolver/p2p.rs	`100.00% <100.00%> (ø)`
consensus/src/marshal/standard/deferred.rs	`93.73% <100.00%> (+0.36%)`	⬆️
consensus/src/marshal/standard/inline.rs	`88.85% <100.00%> (+0.93%)`	⬆️
consensus/src/simplex/actors/resolver/ingress.rs	`100.00% <100.00%> (ø)`
resolver/src/p2p/inflight.rs	`100.00% <100.00%> (ø)`
... and 14 more

... and 14 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6725867...ab54361. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

# Conflicts: # consensus/src/marshal/coding/marshaled.rs # consensus/src/marshal/coding/mod.rs # consensus/src/marshal/core/actor.rs # consensus/src/marshal/core/mailbox.rs # consensus/src/marshal/mocks/harness.rs # consensus/src/marshal/resolver/handler.rs # consensus/src/marshal/resolver/p2p.rs # consensus/src/marshal/standard/deferred.rs # consensus/src/marshal/standard/inline.rs # consensus/src/marshal/standard/mod.rs # consensus/src/marshal/standard/validation.rs # consensus/src/simplex/actors/resolver/ingress.rs # consensus/src/simplex/actors/resolver/state.rs # examples/reshare/src/engine.rs # resolver/src/lib.rs # resolver/src/p2p/engine.rs # resolver/src/p2p/inflight.rs # resolver/src/p2p/ingress.rs # resolver/src/p2p/mocks/consumer.rs # resolver/src/p2p/mod.rs

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d0cd9d737f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-05-15T06:51:33Z

+                    if height > self.last_processed_height {
+                        if let Some(bounds) = self.epocher.containing(height) {
+                            self.cache
+                                .put_certified_block_by_height(
+                                    bounds.epoch(),
+                                    height,
+                                    digest,
+                                    block.clone().into(),
+                                )
+                                .await;
+                        }
+                        self.notify_subscribers(&block);
+                    }


Deliver fetched ancestry blocks to waiting subscribers

When a Request::Block response is valid but the Ancestry subscriber metadata does not match (for example, a proposal carries an incorrect child height so expected_height != block.height()), this branch skips notify_subscribers entirely. The fetch_parent caller is waiting on subscribe_by_commitment, so dropping the context without notifying leaves that verification/certification task hanging until external cancellation instead of promptly receiving the parent and rejecting the block. This is a liveness regression on malformed proposals and can stall verification work.

Useful? React with 👍 / 👎.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d0cd9d7. Configure here.}

cursor · 2026-05-15T06:58:20Z

+                        }
+                        self.notify_subscribers(&block);
+                    }
+                    false


Ancestry block delivery skips subscriber notification below floor

Medium Severity

In handle_deliver for Request::Block, the ancestry path only calls notify_subscribers when height > self.last_processed_height. If last_processed_height advances between the time a subscriber is registered (in handle_subscribe, where height > last_processed_height was true) and the time the resolver delivers the block, the block is silently dropped without notifying the still-registered block subscription waiter. The retain call in update_processed_height may cancel the resolver fetch, but if the response was already in-flight, the delivery arrives with valid subscribers while notify_subscribers is skipped. This can leave an AncestorStream's subscribe_parent call waiting indefinitely on a oneshot::Receiver that is never fulfilled nor closed.

^{Reviewed by Cursor Bugbot for commit d0cd9d7. Configure here.}

patrick-ogrady · 2026-05-19T07:05:47Z

Replaced by #3796

patrick-ogrady added 3 commits May 5, 2026 12:28

spike

759136d

spike

237c2e4

simplify sorting

d1fe76f

github-actions Bot assigned patrick-ogrady May 5, 2026

patrick-ogrady commented May 5, 2026

View reviewed changes

progress

ac10f4f

patrick-ogrady added 4 commits May 5, 2026 16:13

nits

598b6aa

nits

7eb4059

Merge branch 'main' into expand-digest-fetching

1db0f9f

# Conflicts: # consensus/src/marshal/coding/marshaled.rs # consensus/src/marshal/core/cache.rs # consensus/src/marshal/standard/deferred.rs # consensus/src/marshal/standard/inline.rs

simplify

6569b24

patrick-ogrady commented May 7, 2026

View reviewed changes

patrick-ogrady added 3 commits May 6, 2026 23:36

Fix reshare marshal API usage

538a4d0

Apply CI rustfmt

6db28ad

Fix consensus clippy lints

36ca7ce

patrick-ogrady commented May 7, 2026

View reviewed changes

patrick-ogrady added 6 commits May 7, 2026 11:55

Add resolver retain keys

3443d74

Split resolver retain key type

6e00e23

Wait for pending blocks during certification

82ecb07

Deliver resolver retain context

e1f0238

Use enum resolver delivery keys

4d4ff5f

simplify

f9a9333

patrick-ogrady added 10 commits May 7, 2026 22:10

nits

d95a3c2

spike

0746bea

more tweaks

1ff5222

change fetch rules

ae023d1

nits

940f65b

nit

b50c6a9

nits

b1463b3

nits

137f089

nit

971c7b4

comments

ab54361

commonware-llm marked this pull request as ready for review May 15, 2026 06:41

chatgpt-codex-connector Bot reviewed May 15, 2026

View reviewed changes

cursor Bot reviewed May 15, 2026

View reviewed changes

Simplify resolver subscriber plumbing

1a93809

patrick-ogrady closed this May 19, 2026

Conversation

patrick-ogrady commented May 5, 2026

Uh oh!

cloudflare-workers-and-pages Bot commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Workers

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark results

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloudflare-workers-and-pages Bot commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying monorepo with Cloudflare Pages

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented May 8, 2026

Codecov Report

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot May 15, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot May 15, 2026

Choose a reason for hiding this comment

Ancestry block delivery skips subscriber notification below floor

Uh oh!

patrick-ogrady commented May 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

cloudflare-workers-and-pages Bot commented May 5, 2026 •

edited

Loading

github-actions Bot commented May 5, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented May 5, 2026 •

edited

Loading