Skip to content

compute: fix shard misrouting for cross-stripe multi-block prefetch batches - #12931

Open
reatcat wants to merge 1 commit into
neondatabase:mainfrom
reatcat:fix/prefetch-register-bufferv-shard-misrouting
Open

compute: fix shard misrouting for cross-stripe multi-block prefetch batches#12931
reatcat wants to merge 1 commit into
neondatabase:mainfrom
reatcat:fix/prefetch-register-bufferv-shard-misrouting

Conversation

@reatcat

@reatcat reatcat commented Aug 11, 2026

Copy link
Copy Markdown

Fixes #12932

Problem

In prefetch_register_bufferv(), when registering a multi-block batch, the shard number of every block is computed from the base block's tag (get_shard_number(&tag)) instead of the per-block tag (hashkey.buftag, whose blockNum has already been advanced by i). Since get_shard_number() hashes blockNum / stripe_size, whenever a multi-block request batch crosses a shard stripe boundary, the blocks beyond the boundary get the shard number of the base block and are sent to the wrong pageserver shard connection.

The receiving pageserver counts it in pageserver_misrouted_pagestream_requests_total and drops the connection without a response (PageStreamError::Reconnect). The compute discards all in-flight prefetches (getpage_prefetch_discards_total) and retries one block at a time, which computes the shard correctly — so the query self-heals, but every crossing batch pays a connection teardown/reconnect and a wasted round trip.

Summary of changes

  • pgxn/neon/communicator.c: compute the shard from &hashkey.buftag (the per-block tag) instead of &tag, with a comment explaining why.
  • Add test_runner/regress/test_prefetch_register_bufferv_wrong_shard.py (PG 17 only, where the read-stream based multi-block path exists):
    • Shards the tenant across two pageservers with a deliberately prime stripe size (71, not a multiple of the <= 16-block read batch), so contiguous multi-block reads are guaranteed to cross a stripe boundary inside a single batch. Asserts the shards land on different nodes, otherwise ShardSelector::Page(key) would silently pick the correct local shard and hide the bug.
    • Writes a ~2000-page table, reads it back cold on a fresh endpoint (shared_buffers=128MB so the read stream builds 16-block batches), and asserts the result is correct and the pageserver misroute counter stays at 0.
    • A control case sets io_combine_limit=1 so every batch is a single block: misrouting is impossible even with buggy code, proving the main test exercises exactly the multi-block cross-stripe path.

Expected behavior:

  • Without the fix: every stripe boundary crossed by a multi-block batch misroutes the tail of the batch, so pageserver_misrouted_pagestream_requests_total and getpage_prefetch_discards_total both grow (roughly one event per crossed boundary), and the main test fails on the misroutes == 0 assertion.
  • With the fix: both counters stay at 0 and both tests pass.

…atches

In prefetch_register_bufferv(), compute the shard number from the
per-block tag (hashkey.buftag) instead of the batch's base tag, which
is never advanced in the registration loop.
@reatcat
reatcat requested review from a team as code owners August 11, 2026 05:02
@reatcat
reatcat requested review from HaoyuHuang, hlinnaka and problame and a lite review from Copilot August 11, 2026 05:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes shard misrouting for vectorized multi-block reads when a single batch crosses a shard stripe boundary by computing the shard number from the per-block BufferTag rather than the base tag. This prevents misrouted pagestream requests (and resulting reconnect/discard churn) during sequential scans on sharded tenants.

Changes:

  • Compute slot->shard_no from hashkey.buftag (per-block tag) inside prefetch_register_bufferv()’s multi-block loop.
  • Add a PostgreSQL 17 regression test that reliably triggers cross-stripe multi-block batches and asserts no misrouted pagestream requests.
  • Add a control test (io_combine_limit=1) to demonstrate the failure mode is specific to multi-block batches.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
pgxn/neon/communicator.c Fixes shard computation to use the per-block tag so multi-block batches crossing stripes route to the correct pageserver shard.
test_runner/regress/test_prefetch_register_bufferv_wrong_shard.py Adds a targeted PG17 regression + control test validating the fix by asserting misroute metrics remain zero under a forced cross-stripe workload.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

compute: shard misrouting for multi-block prefetch batches crossing a stripe boundary

2 participants