compute: a per-process registry of published index arrangements - #38387
Draft
antiguru wants to merge 1 commit into
Draft
compute: a per-process registry of published index arrangements#38387antiguru wants to merge 1 commit into
antiguru wants to merge 1 commit into
Conversation
antiguru
force-pushed
the
mh/interactive-02-sharing
branch
from
August 21, 2026 11:23
0b83619 to
e4df1e7
Compare
antiguru
force-pushed
the
mh/interactive-02-sharing
branch
from
August 21, 2026 13:24
e4df1e7 to
db592b9
Compare
antiguru
force-pushed
the
mh/interactive-02-sharing
branch
from
August 21, 2026 13:42
db592b9 to
acaae62
Compare
antiguru
force-pushed
the
mh/interactive-02-sharing
branch
2 times, most recently
from
August 21, 2026 17:54
e0c1fed to
7eeb13c
Compare
antiguru
force-pushed
the
mh/interactive-02-sharing
branch
from
August 28, 2026 14:05
7eeb13c to
5c1ad88
Compare
Publishing an arrangement produces handles, but a reader on another thread needs a way to find them. This adds a registry keyed by `GlobalId` and worker ordinal, shared across all timely workers of a process the way the persist client cache is, so worker `i` publishes into slot `i` and a reader on worker `i` of another runtime looks up the same slot. That is sound only because both sides shard keys by `key.hashed() % peers`, which the equal-peer requirement guarantees. A slot can be created before it is filled. `get_or_create_placeholder` lets whichever side touches an id first create the slot and the other adopt it in place, so a reader never overwrites a publisher's arrangement or imports over a slot that a later publish replaces. Readers learn about publication and seal through a dirty-id inbox with a coalescing `SyncActivator` rather than by polling. Nothing in the crate calls the registry yet, so it is inert in the same sense as the shared-trace primitive it builds on. `published_logical_holds` is `pub` because its callers arrive later and crate scoping would make it unreachable for dead-code analysis. Tests are out of line in `sharing/tests.rs`, per the convention in `src/compute/AGENTS.md`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
antiguru
force-pushed
the
mh/interactive-02-sharing
branch
from
September 3, 2026 08:48
5c1ad88 to
882fe25
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of eight PRs splitting #37770. Stacks on #38386. Tracked by CPU-215.
Publishing an arrangement produces handles, but a reader on another thread needs a way to find them. This adds a registry keyed by
GlobalIdand worker ordinal, shared across all timely workers of a process the way the persist client cache is. Workeripublishes into slotiand a reader on workeriof another runtime looks up the same slot, which is sound only because both sides shard keys bykey.hashed() % peers.A slot can be created before it is filled.
get_or_create_placeholderlets whichever side touches an id first create the slot and the other adopt it in place, so a reader never overwrites a publisher's arrangement nor imports over a slot a later publish replaces. Readers learn about publication and seal through a dirty-id inbox with a coalescingSyncActivatorrather than by polling.Inert: nothing constructs a registry.