publisher: several channel instances of one feed specification - #98
Conversation
…rocess A venue whose instrument set is sharded across many channels needs one process to operate many channel instances of one specification. Today a process emits at most one feed per specification, and the check that says so is holding shut three more fences that are silent. The key is `shard`, not `route`. `route` is taken twice over — `[egress]` means the IP route to the group by it, and GLOSSARY.md defines `path` as "one of several redundant routes carrying the same data", which makes N routes read as N redundant paths carrying the same instruments when they carry disjoint ones. `category` is a feed-registry arbitration boundary the glossary says outright is not a taxonomy. `shard` is the glossary's own word: a channel is "a logical shard of the instrument set". The reference-data question is not a design choice. reference-data/spec.md heads all seven publisher obligations with "A publisher operating a channel ... MUST", and makes Manifest Seq, Valid, Instrument Count and the definition cycle per channel. One process-wide published set on N refdata ports produces four specific untruths — and obligation 6, restarting the cycle on a Manifest Seq change, turns any admission anywhere into an N-channel definition burst, which is what obligation 2 forbids. The era store is a latent state-corruption bug rather than an inefficiency, and not the one it first looks like: begin_era advances and persists on every call and run.rs calls it per block, so N blocks take N different eras. The era a channel advertises is therefore decided by its position in the document, each start advances the shared counter by N, and moving a block hands a channel an era it has already published under. Three fences beyond the four the request named: the reference copy keys its socket on (spec, port role), the snapshot rotation divides the cycle by the process's published count, and nothing checks Channel ID uniqueness across blocks — which two blocks sharing an ID would turn into two feeds silently sharing four metric series.
Eight tasks of the feed-routes plan, all of them behaviour-neutral: the duplicate-specification check stays in force, so with one shard configured every partition is a partition of one and the suite passes unchanged. Lifting that check is task 9. **The document.** `[[feed]] shard`, absent resolving to the default. A checked `ShardName` — one lowercase path component, at most sixty-four bytes — because the name becomes a path in two places and the third use is the one that forgets to check. Spelling the default explicitly is refused: two spellings of one shard are two era files. `feed_specs()` is distinct now, because it answers *which feeds does this publisher emit* and an adapter handed one specification once per shard is being told about the deployment instead. **A refusal that closes a hole that already existed.** Two blocks sharing a `Channel ID` load today: `channel_ids()` sorts and dedups, so one set of series is pre-created and two channel instances write to it, with nothing saying so. One channel per specification made it unlikely; many make it a matter of time. **The boundary.** `list_on` is required and `list` is defaulted, in that direction on purpose — the reverse lets an implementor who did not update admit every instrument to the default shard, every shard collapsed onto one channel with no error, no counter and no log. A `compile_fail,E0046` doctest pins it. **The registry.** One registry, N published sets. One per shard would be N `Instrument ID` spaces and N writers on one state directory. The shard is an index on the registry's own published entry, not on the instrument table, which would drag the lowering — and the recorder's offline re-lowering with it — into a change that buys them nothing. **The era store.** Keyed on the channel instance, and the default shard keeps `<spec>.era`. That last clause is the whole risk: a renamed file reads as *no file*, which resolves to the first era, so a publisher on era 7 would restart on era 1 and announce nothing — the corruption the corrupt-file refusal exists to prevent, delivered by the upgrade meant to be safe. `ShardName::era_shard` is the one place that mapping is made, because the natural thing to write at a call site is the wrong thing. **The send paths.** Two vectors indexed by shard, built shard-outer. Per block in document order, a file interleaving the blocks would index the two vectors differently and route one instrument's quotes and its levels to two channels, and nothing below `Feeds` re-checks it. **The rotation.** Divided by the shard's own published count, and walking only its own instruments. Fixing the divisor alone leaves the same 1/N error arriving through the cursor, because the slots are one table for the process.
…sions
The end-to-end test the plan calls its centre. Two shards, both
specifications: the arrangement the change exists for, and the one the
duplicate-specification gate refused until it lifted.
Every assertion is an exclusion. "Shard A's reference-data port carries
A's definitions" passes against a publisher that packs everything onto
everything, so each test states what a channel did *not* carry beside
what it did, and the two shards are given different instruments so that
"carried the other's" is a statement that can be false.
Three things the first draft got wrong, all caught by the tests rather
than by reading:
- `poll_listings` is due-gated on `LISTING_POLL` and returns whether it
ran. A second call on the same instant is a no-op, so the manifest
test was reading a manifest that never saw the admission. The gate is
asserted now instead of discarded.
- A definition is type 0x02, not 0x10. The floor assertion ("a port
carried no definition at all") is what said so.
- Symbols are read by decoding with the codec's own decoder rather than
hunting substrings in the datagrams.
The era is deliberately not asserted here: the harness hands each
pipeline a literal `ResetCount` rather than resolving one, so the
assertion would be about the fixture. That property lives in the era
store's suite, where it can be observed.
The centre test asserted "carries mine" and "does not carry theirs" as two membership checks over one instrument per shard. Against a publisher that packs the whole published set onto every reference-data port, that pair does fail — but it fails on the first assertion, "carries none of its own", because the pacer owes one definition per lap and a packing publisher still puts exactly one symbol on each port. The exclusion, the assertion the plan calls its centre, was never reached. Two instruments per shard is what makes a packing publisher show up as four symbols where two belong, and the assertion is now an equality: A's port carries A's two and nothing else. Inclusion and exclusion in one statement, with no ordering or repetition to read around — `symbols_on` dedups and sorts, because a full lap repeats and the question is which instruments a channel described, not how often. The port count is asserted too. Four channel instances is the arrangement under test, and a harness that quietly grew or lost a port would narrow every assertion above it without failing one.
…e default Task 5, which the tasks 1-to-8 commit missed: `TeeConfig::destination` was still `<path>.<spec>.<role>`, so the four channel instances the change exists for fanned out to five sockets. Two shards' copies of one feed's one role arrived on one socket, and a Unix datagram carries neither the destination port nor the group the diff is keyed on — a recorder cannot attribute them without decoding, which is the one thing a record path does not do. The existing argument for the feed being in the name is the same argument one noun further along, so it is extended rather than rewritten. The default shard is spelled by its absence, as its era file is, and for the same reason: what a call site naturally writes is `shard.as_str()`, and that renames the socket every existing deployment's recorder is bound to. The fan-out would then write to a path with nobody on it — every datagram dropped and counted, or an operator who believes copies are still being archived. The tests are path arithmetic and open no socket: the default shard's five destinations as literals, a named shard's distinct from them, and the fifteen sockets three shards of both feeds open asserted distinct rather than sampled. Reverting either half of the naming fails one — the always-append mutant on the default's literals, the never-append mutant on the distinctness.
The built-in record adapter called `list`, so every instrument it offered went to the default shard however many the document configured. That is the whole of what stood between `run()` and a two-shard offline run: a record names a symbol and nothing else, so the partition has to come from configuration, and `[[adapter.upstream.listing]]` is where the source process and the publisher already agree out of band. `shard` is an `Option` rather than a `String` defaulted to the token. A source with no partition to state says nothing, `poll_listings` then calls `list`, and the default keeps one spelling in the crate that owns it. Unchecked on this side on purpose: a name that is not a configured shard is declined at admission and counted, and checking that a *block's* name is a safe path component belongs to the block. The tests state the whole sequence rather than one offer — an adapter passing the token for everything satisfies any assertion about the defaulted row, and one reaching for the first listing's shard satisfies any assertion about the first. Three rows with three answers is what neither survives. The second test is the boundary's own documented case: a `None` back costs that listing and the poll continues, because the listings after it are the ones on shards that do exist.
…bscribers Task 12. `examples/replay.sh` was one feed of one shard; it is now four `[[feed]]` blocks of **one** specification — the arrangement the duplicate-specification gate refused until it lifted — through `run()`, over real multicast sockets, read by four instances of this repository's Go subscriber. One block names no shard and is therefore the default one, so the run tests the upgrade as well as the feature. Every assertion is the subscriber's own output. Per channel instance: its own `Channel ID`, its own sequence series from 0, its own `Reset Count` equal to its own era file, and — stated as an equality rather than as "carries mine" — only its own shard's two instrument definitions. Two symbols per shard for the end-to-end test's reason: with one each, a publisher that packed everything onto everything still puts exactly one symbol on each port and the exclusion is never reached. It runs twice, because an era is only observable across a restart. Four era files under the state directory, the default shard's still `top-of-book.era`, and each advanced by exactly one — a shared counter advances every channel by the number of blocks, which would read here as four. `--symbol` on `write_records` is repeatable, and every symbol carries the same three events, so a difference between two subscribers' outputs is the publisher's partitioning and not the recording's. Two things the run itself found. `GROUPS` is bash's own array of the invoking user's group ids and assignments to it are ignored, so the document came out naming a multicast group of `1000`; the load refused it by name, which is the failure this publisher is written to produce rather than start on. And the clippy stable runs is stricter than the one this test was written under — `iter().any(|id| *id == x)` on a slice is `contains`.
…t do Forty-eight boxes, and an "It ran" section holding the output rather than a claim about it: four channel instances of one specification through `run()`, over real multicast sockets, read by four instances of this repository's Go subscriber, twice — because an era is only observable across a restart. The honest part is what did not run. Task 12 asked for two shards and both specifications; the built-in record adapter holds no book, answers no snapshot, and `AdapterRegistry` refuses it beside a `market-by-price` block rather than publish deltas a mid-session joiner can never resynchronise. That refusal is right and predates this plan, and what it means is that a real depth run needs a venue with a book, which this repository does not have. The bullet says so now instead of asking for something that is refused at startup. Two things only the real run could find, both recorded: a document that came out naming a multicast group of `1000`, because `GROUPS` is bash's own array and assignments to it are ignored; and the reference-copy fan-out still keyed on the feed and the port role alone, which is task 5 — missed by the tasks 1-to-8 commit and found by four channel instances fanning out to five sockets.
… assumed Acceptance criterion 1 had no test. Everything else about shards is asserted on two of them — enough to make a partition falsifiable, not enough to say the document scales — and the set checks are the ones that would disagree with themselves at size: one shard missing a specification, two blocks on one `(spec, shard)` pair, a repeated `Channel ID`. 31 shards of both specifications, every port distinct across the document, 62 distinct `Channel ID`s counted rather than assumed, and every shard carrying a block for every specification asserted across the whole set. One shard is the default, named by the absence of the key, because a deployment that grows into shards grows out of a document that had none.
The last acceptance clause asks that reverting any one of tasks 3, 4, 7 or 9 fail a named test, and a plan that only asks for that has not checked it. Each revert was run and the plan now names the tests that failed under it, including the two directions task 5's naming can be wrong in — always appending the shard fails the default's literal destinations, never appending it fails the distinctness of the fifteen sockets three shards open. The clause earned its keep: the revert that failed nothing was task 5, because the reference-copy socket had never been keyed on the shard in the first place.
… owns `GLOSSARY.md` 1.3.0 replaces the word this repository still uses for the reference copy with `fan-out`, and the plan's own vocabulary constraint says so in as many words — it governs identifiers, comments, test names, config keys and commit messages alike. What this change wrote had followed the file's existing prose, which was already right, and not its test names. So the concept takes the glossary's word: three test names, the local bindings beside them, the example `path` an operator copies out of the doc comment, and every identifier and comment in `replay.sh` — where the run's own output now prints `fan-out.top-of-book.alpha.mktdata`. The recorded output in the plan is the new run's, not the old one edited. What deliberately does not move: the `[adapter.tee]` key, `TeeConfig`, `Tee`, `ReferenceStream` and the `StartupError` variants named after them. Those are a config key operators have written down and public API of published crates, so renaming them is a change with its own migration and its own PR, not a side effect of one about shards. The two test names that describe the *key* keep it too, because naming a key is not using the word. The design document's blockquote of the old doc comment stays as it was: it quotes what the comment said on the day the design was written, which is what a dated document records.
…ligned `Feeds` held two parallel vectors indexed by shard. The index was the whole of the pairing: the pipeline at position `i` of one vector and the pipeline at position `i` of the other had to be the same shard, or an instrument's quotes and its levels left by two different channels — under two different sequence series, which a subscriber reads as its own and cannot detect. What held that together was four `assert!`s in `push_shard`, and its own doc comment called them the second line of defence. There is nothing left to align. A `ShardFeeds` owns one shard's two send paths and derives its name from one of them at construction, so a document that interleaves its blocks cannot separate a shard's specifications — they are built together and held together. The four assertions go, `shard_count()` stops being `max(len, len)`, and `shard_name()` stops being first-non-`None`. `ShardFeeds::new` returns `None` for the pair with neither specification, which no resolved document produces — the shard set is the distinct shards of the enabled blocks — and `run.rs` turns it into a new `StartupError::ShardWithNoFeed` rather than a skip. A skip is the silent failure here: `Feeds` and the reference-data registry's shard list are both indexed by the same `Config::shards()`, so dropping an entry from one shifts every later shard onto another channel instance's numbering. A struct and not a three-variant enum, which is the other way to make the empty pair unrepresentable: the variant holding both feeds is one `FeedPipeline` larger than the ones holding one, so every entry in the vector would be sized for it, and `clippy::large_enum_variant` is denied in CI. A document carries the same specifications on every shard anyway, so that padding would be paid on all of them or none. Also here, because they were reading the vectors directly: the definition tick and the final manifest go through the shard, the teardown's two whole-publisher passes iterate shards while keeping the steps ordered against each other across the publisher, and the snapshot rotations are built one per shard rather than one per depth pipeline — same length, and now the same reason. `cargo clippy --all-targets -- -D warnings` clean and 1408 tests pass, the same count as before: no test moved, so nothing about the behaviour did.
Task 13. Reversing the shard order in `run.rs`'s composition — the edit that publishes each shard's instruments under another channel instance's sequence series, which no subscriber can detect — passed all 1408 tests. Nothing in the suite reached `compose_and_run`, and the by-hand offline run did not catch it either. Two things hid it, and both are now written where the next reader will find them. The end-to-end harness composes its **own** `Feeds`, shard-outer and block-inner, so those tests assert the harness's ordering rather than the runtime's. And the definition path is keyed on a shard's *name* — `ShardFeeds` derives it from one of its own send paths — while the event path is keyed on its *index*: so a permutation leaves every reference-data port carrying exactly its own shard's definitions, and sends a quote into another shard's pipeline, whose lowering does not hold the instrument, where it is dropped before any wire. `RouteLookup` was not the missing seam. It exists because "a test that needs a route to a multicast group is a test that does not run in CI", and that is true, but `MulticastTransmitter::open` binds and connects a socket — so a composition holding a route still needs a network. What goes behind a trait is the thing that produces a feed's `Ports`, and `KernelPorts` is the real one. `compose_feeds` takes the shard list and the feed list **separately** rather than a `Config`, which is what makes `StartupError::ShardWithNoFeed` reachable: no document and no resolved `Config` can make the two disagree, so the variant guarded a refactor and nothing exercised it. Now one test hands it a shard the feed list does not mention. Four tests over the harness's own recording ports — the order asserted as a whole sequence, because the failure is a permutation and every shard is present in it; a shard's blocks opened together before the next shard's, read off the opener's own record; one era file per channel instance, against a real store in a temporary directory, because a fake would assert the fake; and the refusal. The empty pair is tested directly too, where a document cannot reach it: every method on `ShardFeeds` is written against there being a send path to read a name off, and `new` returning `None` is what makes `name()` total. And `replay.sh` gains the floor it was missing. It asserted definitions and identity and never asserted that a **quote** arrived — which is exactly what a permuted composition produces: definitions intact on every channel, no market data anywhere, 38 messages become 32. It now asserts the kinds per channel, and that every instrument moving is one that channel described.
|
Verified on the assembled tree: workspace tests green with the recorder crates excluded, since they do not build on macOS; The shard routing itself is right. One index resolved per event and both of that shard's pipelines taken from it,
|
There was a problem hiding this comment.
🟡 Changes recommended
There are at least two concrete issues in changed regions (a misleading crate-level boundary statement and an example that can emit invalid zero-length record files on refusal) that should be corrected before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR implements multi-shard publishing (multiple channel instances per feed specification) and the supporting invariants around reference-data partitioning, era persistence, composition ordering, metrics, and adapter boundary changes, plus an additional feature enabling adapters to write upstream on an already-established connection.
Changes:
- Introduces
[[feed]] shard/ShardNameand composes feeds shard-outer to keep routing, refdata partitioning, and sequencing consistent across many channel instances. - Keys eras and refdata state correctly per channel instance (spec × shard), updates snapshot rotation pacing/membership to be shard-scoped, and adjusts metrics to track per-channel instrument counts.
- Adds
Adapter::poll_upstreamand a driver cadence (UPSTREAM_POLL) to support mid-session upstream writes; makes[egress] ttlrequired at config resolution with a dedicated startup refusal.
File summaries
| File | Description |
|---|---|
| rust/recorder/dz-recorder-relower/src/relower.rs | Updates ListingSink implementation to the new list_on boundary. |
| rust/publisher/README.md | Documents shard vs channel-instance semantics and new startup refusals. |
| rust/publisher/dz-publisher-runtime/tests/shutdown.rs | Adjusts runtime tests for shard-aware refdata validity checks. |
| rust/publisher/dz-publisher-runtime/tests/shards_end_to_end.rs | Adds end-to-end tests asserting shard isolation across refdata and routing. |
| rust/publisher/dz-publisher-runtime/tests/harness/mod.rs | Extends test harness to build shard-outer feed composition and recorders. |
| rust/publisher/dz-publisher-runtime/tests/depth_end_to_end.rs | Adds shard-focused assertions for depth routing, resets, manifests, rotation. |
| rust/publisher/dz-publisher-runtime/tests/composition.rs | Adds composition-level tests via PortOpener seam and compose_feeds. |
| rust/publisher/dz-publisher-runtime/src/run.rs | Refactors composition into compose_feeds, adds PortOpener, shard-era handling, tee destination shard keying. |
| rust/publisher/dz-publisher-runtime/src/rotation.rs | Makes snapshot rotation shard-scoped (pacing divisor + membership predicate). |
| rust/publisher/dz-publisher-runtime/src/pipeline.rs | Carries ShardName on pipelines to keep name/index coupled. |
| rust/publisher/dz-publisher-runtime/src/lib.rs | Re-exports shard/composition types and functions. |
| rust/publisher/dz-publisher-runtime/src/error.rs | Adds shard-related startup errors and TTL-required startup refusal. |
| rust/publisher/dz-publisher-runtime/src/builtin.rs | Adds optional shard to built-in listing records. |
| rust/publisher/dz-publisher-runtime/examples/loopback_publisher.rs | Updates example to use shard-aware era and registry config. |
| rust/publisher/dz-publisher-refdata/tests/shards.rs | Adds refdata shard behavior tests (set isolation, ID space, refusals). |
| rust/publisher/dz-publisher-refdata/tests/listings.rs | Updates tests for shard-configured registry initialization. |
| rust/publisher/dz-publisher-refdata/tests/identity.rs | Adds default-shard equivalence test (list vs list_on(DEFAULT_SHARD)). |
| rust/publisher/dz-publisher-refdata/tests/definitions.rs | Updates tests for shard-aware manifest sequencing. |
| rust/publisher/dz-publisher-refdata/src/refusal.rs | Adds shard-specific refusals and clarifies boundary semantics. |
| rust/publisher/dz-publisher-refdata/src/lib.rs | Documents shard partitioning at the crate level and exports ShardConfig. |
| rust/publisher/dz-publisher-refdata/src/error.rs | Adds shard-config validation errors (NoShardConfigured, ShardConfiguredTwice). |
| rust/publisher/dz-publisher-metrics/tests/precreated_at_startup.rs | Extends assertions for channel-keyed series and per-channel instrument count gauge. |
| rust/publisher/dz-publisher-metrics/tests/normative_names.rs | Updates metrics touch suite for new set_instruments_current(channel_id, count) API. |
| rust/publisher/dz-publisher-metrics/src/metrics/refdata.rs | Converts instrument count metric to IntGaugeVec keyed by channel_id and pre-creates per declared channel. |
| rust/publisher/dz-publisher-metrics/README.md | Documents sizing/impact of many declared channel instances. |
| rust/publisher/dz-publisher-egress/src/policy.rs | Clarifies DEFAULT_TTL meaning vs config requirement. |
| rust/publisher/dz-publisher-egress/src/lib.rs | Re-exports Shard and updates module docs for shard-keyed eras. |
| rust/publisher/dz-publisher-egress/src/era.rs | Keys era files by (spec, shard) with default-shard filename preservation. |
| rust/ingress/dz-ingress-core/tests/driver.rs | Adds tests for mid-session upstream writes, cadence gating, refusal semantics, pacing. |
| rust/ingress/dz-ingress-core/src/lib.rs | Re-exports UPSTREAM_POLL. |
| rust/ingress/dz-ingress-core/src/driver.rs | Implements UPSTREAM_POLL cadence and calls adapter.poll_upstream mid-session. |
| rust/adapter/dz-adapter-uds/tests/lossless_through_the_lowering.rs | Updates test sink and listing payloads for shard-aware listing path. |
| rust/adapter/dz-adapter-uds/tests/listings_name_their_shard.rs | Adds tests asserting UDS listings forward the shard (default vs named). |
| rust/adapter/dz-adapter-uds/src/lib.rs | Adds shard: Option<String> to UDS listing and routes via list_on/list. |
| rust/adapter/dz-adapter-uds/examples/write_records.rs | Extends record writer example to support multiple --symbol values. |
| rust/adapter/dz-adapter-core/tests/adapter_is_usable.rs | Updates test listing sink to require list_on and asserts default-shard behavior. |
| rust/adapter/dz-adapter-core/src/sink.rs | Introduces DEFAULT_SHARD and makes list_on required with list defaulting to it; expands boundary docs. |
| rust/adapter/dz-adapter-core/src/lib.rs | Adds crate docs about the breaking list_on change and exports DEFAULT_SHARD. |
| rust/adapter/dz-adapter-core/src/adapter.rs | Adds defaulted Adapter::poll_upstream with detailed contract. |
| docs/superpowers/specs/2026-09-09-upstream-write-after-a-listing-change-design.md | Design doc for mid-session upstream writes. |
| docs/superpowers/specs/2026-09-09-session-transport-design.md | Design doc for a future session transport. |
| docs/superpowers/specs/2026-09-09-recorder-venue-observation-design.md | Design doc for venue-side observation/race support. |
| docs/superpowers/specs/2026-09-09-polled-transport-design.md | Design doc for a future polled transport (poll token). |
| docs/superpowers/specs/2026-09-09-egress-ttl-and-a-boundary-correction-design.md | Design doc for required TTL and boundary doc correction. |
| docs/superpowers/plans/2026-09-09-upstream-write-after-a-listing-change.md | Implementation plan for upstream mid-session writes. |
| docs/superpowers/plans/2026-09-09-session-transport.md | Plan for session transport work (future). |
| docs/superpowers/plans/2026-09-09-recorder-venue-observation.md | Plan for venue-side observation (future). |
| docs/superpowers/plans/2026-09-09-polled-transport.md | Plan for polled transport (future). |
| docs/superpowers/plans/2026-09-09-egress-ttl-and-a-boundary-correction.md | Plan for required TTL and boundary doc correction. |
| docs/README.md | Adds index entries for the new specs/plans. |
| BRINGING-UP-A-FEED.md | Updates bring-up guidance for shards, required TTL, and poll_upstream. |
Review details
- Files reviewed: 59/59 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…lan counts thirteen Three corrections a review found, all of them in what the shards work itself wrote. `RecordWriter::write` appends nothing when it refuses, and the example wrote the buffer anyway — leaving a zero-length `.record` a reader cannot decode. A refusal is meant to cost that record; it was costing the replay. The numbering gap that skipping leaves costs nothing: the order is what the names carry and a reader takes the files that exist. The plan's Scope said twelve tasks while it carries thirteen, so a reader tracking progress stopped one short of the task that put the composition under test. And the row rejecting `category` attributed a definition and a quotation to an external glossary that contains neither — `Category` is not defined anywhere in that repository. The argument does not need a citation: the word describes what a market is about, and what is being named is the opposite of a taxonomy. `PortOpener`'s doc states the property rather than the run that found it.
df4d125 to
b86300e
Compare
… names the shard
Three points a review found, none of them about shards being wrong.
**`Shard::resolve` had no caller and `ShardName::era_shard` reimplemented it.**
The point of that method is that the mapping lives in exactly one place, and it
lived in two. `era_shard` delegates now: `resolve` takes the default token as an
argument because `dz-publisher-egress` does not depend on the crate that owns
the constant, so the decision belongs there and supplying the token is this
method's job. The mapping is what a renamed era file costs — it reads as no
file, resolves to the first era, and a publisher on era 7 restarts on era 1
announcing nothing — and it now has a direct test.
**Four `.expect("checked above")` on the send path are counted refusals.** The
guard answers `true` for an instrument on no shard deliberately, so the lowering
refuses it as an unknown instrument, which is the better diagnostic. What that
left is a send reached with an index the send paths may not hold. It survives in
production only because the lowering opens with `instruments.get(instrument)?`
and the registry clears that table and the slot together — an invariant in
another crate that nothing at the send site stated. The pipeline is resolved
before the send now, and a miss is counted.
**`DuplicateChannelId` names each block's shard.** The specification alone
stopped identifying a block the moment two of them could carry one, and two
blocks of one specification on different shards is the likely shape of the
mistake: the message printed the same word twice and identified neither.
… it claimed
The revert was run and the mutant survived. Restoring `expect("checked above")`
left the new test passing, because the guard answers `false` for this state and
returns before the lowering — so the test exercises the guard, not the site that
panicked. It counted the right number for the wrong reason.
The test now claims the guard, which is what it asserts, and its doc states
what has no test and why. Reaching one of those sites needs the guard to answer
`true` *and* the pipeline to be missing, which happens only for an instrument
whose shard is `None` and whose lowering succeeds — and the registry owns the
lowering's instrument table and clears it with the slot, so the two cannot
disagree through any public path.
So the four refusals replace a panic that was unreachable: a latent hazard
removed rather than a behaviour changed, and the reviewer's point stands on the
invariant being stated in another crate rather than on a reachable crash.
|
All five, and the two that are decisions came back as you argued them. 1. Scope — split. This PR now carries the shards work alone. One thing the split turned up: taking 2. Two mandatory configuration edits in one window. Agreed, and #100 is now separable: it can wait behind the shards without holding them up. The 3. 4. The four Reaching one of those sites needs the guard to answer 5. Gates on this head: |
…rocess (#97) Documents only — a design and its plan, no code. The publisher emits one channel per feed specification and refuses a second block of one; this is the document that answers what a venue does when one channel cannot carry its instrument set. ## The question it starts from A `[[feed]]` block is a channel, and `DuplicateFeedSpec` refuses two blocks naming one specification. So a venue whose top-of-book set outgrows one channel has no document to write: the publisher it would need is 62 channel instances from one process — 31 shards of two specifications — and the gate that stops it is one check. Lifting that check alone produces a publisher that starts and is wrong in four places at once, which is why this is a design and not a patch. ## What the spec settles - **The key is `shard`**, and the Naming section argues down three better-sounding words. `route` is taken twice over — `[egress]` already means the IP route by it, and `GLOSSARY.md` defines `path` as "one of several redundant routes carrying the same data", which makes "62 routes" say the opposite of what is meant, since these carry **disjoint** sets. `category` is the feed registry's own field and explicitly "an arbitration boundary, not a taxonomy". `lane` is banned outright. `shard` is the glossary's own word inside its definition of a channel: "a logical shard of the instrument set, named by `Channel ID`". - **A shard is the most a venue may say.** `ListingSink::list_on(shard, spec)` is required and `list` is defaulted, in that direction on purpose: defaulting `list_on` would leave an adapter that had not been updated admitting its whole universe to one channel with no error, no counter and no log. Which `Channel ID`, group, port, sequence series and era a shard resolves to stays the operator's, and is unreachable from the adapter. - **Two units that are not the same thing.** The shard is the unit of reference data — one published set, one `Instrument Count`, one `Manifest Seq`, one definition cycle. The channel instance is the unit of sequencing — its own sequence series, `Reset Count` and era. Everything in the plan turns on those being different. - **The four fences**, each a place where the obvious implementation is wrong: the gate is a gate and not a check; the two typed `Option`s stay typed because the datagram path must not dispatch dynamically; the era store is a latent state-corruption bug rather than an inefficiency, and not the one it looks like — `begin_era` persists per call and `run.rs` calls it per block, so N blocks take N eras, the era a channel advertises depends on its position in the document, and moving a block hands a channel an era it has already published under; and the single drain, without which any admission anywhere becomes an N-channel definition burst. - **Three fences beyond the four the request named**: the reference copy keys its socket on `(spec, port role)` and so cannot attribute two shards' copies; the snapshot rotation divides the cycle by the process's published count; and nothing checks `Channel ID` uniqueness across blocks — which two blocks sharing an ID would turn into two channel instances silently writing one set of metric series. - **What it costs, before somebody discovers it**: the pre-created channel-keyed exposition goes from 12 series to 372, exactly 31-fold and entirely at startup, plus one family that gains a `channel_id` label because it mirrors a per-channel wire field; a longer document; and a `u8` `Channel ID`, so 256 is the ceiling. ## What it refuses - **Reference data per shard is not optional.** The specification defines `Instrument Count` and the definition cycle per channel, so a process-wide published set is a wire-level lie however convenient the code would be. - **One registry, N published sets** — not N registries. N would be N `Instrument ID` spaces under one `Source ID` and N writers on one state directory. - **The shard is an index on the registry's published entry**, not on the instrument table, which would drag the lowering and the recorder's offline re-lowering into a change that buys them nothing. - **`EventSink` is untouched** — the trait, its methods and its parameters. A task that changes one has moved routing into the adapter. ## The plan Twelve tasks with an ordering constraint that is the whole shape of it: tasks 1 through 8 are behaviour-neutral because the duplicate-specification gate stays in force, so with one shard configured every partition is a partition of one and the suite passes unchanged. Task 9 lifts the gate, and it is the change that makes 3, 4 and 7 load-bearing rather than latent. Two global constraints worth reading before the code: every test must be shown to kill its mutant — revert, watch it fail, restore — because several of these tests have an absence as their subject, and `GLOSSARY.md` governs every identifier, comment, test name, config key and commit message. The default shard is the plan's quietest risk and it says so: it keeps `<spec>.era`, because a renamed era file reads as *no file*, which resolves to the first era — a publisher on era 7 restarts on era 1 and announces nothing, which is the corruption the corrupt-file refusal exists to prevent, delivered by the upgrade meant to be safe. ## The plan carries its own execution now This branch is no longer only the argument. The plan's task boxes are marked, and it carries two sections written after the code ran rather than before: - **It ran** — four channel instances of one specification through `run()`, over real multicast sockets, read by four instances of this repository's Go subscriber, twice, with the output recorded as the run printed it. It also states what could **not** run and why: the built-in record adapter holds no book, answers no snapshot, and `AdapterRegistry` refuses it beside a `market-by-price` block, so the two-specification arrangement the plan asked for is refused at startup and a real depth run needs a venue with a book. - **The four reverts the plan demands** — a table naming which test died under each. It earned its keep: the revert that failed nothing was task 5, because the reference-copy fan-out had never been keyed on the shard at all. Reading the marked plan against #98 is the intended order: this says what was decided and what happened, that one says how. ## Testing Verification No code, so the checks are the ones documents can fail: `scripts/check-public-repo-rules.sh` exits 0; every referenced path, type, method and error variant was opened and confirmed against source rather than assumed — `FeedSection`, `DuplicateFeedSpec`, `ListingSink`, `Registry`, `RegistryConfig`, `EraStore::begin_era` and its two call sites, `TeeConfig::destination`, `Feeds`, `SnapshotRotation`, `channel_ids`, `NORMATIVE_NAMES` — and the vocabulary was read against `GLOSSARY.md` 1.3.0, whose Naming table is quoted rather than paraphrased. The implementation is #98, stacked on this branch so that this diff is the argument and that one is its execution.
nikw9944
left a comment
There was a problem hiding this comment.
Two blocking items. A venue naming an unconfigured shard is dropped with no metric and no log — take_unknown_shards has no caller — and per-shard ManifestSummary breaks the recorder's completeness check on multi-shard captures. The era-mapping duplication and the DuplicateChannelId message are verified fixed in a6aaad5; the [egress] ttl and poll_upstream findings are dropped because that code left this branch with the split.
- Optional —
dz-publisher-runtime/src/lib.rs:64-68still saysManifest Seqdescribes the published set rather than a channel, so one composed manifest is truthful on every refdata port. This change makes it per shard.
Docs & comments (non-blocking)
rust/adapter/dz-adapter-core/src/lib.rs:93— The crate docs still say "every implementor of it is in this workspace", andsink.rs:204repeats it. This is the page an external implementor reads to decide whether the trait break applies to them.
The plan is one document and both branches carry it, so the correction to task 2's bullet — the break is signalled in the tag message because there is no changelog and no per-crate version — reads the same from either side.
…them Two claims in these documents are not true of the tree they describe, and one task is a tier this change reaches and the plan never mentioned. Corrected first, so the code that follows is measured against documents that say what is actually there. **Task 2 promised a log line and delivered half of it.** The registry remembers each unknown shard name once and hands it to a caller; nothing calls it. The checkbox is split in two, the second half is unchecked, and task 14 is the caller. The design's justification for leaving the two counters mapped to no metric family is corrected with it: the gauge it points at covers a venue that misnames *every* offer for a shard, and a venue that misnames some of them leaves the gauge non-zero with those instruments unpublished. For that case the line is the only signal there is, which is what makes its absence a defect rather than a missing nicety. **The snapshot ceiling stopped being detectable when the divisor became per shard.** `rotation.rs` states it as a cycle divided by one shard's count falling below the runtime's tick. The process still serves at most one snapshot per 10 ms tick, so N rotations draw on one budget: 31 shards of 100 instruments on a five-second cycle each derive a 50 ms tick, five times the process tick and comfortable by that statement, while the process is asked for 620 snapshots a second and can send 100. The spec gains the sum the condition actually is, and task 16 says why it is counted rather than refused — at load there is no published count to divide. **A per-shard `ManifestSummary` reaches the recorder.** `Instrument Count` and `Manifest Seq` are the channel's, which is the specification's definition and the point of task 3; `dz-recorder-relower` kept one manifest per archive and compared it against the union of every channel's definitions. Nothing on the wire moved, so no subscriber is affected, and the publisher side is not the thing to fix — a process-wide count would be false on every channel it went out on. Task 15 moves the check to the channel and argues why the channel rather than the channel instance.
`take_unknown_shards` collected the distinct names and nothing called it, so a
venue naming a shard this document has no channel for had those instruments
declined with nothing an operator could see. The mechanism was task 2's; the
caller was not written.
`Publisher::take_unknown_shards` drains the registry and the tick loop writes
one line per distinct name, beside the lines it already writes for a dropped
fan-out member and a refused snapshot. The line carries the name offered **and**
the names configured, because a misspelling is only visible as the pair, and it
says the instruments were declined rather than leaving that to be inferred. The
exit report gains the two counts.
**The two counters stay mapped to no metric family.** The normative set is
closed by the playbook, this repository does not own it, and a series added here
would be one nobody declared. What is corrected is the justification. `Counts`
said the signal already existed as `refdata_instruments_current{channel_id}`
sitting at 0 for that shard's channel — true only where the venue misnames
*every* offer for a shard. One that misnames some of them leaves the gauge
non-zero with those instruments unpublished, and no series in the closed set
tells that from a channel holding fewer instruments. For that case the line is
the only signal there is, which is what makes writing it obligatory rather than
decorative.
Asserted:
`a_shard_the_document_has_no_channel_for_is_named_once_however_often_it_is_offered`
covers the drain — two instruments under one misspelling produce one name, and a
second poll re-offering both produces none —
`the_unknown_shard_line_names_the_offer_and_the_configured_shards` and
`a_publisher_with_no_named_shard_still_names_what_it_has` cover the line.
**The `eprintln!` itself has no test that can fail**, and this is said rather
than covered: no test in the workspace runs `tick_loop` and none captures
stderr. The drain was put on `Publisher` for that reason — it is the furthest
along that path a test reaches.
…ll is counted The rotation divisor became per shard, which is right and is what makes each channel's `[[feed]] snapshot_cycle` mean what it says. The serving rate did not: the tick body takes at most one periodic snapshot, so N shards draw on one budget of one per 10 ms tick while the demand adds up. `rotation.rs` stated the ceiling as a cycle divided by *one shard's* count falling below the runtime's tick. That was the whole of it with one rotation. With N it misses the case shards introduce — 31 shards of 100 instruments on a five-second cycle each derive a 50 ms tick, five times the process tick and comfortable by that reading, while the process is asked for 620 snapshots a second and can send 100. Every channel laps in 31 seconds and `snapshot_cycle` reads as honoured. The note now states the condition as the sum it is, `schedule_share` is that term for one shard in integer arithmetic against `WHOLE_SNAPSHOT_CAPACITY`, and `Publisher::tick` sums it and counts the ticks it cannot meet. A shard with nothing published asks for nothing rather than for `tick`'s clamped set of one, so a long document of empty channels does not add up to a shortfall nobody is having. `run::TICK` becomes public because it is the serving rate: it is the one number in the tick body that depends on the caller's interval, and it has to be, since every other cadence there is a debt read off the clock. **Counted, not refused.** The divisor is the published count, so at load there is nothing to divide; a refusal on the first tick that could compute it would darken a publisher that is already sending, over a shortfall that degrades into a slower lap and never into a wrong answer, and whose remedy is a configuration edit an operator has to be told about rather than one the process can make. The tick loop names it on the decade schedule `worth_a_line` already states and the exit report names the total. Asserted: `every_shards_derived_tick_can_be_comfortable_while_the_process_is_not` and `one_shard_below_the_process_tick_is_still_caught` on the arithmetic, `a_share_is_the_process_tick_over_the_shards_own_derived_tick` and `a_shard_with_nothing_published_asks_for_nothing` on its two edges, and `cycles_that_are_achievable_per_shard_and_not_together_are_counted` on a publisher — with `cycles_the_process_can_serve_are_not_counted` as the control, without which the count passes against a counter that increments on every tick.
`ManifestSummary`'s `Instrument Count` and `Manifest Seq` describe one channel's published set. `ArchivedRefdata` kept one manifest for a whole archive, chosen by highest `Manifest Seq`, and `finalise` compared it against the union of every channel's definitions. Over a capture of one channel that is the same comparison; over a capture of two there is no arrangement in which it is right. Both directions of the error are reachable at once — a complete channel reports `ReferenceDataIncomplete` because the union exceeds its count, and a genuinely short channel with the lower `Manifest Seq` reports nothing — and which one is seen depends on nothing but how the archive is arranged. The publisher side is not the thing to fix. A process-wide `Instrument Count` would be a false statement on every channel it went out on, which is the whole point of the change this reviews. So: a manifest per `Channel ID`, the symbols each channel defined beside it, and `finalise` a loop comparing each channel's count against its own definitions. `Caveat::ReferenceDataIncomplete` names the channel — without that, two channels short by the same numbers are one value and `push_once` collapses them into one line identifying neither. `declared_instrument_count` takes a `Channel ID`: the counts of several channels do not reduce to one number, and their sum is the size of a union of disjoint published sets, which no manifest states. The symbol table stays one for the archive, because the re-lowering resolves a symbol without knowing which channel carried it, and splitting it would mean resolving against a channel a payload archive cannot name. **Per channel, not per channel instance.** `by_symbol` already unions the redundant paths of a channel and raises `ScaleRestated` where they disagree, so the reconstruction this check guards resolves from that union; keyed on the instance it would raise a caveat against a path whose refdata window was shorter even where the union covers the set and nothing is declined — a statement about capture coverage dressed as one about the archive. The key is the datagram header's `Channel ID` and not the summary's own field, because the header is the only channel identity a definition has, and keying the two halves of one comparison on two different fields would compare sets that are not the same channel's. **Not deferred**, having considered it: the honest alternative was to call this its own piece of work and meanwhile refuse to run over an archive covering more than one channel. That is right where the fix is large. Both call sites already hold the `Channel ID`, the map is keyed on a `u8` and `finalise` becomes a loop, so refusing would have cost a public error variant, a runner that stops on captures it can read perfectly well, and the same work again later. Asserted: `each_channels_manifest_is_checked_against_its_own_definitions` over an archive whose complete channel holds the higher `Manifest Seq` and whose short one holds the lower, so the union comparison is wrong on both channels at once; `a_channel_with_no_manifest_of_its_own_borrows_no_other_channels_count` for the channel nothing can be said about. `pack_on` and `refdata_datagrams_on` are what let a fixture hold two channels at all — every fixture published on one until now, which is why the suite passed over this.
#97 landed the same spec and plan from a head that predates two corrections, so both files arrive as add/add conflicts. This branch's copies are the later ones and are resolved as such — every line `main` carries is present here, replaced rather than dropped: - the task count, thirteen against the sixteen this branch now has; - the bullet claiming the break is recorded in a changelog, corrected here to the tag message and the release notes, because there is no changelog in the tree and no crate carries its own version; - the promise that an unknown shard name is *logged* once per distinct value, which is now *remembered* once per distinct value with task 14 being what reaches an operator with it; - the four-revert heading, which is a longer table here. Verified line by line rather than assumed, and the merged tree is green: 1434 passing, fmt and clippy clean, and the repository rules pass.
bgm-malbeclabs
left a comment
There was a problem hiding this comment.
Approve on correctness. Three notes, none blocking.
Per-tick cost grows with shards times instruments. Registry::definition_tick walks self.slots — the whole process instrument table — once per shard per tick and keeps only that shard's entries. SnapshotRotation::due does the same through on_shard. At the number this PR is named for, 31 shards of two specifications, that is 31 walks of a shared table every 10 ms. It is correct and it is bounded, and the comment in the emit loop is honest about it: "Every slot is walked and only this shard's are emitted." Worth a measured number before 62 channel instances go out, since the count is the whole point of the change.
ShardConfig.channel_id is whichever block came first. In run.rs:
channel_id: config.feeds.iter().find(|feed| feed.shard == shard)A shard carrying both specifications has two channel ids, and this takes the one whose [[feed]] block sits earlier in the document. registry.rs argues why that is harmless — the datagram builder stamps the header at push, and one composed summary is truthful on both refdata ports — and I believe it. The call site does not say the choice is arbitrary. One line of comment there, because the next person to read find will read it as the channel id.
The era file naming holds. {feed}.{shard}.era cannot collide with another feed's {feed}.era, because is_one_path_component rejects . in both names. ShardName::era_shard is the single place the default token maps to Shard::DEFAULT, which is the mapping that most wants to be in one place: a renamed era file reads as no file, resolves to the first era, and the publisher restarts on era 1 saying nothing.
Two things I would copy elsewhere. Closing the channel_id collision that already existed, rather than only the one this change makes likely. And finding task 5 by running the plan's own revert clause instead of assuming it: the revert that failed nothing was the task nobody had written, which is the only way that class of miss gets found.
elitegreg
left a comment
There was a problem hiding this comment.
Approving. I re-verified every fix claimed on this PR against the head rather than reading the replies, and the substantive ones hold up.
schedule_share and WHOLE_SNAPSHOT_CAPACITY are the right answer to the snapshot-budget finding: the per-shard divisor made every shard's arithmetic locally comfortable while the process-wide serving rate stayed at one per tick, and count_snapshot_schedule now measures the sum against what one process can send. Making it a counted overrun rather than a startup refusal is also right, since published counts move at runtime. The exit line naming the shared budget is what an operator would actually need.
DuplicateChannelId carrying first_shard and second_shard fixes a message that would otherwise have read "the blocks for top-of-book and top-of-book" in exactly the case the refusal exists for. take_unknown_shards has a caller now, and both shard counters get exit lines, which closes the gap where a venue misnaming a shard had no signal at all.
Four notes, none of them blocking. One is bookkeeping rather than code, and two are mine and previously unraised.
`ListingSink::list_on` justified its version bump with "every implementor of it is in this workspace", in two places: the crate-level documentation and the trait's own. Implementors outside this workspace exist — a test double for a venue's own adapter implements the sink even where the adapter under test only calls it — so the claim was untrue in both copies, and a claim stated twice is a claim corrected once. The correction itself was written on `jo/egress-ttl-and-a-boundary-correction` and moves here, because the sentence is in this branch's diff and this branch lands first. The review thread that raised it had been resolved against a commit a later rebase removed, so nothing on this head pointed at a fix and the audit would have read as answered. What the trait chose is unchanged and the wording does not pretend otherwise: what makes the break worth asking for is not that nobody pays it but where it lands. A compile error in a test double is found by the next `cargo test`; a published set silently collapsed onto one channel is found by a subscriber. Documentation only. There is no behaviour here and no mutant to kill.
…k's ceiling Two things a review found in what the shards work wrote, both documentation. `PortOpener` was inserted directly above `open_ports`'s doc comment with no blank line between them, so the whole port-role argument — why a dead refdata socket or a dead snapshot socket ends the process rather than degrading — became the trait's documentation, and `open_ports`, which moved below `compose_feeds`, was left with none. The trait's rendered docs opened with three paragraphs about socket failure scope and then started again at its own summary. The block goes back onto the function it argues about; the trait keeps its summary and the two paragraphs on why `RouteLookup` was not the missing seam. `Registry::definition_tick` now states the ceiling `rotation.rs` already states for the rotation. Both walk the same shared slot table for the same reason — the slots are one table because an `Instrument ID` is — and someone sizing a shard count would have read one and not the other. It is written down rather than removed, and the arithmetic is why: the cursor persists across ticks and a shard emits its whole published set once a lap, so a lap is exactly one revolution of the table, `N * slots` visits a lap over the process. At 31 shards of 100 instruments on the default 30 s cycle that is 40 slot visits a tick against a 10 ms tick, and the walk is entered only on the one tick in twenty-four a shard owes anything. Measuring it found the term that is actually larger, which is not the one the review named. `shard_index` resolves the shard name by scanning the configured shards and comparing strings, and it runs before `due` is consulted, so it costs N comparisons on every tick rather than on the ticks something is owed. Timed over a whole lap, 31 shards of 100 cost 1.3 microseconds a tick with 1.1 of that the name scan; 128 shards of 100 cost 18.6 with 15.0 the name scan. Both are far enough inside a 10 ms tick to be stated rather than optimised, and `Channel ID` being a `u8` caps any document at 256 shards regardless. The note says which of the two to take first if either is ever taken, so that the next reader does not buy the smaller one. `rotation.rs` gains the pointer back, so the two ceilings are found together. Documentation only. No behaviour changes and there is no mutant to kill; the 1434 tests pass unchanged.
… it is now `lib.rs` still said `Manifest Seq` describes the published set rather than a channel, and concluded that one composed manifest is truthful on every feed's refdata port. That was true before the shards work and is false after it: `Registry::manifest_seq` and `Registry::published_on` each take the shard the channel carries, so both numbers are a channel's. The paragraph now splits the two claims that used to sit in one sentence. `Instrument ID` identity really is the one thing there can only be one of, and that half is unchanged. The published set is not one, and the reason is the one the recorder's completeness check needed: summing the counts over shards states the size of a union of disjoint published sets, which no manifest declares and no subscriber sees. A manifest is truthful on the refdata port it leaves by, rather than on every one of them. Documentation only. No link was added, so the crate's existing rustdoc warnings are unchanged.
|
@nikw9944 — both blocking items are fixed and verified on Blocking 1 — a venue naming an unconfigured shard was silent. Fixed in Blocking 2 — per-shard The fix is on the recorder's side, not the publisher's — a process-wide Minor — the rotation divisor. Docs — both clauses gone, in Optional — Gates on |
**Stacked on #98**, which carries the shards work; this diff is the upstream write. Split out of #98 at a reviewer's request: it changes behaviour on its own and should not have to be approved in order to approve the shards. ## The failure, which is measured rather than argued `UpstreamSink` reaches an adapter in exactly one method — `on_connected`, called on every successful connect — and `poll_listings` is handed a `ListingSink` and nothing else. So an instrument discovered mid-session can be admitted completely: minted an `Instrument ID`, defined on the reference-data port, counted in the manifest, given a `Manifest Seq` bump. And never subscribed, because the subscription was composed at logon from the set the adapter held then. Nothing reports it. The feed is healthy, the manifest says the instrument is published, and no message for it will ever arrive. Over thirty days on one venue, 5 instruments were listed in two mid-session batches. A publisher reading over a websocket carried all five inside one 17-day process lifetime, because its transport re-subscribes on its own poll. A publisher of the same venue reading over a session transport carried none of them until its next restart — 10.35 and 7.32 days later. Same boundary, same instruments; the difference is whether the transport happened to have a reason of its own to write again. ## Why neither suggested shape works Two were offered: a second argument to `poll_listings`, or a defaulted `listings_changed(&mut dyn UpstreamSink)`. Both are refused for one reason that is fatal and one that follows from it. **Neither names the connection, and the write has to.** `on_connected` already argues this at length, because it is where the same mistake was found before: one adapter serves every source a publisher opens, so state that belongs to a connection has to be stored per `conn` — "an adapter that keeps one upstream sequence cursor, or one authentication token, or one 'have I subscribed yet' flag, is correct with one source and wrong the moment a second is configured — and the way it is wrong is silent". An `UpstreamSink` with no `ConnectionId` beside it is exactly that flag, handed to the adapter by the boundary. **And there is nowhere to send it from.** `poll_listings` runs in the runtime's tick loop, which holds the adapter and no transport: the drivers own the `Input`s, one per source. A sink handed to the tick loop would be a queue whose consumer is *some* driver, and picking one is picking the connection the adapter was supposed to name. ## The shape One defaulted `poll_upstream(conn, out)` on the boundary, asked by each driver on its own connection when the cadence is due, flushed through the same rate limit a logon pays. Named for the mechanism rather than for the listing change, because the repair path is a second cause — a mid-session snapshot request had nowhere to go either — and a venue's own reasons are a third. **No reason parameter**: the adapter is the only thing that knows what it has already written on that connection, and a reason would invite state keyed on the reason rather than on `conn`. **The runtime does not change. Not one line.** `poll_listings` admits, the adapter records the handle, the driver asks, the adapter writes what it has not written on that connection. **This write is not deduplicated, and the doc comment says so in those words.** Re-offering a listing is free because the sink returns the handle already minted; this reaches a venue, and nothing here understands a venue's bytes. Getting it wrong looks from here like a venue rate limiting a publisher that is working perfectly. **Two asymmetries with `on_connected`, both argued.** A refusal is counted and the connection survives, because mid-session it is still delivering everything subscribed at logon and dropping it would cost those to save the one. A failed send ends the connection through the path that already exists, and the reconnect writes the whole set again, so nothing has to be reconciled. `UPSTREAM_POLL` is five seconds and deliberately slower than the runtime's one-second listing poll: that one is affordable by construction, and this one is a venue's inbox. Its doc also states what it does **not** do — fire on a silent connection. Bounding the receive budget by the cadence would have made an elapsed budget mean two things, and `Received::Idle` means exactly one. ## Testing Verification - `cargo test --workspace` — **1418 passing, 0 failed** on this head (the shards base is 1413; five tests are new here), over the scripted transport and the test clock. - `cargo fmt --all --check` and `cargo clippy --all-targets -- -D warnings` clean. `scripts/check-public-repo-rules.sh` exits 0. - **The centre revert corrected its own test.** Moving the call to the connect path, beside `on_connected`, failed nothing on its first run: the test asserted "asked once, on this connection, and the message reached `send`", every word of which a call at logon satisfies. It now records how many payloads the adapter had seen when it was asked and asserts one; a connect-time ask records zero. With that, the revert fails three tests. - A second revert — dropping the due-ness check and asking on every receive — fails two, because the count is then per payload.
Stacked on #97, which carries the design and the plan; this diff is the code. All twelve tasks.
Sixty-two channel instances from one process — 31 shards of two feed specifications — where the publisher emitted one channel per specification and refused a second block of one. Lifting that refusal alone would produce a publisher that starts and is wrong in four places, so tasks 1 through 8 land behaviour-neutral with the gate still in force, and task 9 lifts it.
Tasks 1–8, behaviour-neutral by construction
With one shard configured every partition is a partition of one, so the existing suite is the gate on this half.
The document.
[[feed]] shard, absent resolving to the default. A checkedShardName— one lowercase path component, at most sixty-four bytes — because the name becomes a path in two places and the third use is the one that forgets. Spelling the default explicitly is refused: two spellings of one shard are two era files.feed_specs()is distinct now, because it answers which feeds does this publisher emit and an adapter handed one specification once per shard is being told about the deployment instead.A refusal that closes a hole that already existed. Two blocks sharing a
channel_idloaded fine:channel_ids()sorts and dedups, so one set of metric series was pre-created and two channel instances wrote to it with nothing saying so. One channel per specification made it unlikely; many make it a matter of time.The boundary.
list_onis required andlistis defaulted, in that direction on purpose — the reverse lets an adapter that had not been updated admit every instrument to the default shard, every shard collapsed onto one channel with no error, no counter and no log. Acompile_fail,E0046doctest pins it.dz-adapter-coretakes the major version that costs.The registry. One registry, N published sets — N registries would be N
Instrument IDspaces under oneSource IDand N writers on one state directory. The shard is an index on the registry's own published entry, not on the instrument table, which would drag the lowering and the recorder's offline re-lowering into a change that buys them nothing.The era store. Keyed on the channel instance, and the default shard keeps
<spec>.era. That last clause is the whole risk: a renamed file reads as no file, which resolves to the first era, so a publisher on era 7 would restart on era 1 and announce nothing.ShardName::era_shardis the one place that mapping is made, because the natural thing to write at a call site is the wrong thing.The send paths. Two typed vectors indexed by shard, built shard-outer, with the index resolved once per event beside the instrument lookup — no string comparison on the datagram path and no dynamic dispatch.
EventSinkis untouched: the trait, its methods and its parameters.The rotation. Divided by the shard's own published count, and walking only its own instruments. Fixing the divisor alone leaves the same 1/N error arriving through the cursor, because the slots are one table for the process.
Task 5, which the tasks-1-to-8 commit had missed
TeeConfig::destinationwas still<path>.<spec>.<role>, so four channel instances fanned out to five sockets and two shards' copies of one role arrived on one — datagrams a recorder cannot attribute without decoding, the one thing a record path does not do. It now carries the shard, absent for the default, for the era file's reason: what a call site naturally writes isshard.as_str(), and that moves the socket every existing deployment's recorder is bound to.It was found by the plan's own acceptance clause — revert each of tasks 3, 4, 7 and 9 and watch a named test fail. The revert that failed nothing was the task nobody had written.
Task 9, the gate
DuplicateFeedShard { spec, shard }keyed on the pair,ShardSpecsDisagreenaming the shard and the specification it has no block for — the check that makeslist_ontotal — andrun.rscomposing one pipeline per block instead of assigning into anOption.The end-to-end test is the plan's centre, and every assertion in it is an exclusion: shard A's reference-data port carries A's definitions passes against a publisher that packs everything onto everything, so each test states what a channel did not carry beside what it did. Two instruments per shard, not one — with a single instrument each, a packing publisher still puts exactly one symbol on each port, the failure reads as "carries none of its own", and the exclusion is never reached.
What an operator has to change:
dz_publisher_refdata_instruments_currentgains achannel_idlabelCalled out on its own rather than in the cost list, because it is the one thing here that changes the meaning of a series an operator may already be selecting.
Instrument Countis a channel's number and not a process's — it is what that channel's manifest states and what a subscriber bound to that channel reconciles against. So the gauge reporting it is now one series perChannel IDinstead of one for the publisher. Left process-wide it would write the sum of every shard's published set to a series read as the count on the channel I am bound to: on 31 shards, thirty-one times what any one subscriber will ever receive a message for.The break is in the selector, not in the value. A panel or alert selecting
dz_publisher_refdata_instruments_currentwith no aggregation was selecting one series and now selects one per channel; a rule that assumed a scalar gets a vector on the deploy that carries this, with nothing failing to announce it. A single-shard publisher gets one series where it had one and the number is unchanged — but the selector still has to tolerate a label it did not have.What to write instead:
dz_publisher_refdata_instruments_current{channel_id="3"}, which is the question this series was always being asked and could not answer;dz_publisher_instruments_published, which is unlabelled, still exact, and does not need asum()over a label whose cardinality is a configuration choice.The gauge is pre-created at 0 for every declared
Channel ID, and unlike the two manifest gauges it is not gated on therefdataport role: a channel that exists and holds nothing is the case worth alerting on, and a series that only appears on first use cannot carry an alert for a thing never happening.Its
HELPtext is corrected in the same commit. It read "Instruments currently known to the upstream reference-data source", which the call site had never matched — the value has always been the size of the published set, which is smaller wherever the selection policy declines. Nothing about the number changes; the string now describes it.Nothing else in the reference-data set moves.
dz_publisher_refdata_definitions_emitted_totaland the listing counters stay process-wide on purpose, because they are rates and a rate over channels aggregates honestly.manifest_seqandmanifest_validalready carriedchannel_id.Task 12 — it ran, and one thing it could not
examples/replay.shis four[[feed]]blocks of one specification throughrun(), over real multicast sockets, read by four instances of this repository's Go subscriber, twice — because an era is only observable across a restart. One block names no shard, so the run covers the upgrade as well as the feature.Each subscriber read its own
Channel IDand no other, its own sequence series from 0, aReset Countequal to its own era file, and exactly its own shard's two definitions. Four era files, the default shard's stilltop-of-book.era, each advanced by one rather than by four.What could not run. The plan asked for two shards and both specifications. The built-in record adapter holds no book, answers no snapshot, and
AdapterRegistryrefuses it beside amarket-by-priceblock rather than publish deltas a mid-session joiner can never resynchronise. That refusal is right and predates this plan; what it means is that a real depth run needs a venue with a book and this repository has none. The bullet in the plan says so now instead of asking for something refused at startup, and the depth path's shard routing stays covered bydepth_end_to_end.rsandshards_end_to_end.rsagainst recording sinks.Two things only the real run could find, both recorded in the plan: a document that came out naming
multicast_group = "1000", becauseGROUPSis bash's own array of the invoking user's group ids and assignments to it are ignored — refused by name at load, which is the failure this publisher is built to produce rather than start on; and task 5 above.Vocabulary
GLOSSARY.md1.3.0 replaces the word this repository uses for the reference copy withfan-out, and the plan's constraint governs test names and identifiers, not only prose. The concept now takes the glossary's word throughout what this change wrote: three test names, the examplepatha reader copies out of the doc comment, and every identifier and comment inreplay.sh.Deliberately not moved, because each is a change with its own migration: the
[adapter.tee]config key,TeeConfig,Tee,ReferenceStreamand theStartupErrorvariants named after them. The two test names that describe the key keep it, since naming a key is not using the word. The design document's blockquote of the old doc comment stays as written — it records what the comment said the day the design was written.Split, at a reviewer's request
This PR carried five changes and now carries one.
[egress] ttlbecoming required is #100 andpoll_upstreamis #101, each stacked on this branch with its own spec and plan, because each changes behaviour on its own and neither should have to be approved in order to approve the shards. The three design documents for work that is not built — the venue half of a feed race, a polled transport, a session transport — are onjo/venue-boundary-designswith no PR yet.What stays here is the shards work and the corrections a review found in what it wrote: the record example writing no file for a refused event, the plan's task count, and a naming-table row that cited a glossary definition which does not exist.
What a review round added after the split
The clause at the adapter boundary is corrected here, not in #100.
ListingSink::list_onjustified its version bump with "every implementor of it is in this workspace" — indz-adapter-core's crate documentation and in the trait's own doc comment — and implementors outside this workspace exist, a venue's own test doubles among them. Both copies are in this diff and this PR lands first, so the correction moved here from #100 rather than arriving a merge behind the sentence it corrects. #100's spec, plan, title and body say where it went; #101, which branches from here as a sibling of #100, now inherits the corrected text instead of the stale text.The thread that raised it was resolved earlier against
df4d125, a commit the split of this branch into #98/#100/#101 rewrote away. Nothing on this head pointed at a fix and an audit of "were the comments addressed" would have read as answered.open_ports's doc comment is back onopen_ports. ThePortOpenertrait was inserted directly above it with no blank line, so the whole port-role argument — why a deadrefdatasocket or a deadsnapshotsocket ends the process rather than degrading — became the trait's documentation, and the function, which moved belowcompose_feeds, was left with none.The definition tick's per-tick cost is written down where the next reader meets it.
rotation.rsalready states that a rotation walks past the slots of shards that are not its own;Registry::definition_tickdoes the same walk over the same shared table and had no such note, so a shard count was sized against one and not the other. It is stated rather than optimised, and the arithmetic is the reason: the cursor persists across ticks and a shard emits its published set once a lap, so a lap is exactly one revolution of the table —N × slotsvisits a lap, which at 31 shards of 100 instruments on the default 30 s cycle is 40 slot visits a tick against a 10 ms tick, and the walk is entered only on the one tick in twenty-four a shard owes anything.Measuring it found the larger term, which is not the one the review named.
shard_indexresolves the shard by scanning the configured shards and comparing strings, and it runs beforedueis consulted — so it costs N comparisons on every tick rather than on the ticks something is owed. Timed over a whole lap: 31 shards of 100 cost 1.3 µs a tick, of which 1.1 µs is the name scan; 128 shards of 100 cost 18.6 µs, of which 15.0 µs is the name scan. Both sit far inside a 10 ms tick — 1.3 µs is thirteen parts in a hundred thousand of one — andChannel IDbeing au8caps any document at 256 shards regardless, so neither is bought. The note says which to take first if either ever is.Testing Verification
cargo test --workspace— 1434 passing, 0 failed, includingshards_end_to_end.rs,composition.rs,listings_name_their_shard.rs, the era store's shard suite, the driver's mid-session write cases andprecreated_at_startup.rs. Every gate was re-run on the assembled tree rather than per branch.cargo fmt --all --check,cargo clippy --all-targets -- -D warningsandscripts/check-public-repo-rules.shre-run clean on the head that carries them.cargo clippy --all-targets -- -D warningsclean on stable 1.96.0 — it caught aniter().any(|id| *id == x)on a slice in a test committed earlier on this branch.cargo fmt --checkclean.scripts/check-public-repo-rules.shexits 0.Channel IDs counted and every shard asserted to carry a block for every specification.examples/replay.shrun by hand on the assembled tree, twice over, with the output above recorded in the plan's It ran section. Its floor is asserted now: quotes and trades per channel, and every instrument that moves is one that channel described — which is what a permuted composition fails. It is deliberately not a CI gate: it needs a multicast group and five processes, and a test that can only run by hand must not be able to fail the build.