Skip to content

publisher: several channel instances of one feed specification - #98

Merged
juan-malbeclabs merged 26 commits into
mainfrom
jo/publisher-feed-routes
Sep 10, 2026
Merged

publisher: several channel instances of one feed specification#98
juan-malbeclabs merged 26 commits into
mainfrom
jo/publisher-feed-routes

Conversation

@juan-malbeclabs

@juan-malbeclabs juan-malbeclabs commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Stacked on #97, which carries the design and the plan; this diff is the code. All twelve tasks.

Sixty-two channel instances from one process — 31 shards of two feed specifications — where the publisher emitted one channel per specification and refused a second block of one. Lifting that refusal alone would produce a publisher that starts and is wrong in four places, so tasks 1 through 8 land behaviour-neutral with the gate still in force, and task 9 lifts it.

Tasks 1–8, behaviour-neutral by construction

With one shard configured every partition is a partition of one, so the existing suite is the gate on this half.

The document. [[feed]] shard, absent resolving to the default. A checked ShardName — one lowercase path component, at most sixty-four bytes — because the name becomes a path in two places and the third use is the one that forgets. Spelling the default explicitly is refused: two spellings of one shard are two era files. feed_specs() is distinct now, because it answers which feeds does this publisher emit and an adapter handed one specification once per shard is being told about the deployment instead.

A refusal that closes a hole that already existed. Two blocks sharing a channel_id loaded fine: channel_ids() sorts and dedups, so one set of metric series was pre-created and two channel instances wrote to it with nothing saying so. One channel per specification made it unlikely; many make it a matter of time.

The boundary. list_on is required and list is defaulted, in that direction on purpose — the reverse lets an adapter that had not been updated admit every instrument to the default shard, every shard collapsed onto one channel with no error, no counter and no log. A compile_fail,E0046 doctest pins it. dz-adapter-core takes the major version that costs.

The registry. One registry, N published sets — N registries would be N Instrument ID spaces under one Source ID and N writers on one state directory. The shard is an index on the registry's own published entry, not on the instrument table, which would drag the lowering and the recorder's offline re-lowering into a change that buys them nothing.

The era store. Keyed on the channel instance, and the default shard keeps <spec>.era. That last clause is the whole risk: a renamed file reads as no file, which resolves to the first era, so a publisher on era 7 would restart on era 1 and announce nothing. ShardName::era_shard is the one place that mapping is made, because the natural thing to write at a call site is the wrong thing.

The send paths. Two typed vectors indexed by shard, built shard-outer, with the index resolved once per event beside the instrument lookup — no string comparison on the datagram path and no dynamic dispatch. EventSink is untouched: the trait, its methods and its parameters.

The rotation. Divided by the shard's own published count, and walking only its own instruments. Fixing the divisor alone leaves the same 1/N error arriving through the cursor, because the slots are one table for the process.

Task 5, which the tasks-1-to-8 commit had missed

TeeConfig::destination was still <path>.<spec>.<role>, so four channel instances fanned out to five sockets and two shards' copies of one role arrived on one — datagrams a recorder cannot attribute without decoding, the one thing a record path does not do. It now carries the shard, absent for the default, for the era file's reason: what a call site naturally writes is shard.as_str(), and that moves the socket every existing deployment's recorder is bound to.

It was found by the plan's own acceptance clause — revert each of tasks 3, 4, 7 and 9 and watch a named test fail. The revert that failed nothing was the task nobody had written.

Task 9, the gate

DuplicateFeedShard { spec, shard } keyed on the pair, ShardSpecsDisagree naming the shard and the specification it has no block for — the check that makes list_on total — and run.rs composing one pipeline per block instead of assigning into an Option.

The end-to-end test is the plan's centre, and every assertion in it is an exclusion: shard A's reference-data port carries A's definitions passes against a publisher that packs everything onto everything, so each test states what a channel did not carry beside what it did. Two instruments per shard, not one — with a single instrument each, a packing publisher still puts exactly one symbol on each port, the failure reads as "carries none of its own", and the exclusion is never reached.

What an operator has to change: dz_publisher_refdata_instruments_current gains a channel_id label

Called out on its own rather than in the cost list, because it is the one thing here that changes the meaning of a series an operator may already be selecting.

Instrument Count is a channel's number and not a process's — it is what that channel's manifest states and what a subscriber bound to that channel reconciles against. So the gauge reporting it is now one series per Channel ID instead of one for the publisher. Left process-wide it would write the sum of every shard's published set to a series read as the count on the channel I am bound to: on 31 shards, thirty-one times what any one subscriber will ever receive a message for.

The break is in the selector, not in the value. A panel or alert selecting dz_publisher_refdata_instruments_current with no aggregation was selecting one series and now selects one per channel; a rule that assumed a scalar gets a vector on the deploy that carries this, with nothing failing to announce it. A single-shard publisher gets one series where it had one and the number is unchanged — but the selector still has to tolerate a label it did not have.

What to write instead:

  • per channeldz_publisher_refdata_instruments_current{channel_id="3"}, which is the question this series was always being asked and could not answer;
  • process-widedz_publisher_instruments_published, which is unlabelled, still exact, and does not need a sum() over a label whose cardinality is a configuration choice.

The gauge is pre-created at 0 for every declared Channel ID, and unlike the two manifest gauges it is not gated on the refdata port role: a channel that exists and holds nothing is the case worth alerting on, and a series that only appears on first use cannot carry an alert for a thing never happening.

Its HELP text is corrected in the same commit. It read "Instruments currently known to the upstream reference-data source", which the call site had never matched — the value has always been the size of the published set, which is smaller wherever the selection policy declines. Nothing about the number changes; the string now describes it.

Nothing else in the reference-data set moves. dz_publisher_refdata_definitions_emitted_total and the listing counters stay process-wide on purpose, because they are rates and a rate over channels aggregates honestly. manifest_seq and manifest_valid already carried channel_id.

Task 12 — it ran, and one thing it could not

examples/replay.sh is four [[feed]] blocks of one specification through run(), over real multicast sockets, read by four instances of this repository's Go subscriber, twice — because an era is only observable across a restart. One block names no shard, so the run covers the upgrade as well as the feature.

  channel  0 (the default shard)  38 messages, era 2, seq from 0, definitions REPLAY-D1,REPLAY-D2
  channel  1 alpha                38 messages, era 2, seq from 0, definitions REPLAY-A1,REPLAY-A2
  channel  2 beta                 38 messages, era 2, seq from 0, definitions REPLAY-B1,REPLAY-B2
  channel  3 gamma                38 messages, era 2, seq from 0, definitions REPLAY-G1,REPLAY-G2

  four eras, one per channel instance: [2, 2, 2, 2]
  each advanced by exactly one across the restart

  fan-out.top-of-book.mktdata       20 datagrams, longest 84 bytes, first magic 0x5a44
  fan-out.top-of-book.alpha.mktdata 20 datagrams, longest 84 bytes, first magic 0x5a44

Each subscriber read its own Channel ID and no other, its own sequence series from 0, a Reset Count equal to its own era file, and exactly its own shard's two definitions. Four era files, the default shard's still top-of-book.era, each advanced by one rather than by four.

What could not run. The plan asked for two shards and both specifications. The built-in record adapter holds no book, answers no snapshot, and AdapterRegistry refuses it beside a market-by-price block rather than publish deltas a mid-session joiner can never resynchronise. That refusal is right and predates this plan; what it means is that a real depth run needs a venue with a book and this repository has none. The bullet in the plan says so now instead of asking for something refused at startup, and the depth path's shard routing stays covered by depth_end_to_end.rs and shards_end_to_end.rs against recording sinks.

Two things only the real run could find, both recorded in the plan: a document that came out naming multicast_group = "1000", because GROUPS is bash's own array of the invoking user's group ids and assignments to it are ignored — refused by name at load, which is the failure this publisher is built to produce rather than start on; and task 5 above.

Vocabulary

GLOSSARY.md 1.3.0 replaces the word this repository uses for the reference copy with fan-out, and the plan's constraint governs test names and identifiers, not only prose. The concept now takes the glossary's word throughout what this change wrote: three test names, the example path a reader copies out of the doc comment, and every identifier and comment in replay.sh.

Deliberately not moved, because each is a change with its own migration: the [adapter.tee] config key, TeeConfig, Tee, ReferenceStream and the StartupError variants named after them. The two test names that describe the key keep it, since naming a key is not using the word. The design document's blockquote of the old doc comment stays as written — it records what the comment said the day the design was written.

Split, at a reviewer's request

This PR carried five changes and now carries one. [egress] ttl becoming required is #100 and poll_upstream is #101, each stacked on this branch with its own spec and plan, because each changes behaviour on its own and neither should have to be approved in order to approve the shards. The three design documents for work that is not built — the venue half of a feed race, a polled transport, a session transport — are on jo/venue-boundary-designs with no PR yet.

What stays here is the shards work and the corrections a review found in what it wrote: the record example writing no file for a refused event, the plan's task count, and a naming-table row that cited a glossary definition which does not exist.

What a review round added after the split

The clause at the adapter boundary is corrected here, not in #100. ListingSink::list_on justified its version bump with "every implementor of it is in this workspace" — in dz-adapter-core's crate documentation and in the trait's own doc comment — and implementors outside this workspace exist, a venue's own test doubles among them. Both copies are in this diff and this PR lands first, so the correction moved here from #100 rather than arriving a merge behind the sentence it corrects. #100's spec, plan, title and body say where it went; #101, which branches from here as a sibling of #100, now inherits the corrected text instead of the stale text.

The thread that raised it was resolved earlier against df4d125, a commit the split of this branch into #98/#100/#101 rewrote away. Nothing on this head pointed at a fix and an audit of "were the comments addressed" would have read as answered.

open_ports's doc comment is back on open_ports. The PortOpener trait was inserted directly above it with no blank line, so the whole port-role argument — why a dead refdata socket or a dead snapshot socket ends the process rather than degrading — became the trait's documentation, and the function, which moved below compose_feeds, was left with none.

The definition tick's per-tick cost is written down where the next reader meets it. rotation.rs already states that a rotation walks past the slots of shards that are not its own; Registry::definition_tick does the same walk over the same shared table and had no such note, so a shard count was sized against one and not the other. It is stated rather than optimised, and the arithmetic is the reason: the cursor persists across ticks and a shard emits its published set once a lap, so a lap is exactly one revolution of the table — N × slots visits a lap, which at 31 shards of 100 instruments on the default 30 s cycle is 40 slot visits a tick against a 10 ms tick, and the walk is entered only on the one tick in twenty-four a shard owes anything.

Measuring it found the larger term, which is not the one the review named. shard_index resolves the shard by scanning the configured shards and comparing strings, and it runs before due is consulted — so it costs N comparisons on every tick rather than on the ticks something is owed. Timed over a whole lap: 31 shards of 100 cost 1.3 µs a tick, of which 1.1 µs is the name scan; 128 shards of 100 cost 18.6 µs, of which 15.0 µs is the name scan. Both sit far inside a 10 ms tick — 1.3 µs is thirteen parts in a hundred thousand of one — and Channel ID being a u8 caps any document at 256 shards regardless, so neither is bought. The note says which to take first if either ever is.

Testing Verification

  • cargo test --workspace1434 passing, 0 failed, including shards_end_to_end.rs, composition.rs, listings_name_their_shard.rs, the era store's shard suite, the driver's mid-session write cases and precreated_at_startup.rs. Every gate was re-run on the assembled tree rather than per branch.
  • The review round above changed no behaviour, and the count says so: 1434 before it and 1434 after. All three changes are prose — two doc comments and a cost note — so no test is claimed to kill a mutant for them, rather than a test written to assert a sentence against itself. cargo fmt --all --check, cargo clippy --all-targets -- -D warnings and scripts/check-public-repo-rules.sh re-run clean on the head that carries them.
  • cargo clippy --all-targets -- -D warnings clean on stable 1.96.0 — it caught an iter().any(|id| *id == x) on a slice in a test committed earlier on this branch. cargo fmt --check clean. scripts/check-public-repo-rules.sh exits 0.
  • The plan's last acceptance clause was run, not assumed: each of tasks 3, 4, 7 and 9 was reverted, and the plan now names the tests that failed under each — including task 5 in both directions, since always appending the shard fails the default's literal destinations and never appending it fails the distinctness of the fifteen sockets three shards of both feeds open. That exercise is what surfaced task 5 having never been done.
  • Acceptance criterion 1 had no test and now has one: 31 shards of both specifications resolve as 62 channel instances, with 62 distinct Channel IDs counted and every shard asserted to carry a block for every specification.
  • examples/replay.sh run by hand on the assembled tree, twice over, with the output above recorded in the plan's It ran section. Its floor is asserted now: quotes and trades per channel, and every instrument that moves is one that channel described — which is what a permuted composition fails. It is deliberately not a CI gate: it needs a multicast group and five processes, and a test that can only run by hand must not be able to fail the build.

…rocess

A venue whose instrument set is sharded across many channels needs one process
to operate many channel instances of one specification. Today a process emits at
most one feed per specification, and the check that says so is holding shut
three more fences that are silent.

The key is `shard`, not `route`. `route` is taken twice over — `[egress]` means
the IP route to the group by it, and GLOSSARY.md defines `path` as "one of
several redundant routes carrying the same data", which makes N routes read as N
redundant paths carrying the same instruments when they carry disjoint ones.
`category` is a feed-registry arbitration boundary the glossary says outright is
not a taxonomy. `shard` is the glossary's own word: a channel is "a logical
shard of the instrument set".

The reference-data question is not a design choice. reference-data/spec.md heads
all seven publisher obligations with "A publisher operating a channel ... MUST",
and makes Manifest Seq, Valid, Instrument Count and the definition cycle per
channel. One process-wide published set on N refdata ports produces four
specific untruths — and obligation 6, restarting the cycle on a Manifest Seq
change, turns any admission anywhere into an N-channel definition burst, which
is what obligation 2 forbids.

The era store is a latent state-corruption bug rather than an inefficiency, and
not the one it first looks like: begin_era advances and persists on every call
and run.rs calls it per block, so N blocks take N different eras. The era a
channel advertises is therefore decided by its position in the document, each
start advances the shared counter by N, and moving a block hands a channel an
era it has already published under.

Three fences beyond the four the request named: the reference copy keys its
socket on (spec, port role), the snapshot rotation divides the cycle by the
process's published count, and nothing checks Channel ID uniqueness across
blocks — which two blocks sharing an ID would turn into two feeds silently
sharing four metric series.
Eight tasks of the feed-routes plan, all of them behaviour-neutral: the
duplicate-specification check stays in force, so with one shard configured every
partition is a partition of one and the suite passes unchanged. Lifting that
check is task 9.

**The document.** `[[feed]] shard`, absent resolving to the default. A checked
`ShardName` — one lowercase path component, at most sixty-four bytes — because
the name becomes a path in two places and the third use is the one that forgets
to check. Spelling the default explicitly is refused: two spellings of one shard
are two era files. `feed_specs()` is distinct now, because it answers *which
feeds does this publisher emit* and an adapter handed one specification once per
shard is being told about the deployment instead.

**A refusal that closes a hole that already existed.** Two blocks sharing a
`Channel ID` load today: `channel_ids()` sorts and dedups, so one set of series
is pre-created and two channel instances write to it, with nothing saying so.
One channel per specification made it unlikely; many make it a matter of time.

**The boundary.** `list_on` is required and `list` is defaulted, in that
direction on purpose — the reverse lets an implementor who did not update admit
every instrument to the default shard, every shard collapsed onto one channel
with no error, no counter and no log. A `compile_fail,E0046` doctest pins it.

**The registry.** One registry, N published sets. One per shard would be N
`Instrument ID` spaces and N writers on one state directory. The shard is an
index on the registry's own published entry, not on the instrument table, which
would drag the lowering — and the recorder's offline re-lowering with it — into
a change that buys them nothing.

**The era store.** Keyed on the channel instance, and the default shard keeps
`<spec>.era`. That last clause is the whole risk: a renamed file reads as *no
file*, which resolves to the first era, so a publisher on era 7 would restart on
era 1 and announce nothing — the corruption the corrupt-file refusal exists to
prevent, delivered by the upgrade meant to be safe. `ShardName::era_shard` is
the one place that mapping is made, because the natural thing to write at a call
site is the wrong thing.

**The send paths.** Two vectors indexed by shard, built shard-outer. Per block
in document order, a file interleaving the blocks would index the two vectors
differently and route one instrument's quotes and its levels to two channels,
and nothing below `Feeds` re-checks it.

**The rotation.** Divided by the shard's own published count, and walking only
its own instruments. Fixing the divisor alone leaves the same 1/N error arriving
through the cursor, because the slots are one table for the process.
…sions

The end-to-end test the plan calls its centre. Two shards, both
specifications: the arrangement the change exists for, and the one the
duplicate-specification gate refused until it lifted.

Every assertion is an exclusion. "Shard A's reference-data port carries
A's definitions" passes against a publisher that packs everything onto
everything, so each test states what a channel did *not* carry beside
what it did, and the two shards are given different instruments so that
"carried the other's" is a statement that can be false.

Three things the first draft got wrong, all caught by the tests rather
than by reading:

- `poll_listings` is due-gated on `LISTING_POLL` and returns whether it
  ran. A second call on the same instant is a no-op, so the manifest
  test was reading a manifest that never saw the admission. The gate is
  asserted now instead of discarded.
- A definition is type 0x02, not 0x10. The floor assertion ("a port
  carried no definition at all") is what said so.
- Symbols are read by decoding with the codec's own decoder rather than
  hunting substrings in the datagrams.

The era is deliberately not asserted here: the harness hands each
pipeline a literal `ResetCount` rather than resolving one, so the
assertion would be about the fixture. That property lives in the era
store's suite, where it can be observed.
The centre test asserted "carries mine" and "does not carry theirs" as two
membership checks over one instrument per shard. Against a publisher that
packs the whole published set onto every reference-data port, that pair does
fail — but it fails on the first assertion, "carries none of its own", because
the pacer owes one definition per lap and a packing publisher still puts
exactly one symbol on each port. The exclusion, the assertion the plan calls
its centre, was never reached.

Two instruments per shard is what makes a packing publisher show up as four
symbols where two belong, and the assertion is now an equality: A's port
carries A's two and nothing else. Inclusion and exclusion in one statement,
with no ordering or repetition to read around — `symbols_on` dedups and sorts,
because a full lap repeats and the question is which instruments a channel
described, not how often.

The port count is asserted too. Four channel instances is the arrangement
under test, and a harness that quietly grew or lost a port would narrow every
assertion above it without failing one.
…e default

Task 5, which the tasks 1-to-8 commit missed: `TeeConfig::destination` was
still `<path>.<spec>.<role>`, so the four channel instances the change exists
for fanned out to five sockets. Two shards' copies of one feed's one role
arrived on one socket, and a Unix datagram carries neither the destination port
nor the group the diff is keyed on — a recorder cannot attribute them without
decoding, which is the one thing a record path does not do. The existing
argument for the feed being in the name is the same argument one noun further
along, so it is extended rather than rewritten.

The default shard is spelled by its absence, as its era file is, and for the
same reason: what a call site naturally writes is `shard.as_str()`, and that
renames the socket every existing deployment's recorder is bound to. The
fan-out would then write to a path with nobody on it — every datagram dropped
and counted, or an operator who believes copies are still being archived.

The tests are path arithmetic and open no socket: the default shard's five
destinations as literals, a named shard's distinct from them, and the fifteen
sockets three shards of both feeds open asserted distinct rather than sampled.
Reverting either half of the naming fails one — the always-append mutant on the
default's literals, the never-append mutant on the distinctness.
The built-in record adapter called `list`, so every instrument it offered went
to the default shard however many the document configured. That is the whole
of what stood between `run()` and a two-shard offline run: a record names a
symbol and nothing else, so the partition has to come from configuration, and
`[[adapter.upstream.listing]]` is where the source process and the publisher
already agree out of band.

`shard` is an `Option` rather than a `String` defaulted to the token. A source
with no partition to state says nothing, `poll_listings` then calls `list`, and
the default keeps one spelling in the crate that owns it. Unchecked on this
side on purpose: a name that is not a configured shard is declined at
admission and counted, and checking that a *block's* name is a safe path
component belongs to the block.

The tests state the whole sequence rather than one offer — an adapter passing
the token for everything satisfies any assertion about the defaulted row, and
one reaching for the first listing's shard satisfies any assertion about the
first. Three rows with three answers is what neither survives. The second test
is the boundary's own documented case: a `None` back costs that listing and
the poll continues, because the listings after it are the ones on shards that
do exist.
…bscribers

Task 12. `examples/replay.sh` was one feed of one shard; it is now four
`[[feed]]` blocks of **one** specification — the arrangement the
duplicate-specification gate refused until it lifted — through `run()`, over
real multicast sockets, read by four instances of this repository's Go
subscriber. One block names no shard and is therefore the default one, so the
run tests the upgrade as well as the feature.

Every assertion is the subscriber's own output. Per channel instance: its own
`Channel ID`, its own sequence series from 0, its own `Reset Count` equal to
its own era file, and — stated as an equality rather than as "carries mine" —
only its own shard's two instrument definitions. Two symbols per shard for the
end-to-end test's reason: with one each, a publisher that packed everything
onto everything still puts exactly one symbol on each port and the exclusion is
never reached.

It runs twice, because an era is only observable across a restart. Four era
files under the state directory, the default shard's still `top-of-book.era`,
and each advanced by exactly one — a shared counter advances every channel by
the number of blocks, which would read here as four.

`--symbol` on `write_records` is repeatable, and every symbol carries the same
three events, so a difference between two subscribers' outputs is the
publisher's partitioning and not the recording's.

Two things the run itself found. `GROUPS` is bash's own array of the invoking
user's group ids and assignments to it are ignored, so the document came out
naming a multicast group of `1000`; the load refused it by name, which is the
failure this publisher is written to produce rather than start on. And the
clippy stable runs is stricter than the one this test was written under —
`iter().any(|id| *id == x)` on a slice is `contains`.
…t do

Forty-eight boxes, and an "It ran" section holding the output rather than a
claim about it: four channel instances of one specification through `run()`,
over real multicast sockets, read by four instances of this repository's Go
subscriber, twice — because an era is only observable across a restart.

The honest part is what did not run. Task 12 asked for two shards and both
specifications; the built-in record adapter holds no book, answers no snapshot,
and `AdapterRegistry` refuses it beside a `market-by-price` block rather than
publish deltas a mid-session joiner can never resynchronise. That refusal is
right and predates this plan, and what it means is that a real depth run needs
a venue with a book, which this repository does not have. The bullet says so
now instead of asking for something that is refused at startup.

Two things only the real run could find, both recorded: a document that came
out naming a multicast group of `1000`, because `GROUPS` is bash's own array
and assignments to it are ignored; and the reference-copy fan-out still keyed
on the feed and the port role alone, which is task 5 — missed by the tasks
1-to-8 commit and found by four channel instances fanning out to five sockets.
… assumed

Acceptance criterion 1 had no test. Everything else about shards is asserted on
two of them — enough to make a partition falsifiable, not enough to say the
document scales — and the set checks are the ones that would disagree with
themselves at size: one shard missing a specification, two blocks on one
`(spec, shard)` pair, a repeated `Channel ID`.

31 shards of both specifications, every port distinct across the document,
62 distinct `Channel ID`s counted rather than assumed, and every shard carrying
a block for every specification asserted across the whole set. One shard is the
default, named by the absence of the key, because a deployment that grows into
shards grows out of a document that had none.
The last acceptance clause asks that reverting any one of tasks 3, 4, 7 or 9
fail a named test, and a plan that only asks for that has not checked it. Each
revert was run and the plan now names the tests that failed under it, including
the two directions task 5's naming can be wrong in — always appending the shard
fails the default's literal destinations, never appending it fails the
distinctness of the fifteen sockets three shards open.

The clause earned its keep: the revert that failed nothing was task 5, because
the reference-copy socket had never been keyed on the shard in the first place.
… owns

`GLOSSARY.md` 1.3.0 replaces the word this repository still uses for the
reference copy with `fan-out`, and the plan's own vocabulary constraint says so
in as many words — it governs identifiers, comments, test names, config keys
and commit messages alike. What this change wrote had followed the file's
existing prose, which was already right, and not its test names.

So the concept takes the glossary's word: three test names, the local bindings
beside them, the example `path` an operator copies out of the doc comment, and
every identifier and comment in `replay.sh` — where the run's own output now
prints `fan-out.top-of-book.alpha.mktdata`. The recorded output in the plan is
the new run's, not the old one edited.

What deliberately does not move: the `[adapter.tee]` key, `TeeConfig`, `Tee`,
`ReferenceStream` and the `StartupError` variants named after them. Those are a
config key operators have written down and public API of published crates, so
renaming them is a change with its own migration and its own PR, not a side
effect of one about shards. The two test names that describe the *key* keep it
too, because naming a key is not using the word.

The design document's blockquote of the old doc comment stays as it was: it
quotes what the comment said on the day the design was written, which is what a
dated document records.
…ligned

`Feeds` held two parallel vectors indexed by shard. The index was the
whole of the pairing: the pipeline at position `i` of one vector and the
pipeline at position `i` of the other had to be the same shard, or an
instrument's quotes and its levels left by two different channels — under
two different sequence series, which a subscriber reads as its own and
cannot detect. What held that together was four `assert!`s in
`push_shard`, and its own doc comment called them the second line of
defence.

There is nothing left to align. A `ShardFeeds` owns one shard's two send
paths and derives its name from one of them at construction, so a
document that interleaves its blocks cannot separate a shard's
specifications — they are built together and held together. The four
assertions go, `shard_count()` stops being `max(len, len)`, and
`shard_name()` stops being first-non-`None`.

`ShardFeeds::new` returns `None` for the pair with neither specification,
which no resolved document produces — the shard set is the distinct
shards of the enabled blocks — and `run.rs` turns it into a new
`StartupError::ShardWithNoFeed` rather than a skip. A skip is the silent
failure here: `Feeds` and the reference-data registry's shard list are
both indexed by the same `Config::shards()`, so dropping an entry from
one shifts every later shard onto another channel instance's numbering.

A struct and not a three-variant enum, which is the other way to make the
empty pair unrepresentable: the variant holding both feeds is one
`FeedPipeline` larger than the ones holding one, so every entry in the
vector would be sized for it, and `clippy::large_enum_variant` is denied
in CI. A document carries the same specifications on every shard anyway,
so that padding would be paid on all of them or none.

Also here, because they were reading the vectors directly: the definition
tick and the final manifest go through the shard, the teardown's two
whole-publisher passes iterate shards while keeping the steps ordered
against each other across the publisher, and the snapshot rotations are
built one per shard rather than one per depth pipeline — same length, and
now the same reason.

`cargo clippy --all-targets -- -D warnings` clean and 1408 tests pass, the
same count as before: no test moved, so nothing about the behaviour did.
Task 13. Reversing the shard order in `run.rs`'s composition — the edit that
publishes each shard's instruments under another channel instance's sequence
series, which no subscriber can detect — passed all 1408 tests. Nothing in the
suite reached `compose_and_run`, and the by-hand offline run did not catch it
either.

Two things hid it, and both are now written where the next reader will find
them. The end-to-end harness composes its **own** `Feeds`, shard-outer and
block-inner, so those tests assert the harness's ordering rather than the
runtime's. And the definition path is keyed on a shard's *name* — `ShardFeeds`
derives it from one of its own send paths — while the event path is keyed on its
*index*: so a permutation leaves every reference-data port carrying exactly its
own shard's definitions, and sends a quote into another shard's pipeline, whose
lowering does not hold the instrument, where it is dropped before any wire.

`RouteLookup` was not the missing seam. It exists because "a test that needs a
route to a multicast group is a test that does not run in CI", and that is true,
but `MulticastTransmitter::open` binds and connects a socket — so a composition
holding a route still needs a network. What goes behind a trait is the thing
that produces a feed's `Ports`, and `KernelPorts` is the real one.

`compose_feeds` takes the shard list and the feed list **separately** rather
than a `Config`, which is what makes `StartupError::ShardWithNoFeed` reachable:
no document and no resolved `Config` can make the two disagree, so the variant
guarded a refactor and nothing exercised it. Now one test hands it a shard the
feed list does not mention.

Four tests over the harness's own recording ports — the order asserted as a
whole sequence, because the failure is a permutation and every shard is present
in it; a shard's blocks opened together before the next shard's, read off the
opener's own record; one era file per channel instance, against a real store in
a temporary directory, because a fake would assert the fake; and the refusal.

The empty pair is tested directly too, where a document cannot reach it: every
method on `ShardFeeds` is written against there being a send path to read a name
off, and `new` returning `None` is what makes `name()` total.

And `replay.sh` gains the floor it was missing. It asserted definitions and
identity and never asserted that a **quote** arrived — which is exactly what a
permuted composition produces: definitions intact on every channel, no market
data anywhere, 38 messages become 32. It now asserts the kinds per channel, and
that every instrument moving is one that channel described.
@juan-malbeclabs juan-malbeclabs changed the title publisher: several channel instances of one feed specification, from one process publisher: several channel instances of one feed specification, and four changes beside it Sep 9, 2026
@juan-malbeclabs
juan-malbeclabs requested a lite review from Copilot September 9, 2026 15:15
@bgm-malbeclabs

Copy link
Copy Markdown

Verified on the assembled tree: workspace tests green with the recorder crates excluded, since they do not build on macOS; cargo clippy --all-targets -- -D warnings clean on the publisher, adapter and ingress crates; cargo fmt --check clean.

The shard routing itself is right. One index resolved per event and both of that shard's pipelines taken from it, list_on required with list defaulted in the direction that fails loudly, the compile_fail,E0046 doctest pinning that, era_shard keeping the default shard's file name, TeeConfig::destination omitting the default shard for the same reason. Task 5 being found by the revert exercise rather than by reading is the part of this worth keeping.

  1. Scope. 7029 lines over 59 files, five changes. Shards, [egress] ttl becoming required, poll_upstream, one comment correction, and three doc only spec pairs. The ttl change and poll_upstream each stand alone and each change behaviour alone. They should not have to be approved in order to approve the shards.
  2. [egress] ttl is now required, so every existing document needs an edit. With recorder: inline mode, so a host can have the rows without keeping the bytes #95 also requiring --archive, two mandatory configuration edits arrive in one window from two PRs. Worth staging them apart.
  3. Shard::resolve in dz-publisher-egress/src/era.rs has no caller outside its own test. ShardName::era_shard in config.rs:853 reimplements it. Shard::resolve's doc says "This is what a caller holding a resolved shard name wants" and no caller wants it. Keep one. The point of the method is that the mapping lives in exactly one place, and right now it lives in two.
  4. .expect("checked above") at publisher.rs:1679, 1744, 1843 and 1876. top_of_book_carries and market_by_price_carries use shard.is_none_or(...), so both answer true when shard_of returned None, and then shard.and_then(...).expect(...) panics. It survives today only because lower_quote and lower_instrument_reset open with instruments.get(instrument)?, and Registry::withdraw clears the lowering table and the slot together. That invariant lives in another crate and nothing at these call sites states it. Count the message unroutable instead, or have the guard hand back the index it checked.
  5. StartupError::DuplicateChannelId reports first and second as specifications. Two blocks of one specification on different shards sharing a channel_id produce a message naming the same specification twice, and shards are what make that the likely case. Name the shard.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are at least two concrete issues in changed regions (a misleading crate-level boundary statement and an example that can emit invalid zero-length record files on refusal) that should be corrected before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR implements multi-shard publishing (multiple channel instances per feed specification) and the supporting invariants around reference-data partitioning, era persistence, composition ordering, metrics, and adapter boundary changes, plus an additional feature enabling adapters to write upstream on an already-established connection.

Changes:

  • Introduces [[feed]] shard / ShardName and composes feeds shard-outer to keep routing, refdata partitioning, and sequencing consistent across many channel instances.
  • Keys eras and refdata state correctly per channel instance (spec × shard), updates snapshot rotation pacing/membership to be shard-scoped, and adjusts metrics to track per-channel instrument counts.
  • Adds Adapter::poll_upstream and a driver cadence (UPSTREAM_POLL) to support mid-session upstream writes; makes [egress] ttl required at config resolution with a dedicated startup refusal.
File summaries
File Description
rust/recorder/dz-recorder-relower/src/relower.rs Updates ListingSink implementation to the new list_on boundary.
rust/publisher/README.md Documents shard vs channel-instance semantics and new startup refusals.
rust/publisher/dz-publisher-runtime/tests/shutdown.rs Adjusts runtime tests for shard-aware refdata validity checks.
rust/publisher/dz-publisher-runtime/tests/shards_end_to_end.rs Adds end-to-end tests asserting shard isolation across refdata and routing.
rust/publisher/dz-publisher-runtime/tests/harness/mod.rs Extends test harness to build shard-outer feed composition and recorders.
rust/publisher/dz-publisher-runtime/tests/depth_end_to_end.rs Adds shard-focused assertions for depth routing, resets, manifests, rotation.
rust/publisher/dz-publisher-runtime/tests/composition.rs Adds composition-level tests via PortOpener seam and compose_feeds.
rust/publisher/dz-publisher-runtime/src/run.rs Refactors composition into compose_feeds, adds PortOpener, shard-era handling, tee destination shard keying.
rust/publisher/dz-publisher-runtime/src/rotation.rs Makes snapshot rotation shard-scoped (pacing divisor + membership predicate).
rust/publisher/dz-publisher-runtime/src/pipeline.rs Carries ShardName on pipelines to keep name/index coupled.
rust/publisher/dz-publisher-runtime/src/lib.rs Re-exports shard/composition types and functions.
rust/publisher/dz-publisher-runtime/src/error.rs Adds shard-related startup errors and TTL-required startup refusal.
rust/publisher/dz-publisher-runtime/src/builtin.rs Adds optional shard to built-in listing records.
rust/publisher/dz-publisher-runtime/examples/loopback_publisher.rs Updates example to use shard-aware era and registry config.
rust/publisher/dz-publisher-refdata/tests/shards.rs Adds refdata shard behavior tests (set isolation, ID space, refusals).
rust/publisher/dz-publisher-refdata/tests/listings.rs Updates tests for shard-configured registry initialization.
rust/publisher/dz-publisher-refdata/tests/identity.rs Adds default-shard equivalence test (list vs list_on(DEFAULT_SHARD)).
rust/publisher/dz-publisher-refdata/tests/definitions.rs Updates tests for shard-aware manifest sequencing.
rust/publisher/dz-publisher-refdata/src/refusal.rs Adds shard-specific refusals and clarifies boundary semantics.
rust/publisher/dz-publisher-refdata/src/lib.rs Documents shard partitioning at the crate level and exports ShardConfig.
rust/publisher/dz-publisher-refdata/src/error.rs Adds shard-config validation errors (NoShardConfigured, ShardConfiguredTwice).
rust/publisher/dz-publisher-metrics/tests/precreated_at_startup.rs Extends assertions for channel-keyed series and per-channel instrument count gauge.
rust/publisher/dz-publisher-metrics/tests/normative_names.rs Updates metrics touch suite for new set_instruments_current(channel_id, count) API.
rust/publisher/dz-publisher-metrics/src/metrics/refdata.rs Converts instrument count metric to IntGaugeVec keyed by channel_id and pre-creates per declared channel.
rust/publisher/dz-publisher-metrics/README.md Documents sizing/impact of many declared channel instances.
rust/publisher/dz-publisher-egress/src/policy.rs Clarifies DEFAULT_TTL meaning vs config requirement.
rust/publisher/dz-publisher-egress/src/lib.rs Re-exports Shard and updates module docs for shard-keyed eras.
rust/publisher/dz-publisher-egress/src/era.rs Keys era files by (spec, shard) with default-shard filename preservation.
rust/ingress/dz-ingress-core/tests/driver.rs Adds tests for mid-session upstream writes, cadence gating, refusal semantics, pacing.
rust/ingress/dz-ingress-core/src/lib.rs Re-exports UPSTREAM_POLL.
rust/ingress/dz-ingress-core/src/driver.rs Implements UPSTREAM_POLL cadence and calls adapter.poll_upstream mid-session.
rust/adapter/dz-adapter-uds/tests/lossless_through_the_lowering.rs Updates test sink and listing payloads for shard-aware listing path.
rust/adapter/dz-adapter-uds/tests/listings_name_their_shard.rs Adds tests asserting UDS listings forward the shard (default vs named).
rust/adapter/dz-adapter-uds/src/lib.rs Adds shard: Option<String> to UDS listing and routes via list_on/list.
rust/adapter/dz-adapter-uds/examples/write_records.rs Extends record writer example to support multiple --symbol values.
rust/adapter/dz-adapter-core/tests/adapter_is_usable.rs Updates test listing sink to require list_on and asserts default-shard behavior.
rust/adapter/dz-adapter-core/src/sink.rs Introduces DEFAULT_SHARD and makes list_on required with list defaulting to it; expands boundary docs.
rust/adapter/dz-adapter-core/src/lib.rs Adds crate docs about the breaking list_on change and exports DEFAULT_SHARD.
rust/adapter/dz-adapter-core/src/adapter.rs Adds defaulted Adapter::poll_upstream with detailed contract.
docs/superpowers/specs/2026-09-09-upstream-write-after-a-listing-change-design.md Design doc for mid-session upstream writes.
docs/superpowers/specs/2026-09-09-session-transport-design.md Design doc for a future session transport.
docs/superpowers/specs/2026-09-09-recorder-venue-observation-design.md Design doc for venue-side observation/race support.
docs/superpowers/specs/2026-09-09-polled-transport-design.md Design doc for a future polled transport (poll token).
docs/superpowers/specs/2026-09-09-egress-ttl-and-a-boundary-correction-design.md Design doc for required TTL and boundary doc correction.
docs/superpowers/plans/2026-09-09-upstream-write-after-a-listing-change.md Implementation plan for upstream mid-session writes.
docs/superpowers/plans/2026-09-09-session-transport.md Plan for session transport work (future).
docs/superpowers/plans/2026-09-09-recorder-venue-observation.md Plan for venue-side observation (future).
docs/superpowers/plans/2026-09-09-polled-transport.md Plan for polled transport (future).
docs/superpowers/plans/2026-09-09-egress-ttl-and-a-boundary-correction.md Plan for required TTL and boundary doc correction.
docs/README.md Adds index entries for the new specs/plans.
BRINGING-UP-A-FEED.md Updates bring-up guidance for shards, required TTL, and poll_upstream.
Review details
  • Files reviewed: 59/59 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread rust/adapter/dz-adapter-uds/examples/write_records.rs
Comment thread rust/adapter/dz-adapter-core/src/lib.rs
…lan counts thirteen

Three corrections a review found, all of them in what the shards work itself
wrote.

`RecordWriter::write` appends nothing when it refuses, and the example wrote the
buffer anyway — leaving a zero-length `.record` a reader cannot decode. A
refusal is meant to cost that record; it was costing the replay. The numbering
gap that skipping leaves costs nothing: the order is what the names carry and a
reader takes the files that exist.

The plan's Scope said twelve tasks while it carries thirteen, so a reader
tracking progress stopped one short of the task that put the composition under
test. And the row rejecting `category` attributed a definition and a quotation
to an external glossary that contains neither — `Category` is not defined
anywhere in that repository. The argument does not need a citation: the word
describes what a market is about, and what is being named is the opposite of a
taxonomy.

`PortOpener`'s doc states the property rather than the run that found it.
@juan-malbeclabs
juan-malbeclabs force-pushed the jo/publisher-feed-routes branch from df4d125 to b86300e Compare September 9, 2026 15:45
@juan-malbeclabs juan-malbeclabs changed the title publisher: several channel instances of one feed specification, and four changes beside it publisher: several channel instances of one feed specification Sep 9, 2026
… names the shard

Three points a review found, none of them about shards being wrong.

**`Shard::resolve` had no caller and `ShardName::era_shard` reimplemented it.**
The point of that method is that the mapping lives in exactly one place, and it
lived in two. `era_shard` delegates now: `resolve` takes the default token as an
argument because `dz-publisher-egress` does not depend on the crate that owns
the constant, so the decision belongs there and supplying the token is this
method's job. The mapping is what a renamed era file costs — it reads as no
file, resolves to the first era, and a publisher on era 7 restarts on era 1
announcing nothing — and it now has a direct test.

**Four `.expect("checked above")` on the send path are counted refusals.** The
guard answers `true` for an instrument on no shard deliberately, so the lowering
refuses it as an unknown instrument, which is the better diagnostic. What that
left is a send reached with an index the send paths may not hold. It survives in
production only because the lowering opens with `instruments.get(instrument)?`
and the registry clears that table and the slot together — an invariant in
another crate that nothing at the send site stated. The pipeline is resolved
before the send now, and a miss is counted.

**`DuplicateChannelId` names each block's shard.** The specification alone
stopped identifying a block the moment two of them could carry one, and two
blocks of one specification on different shards is the likely shape of the
mistake: the message printed the same word twice and identified neither.
… it claimed

The revert was run and the mutant survived. Restoring `expect("checked above")`
left the new test passing, because the guard answers `false` for this state and
returns before the lowering — so the test exercises the guard, not the site that
panicked. It counted the right number for the wrong reason.

The test now claims the guard, which is what it asserts, and its doc states
what has no test and why. Reaching one of those sites needs the guard to answer
`true` *and* the pipeline to be missing, which happens only for an instrument
whose shard is `None` and whose lowering succeeds — and the registry owns the
lowering's instrument table and clears it with the slot, so the two cannot
disagree through any public path.

So the four refusals replace a panic that was unreachable: a latent hazard
removed rather than a behaviour changed, and the reviewer's point stands on the
invariant being stated in another crate rather than on a reachable crash.
@juan-malbeclabs

Copy link
Copy Markdown
Contributor Author

All five, and the two that are decisions came back as you argued them.

1. Scope — split. This PR now carries the shards work alone. [egress] ttl is #100 and poll_upstream is #101, each stacked on this branch with its own spec and plan, so approving the shards no longer means approving two unrelated behaviour changes. The three design documents for unbuilt work are on jo/venue-boundary-designs with no PR. 7029 lines over 59 files became 1413 tests over the shards and the corrections a review found in what they wrote.

One thing the split turned up: taking sink.rs wholesale onto the ttl branch imported the UpstreamSink doc, which references poll_upstream — a method that does not exist there. Only the clause reword travelled.

2. Two mandatory configuration edits in one window. Agreed, and #100 is now separable: it can wait behind the shards without holding them up. The --archive half is #95's, and the other reviewer's alternative there has been accepted — the mode will be inferred from the configuration rather than switched by a new flag, so that edit stops being mandatory at all.

3. Shard::resolve had no caller and era_shard reimplemented it. Fixed in a6aaad5 by delegating: era_shard now calls Shard::resolve(&self.0, DEFAULT_SHARD). resolve takes the token as an argument because dz-publisher-egress does not depend on the crate that owns the constant, so the decision belongs in the crate that owns Shard and supplying the token is the runtime's job. The mapping had no test at the runtime level either — the_default_shard_contributes_no_component_and_a_named_one_does covers it now.

4. The four .expect("checked above") — replaced, and the revert corrected my claim about them. The pipeline is resolved before the send and a miss is counted, in a6aaad5. Your reading of why they survive is exactly right, and it goes further than I first wrote: I added a test claiming to cover it, ran the revert, and the mutant survived — the guard answers false for the state I constructed and returns before the lowering, so the test exercised the guard and not the site that panicked. It counted the right number for the wrong reason.

Reaching one of those sites needs the guard to answer true and the pipeline to be missing, which happens only for an instrument whose shard is None and whose lowering succeeds — and the registry owns the lowering's table and clears it with the slot, so the two cannot disagree through any public path. So this is a latent hazard removed rather than a reachable crash fixed, and c259140 says so in the test's own doc rather than leaving a test that looks like a gate. The point stands on the invariant living in another crate, which is how you put it.

5. DuplicateChannelId names the shard. Fixed in a6aaad5: the message names each block as a specification and a shard. two_shards_of_one_specification_sharing_a_channel_id_are_refused_naming_both_shards is the case you describe — two blocks of one specification on different shards — and it asserts both shard names, because without them the message printed the same word twice and identified neither block.

Gates on this head: cargo fmt --all --check, cargo clippy --all-targets -- -D warnings, 1416 passing / 0 failing, scripts/check-public-repo-rules.sh exits 0.

Base automatically changed from jo/publisher-feed-routes-docs to main September 9, 2026 16:25
juan-malbeclabs added a commit that referenced this pull request Sep 9, 2026
…rocess (#97)

Documents only — a design and its plan, no code. The publisher emits one
channel per feed specification and refuses a second block of one; this
is the document that answers what a venue does when one channel cannot
carry its instrument set.

## The question it starts from

A `[[feed]]` block is a channel, and `DuplicateFeedSpec` refuses two
blocks naming one specification. So a venue whose top-of-book set
outgrows one channel has no document to write: the publisher it would
need is 62 channel instances from one process — 31 shards of two
specifications — and the gate that stops it is one check.

Lifting that check alone produces a publisher that starts and is wrong
in four places at once, which is why this is a design and not a patch.

## What the spec settles

- **The key is `shard`**, and the Naming section argues down three
better-sounding words. `route` is taken twice over — `[egress]` already
means the IP route by it, and `GLOSSARY.md` defines `path` as "one of
several redundant routes carrying the same data", which makes "62
routes" say the opposite of what is meant, since these carry
**disjoint** sets. `category` is the feed registry's own field and
explicitly "an arbitration boundary, not a taxonomy". `lane` is banned
outright. `shard` is the glossary's own word inside its definition of a
channel: "a logical shard of the instrument set, named by `Channel ID`".
- **A shard is the most a venue may say.** `ListingSink::list_on(shard,
spec)` is required and `list` is defaulted, in that direction on
purpose: defaulting `list_on` would leave an adapter that had not been
updated admitting its whole universe to one channel with no error, no
counter and no log. Which `Channel ID`, group, port, sequence series and
era a shard resolves to stays the operator's, and is unreachable from
the adapter.
- **Two units that are not the same thing.** The shard is the unit of
reference data — one published set, one `Instrument Count`, one
`Manifest Seq`, one definition cycle. The channel instance is the unit
of sequencing — its own sequence series, `Reset Count` and era.
Everything in the plan turns on those being different.
- **The four fences**, each a place where the obvious implementation is
wrong: the gate is a gate and not a check; the two typed `Option`s stay
typed because the datagram path must not dispatch dynamically; the era
store is a latent state-corruption bug rather than an inefficiency, and
not the one it looks like — `begin_era` persists per call and `run.rs`
calls it per block, so N blocks take N eras, the era a channel
advertises depends on its position in the document, and moving a block
hands a channel an era it has already published under; and the single
drain, without which any admission anywhere becomes an N-channel
definition burst.
- **Three fences beyond the four the request named**: the reference copy
keys its socket on `(spec, port role)` and so cannot attribute two
shards' copies; the snapshot rotation divides the cycle by the process's
published count; and nothing checks `Channel ID` uniqueness across
blocks — which two blocks sharing an ID would turn into two channel
instances silently writing one set of metric series.
- **What it costs, before somebody discovers it**: the pre-created
channel-keyed exposition goes from 12 series to 372, exactly 31-fold and
entirely at startup, plus one family that gains a `channel_id` label
because it mirrors a per-channel wire field; a longer document; and a
`u8` `Channel ID`, so 256 is the ceiling.

## What it refuses

- **Reference data per shard is not optional.** The specification
defines `Instrument Count` and the definition cycle per channel, so a
process-wide published set is a wire-level lie however convenient the
code would be.
- **One registry, N published sets** — not N registries. N would be N
`Instrument ID` spaces under one `Source ID` and N writers on one state
directory.
- **The shard is an index on the registry's published entry**, not on
the instrument table, which would drag the lowering and the recorder's
offline re-lowering into a change that buys them nothing.
- **`EventSink` is untouched** — the trait, its methods and its
parameters. A task that changes one has moved routing into the adapter.

## The plan

Twelve tasks with an ordering constraint that is the whole shape of it:
tasks 1 through 8 are behaviour-neutral because the
duplicate-specification gate stays in force, so with one shard
configured every partition is a partition of one and the suite passes
unchanged. Task 9 lifts the gate, and it is the change that makes 3, 4
and 7 load-bearing rather than latent.

Two global constraints worth reading before the code: every test must be
shown to kill its mutant — revert, watch it fail, restore — because
several of these tests have an absence as their subject, and
`GLOSSARY.md` governs every identifier, comment, test name, config key
and commit message.

The default shard is the plan's quietest risk and it says so: it keeps
`<spec>.era`, because a renamed era file reads as *no file*, which
resolves to the first era — a publisher on era 7 restarts on era 1 and
announces nothing, which is the corruption the corrupt-file refusal
exists to prevent, delivered by the upgrade meant to be safe.

## The plan carries its own execution now

This branch is no longer only the argument. The plan's task boxes are
marked, and it carries two sections written after the code ran rather
than before:

- **It ran** — four channel instances of one specification through
`run()`, over real multicast sockets, read by four instances of this
repository's Go subscriber, twice, with the output recorded as the run
printed it. It also states what could **not** run and why: the built-in
record adapter holds no book, answers no snapshot, and `AdapterRegistry`
refuses it beside a `market-by-price` block, so the two-specification
arrangement the plan asked for is refused at startup and a real depth
run needs a venue with a book.
- **The four reverts the plan demands** — a table naming which test died
under each. It earned its keep: the revert that failed nothing was task
5, because the reference-copy fan-out had never been keyed on the shard
at all.

Reading the marked plan against #98 is the intended order: this says
what was decided and what happened, that one says how.

## Testing Verification

No code, so the checks are the ones documents can fail:
`scripts/check-public-repo-rules.sh` exits 0; every referenced path,
type, method and error variant was opened and confirmed against source
rather than assumed — `FeedSection`, `DuplicateFeedSpec`, `ListingSink`,
`Registry`, `RegistryConfig`, `EraStore::begin_era` and its two call
sites, `TeeConfig::destination`, `Feeds`, `SnapshotRotation`,
`channel_ids`, `NORMATIVE_NAMES` — and the vocabulary was read against
`GLOSSARY.md` 1.3.0, whose Naming table is quoted rather than
paraphrased.

The implementation is #98, stacked on this branch so that this diff is
the argument and that one is its execution.

@nikw9944 nikw9944 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two blocking items. A venue naming an unconfigured shard is dropped with no metric and no log — take_unknown_shards has no caller — and per-shard ManifestSummary breaks the recorder's completeness check on multi-shard captures. The era-mapping duplication and the DuplicateChannelId message are verified fixed in a6aaad5; the [egress] ttl and poll_upstream findings are dropped because that code left this branch with the split.

  • Optionaldz-publisher-runtime/src/lib.rs:64-68 still says Manifest Seq describes the published set rather than a channel, so one composed manifest is truthful on every refdata port. This change makes it per shard.

Docs & comments (non-blocking)

  • rust/adapter/dz-adapter-core/src/lib.rs:93 — The crate docs still say "every implementor of it is in this workspace", and sink.rs:204 repeats it. This is the page an external implementor reads to decide whether the trait break applies to them.

Comment thread rust/publisher/dz-publisher-refdata/src/registry.rs
Comment thread rust/publisher/dz-publisher-refdata/src/registry.rs
Comment thread rust/publisher/dz-publisher-runtime/src/rotation.rs Outdated
The plan is one document and both branches carry it, so the correction to task
2's bullet — the break is signalled in the tag message because there is no
changelog and no per-crate version — reads the same from either side.
…them

Two claims in these documents are not true of the tree they describe, and one
task is a tier this change reaches and the plan never mentioned. Corrected
first, so the code that follows is measured against documents that say what is
actually there.

**Task 2 promised a log line and delivered half of it.** The registry remembers
each unknown shard name once and hands it to a caller; nothing calls it. The
checkbox is split in two, the second half is unchecked, and task 14 is the
caller. The design's justification for leaving the two counters mapped to no
metric family is corrected with it: the gauge it points at covers a venue that
misnames *every* offer for a shard, and a venue that misnames some of them
leaves the gauge non-zero with those instruments unpublished. For that case the
line is the only signal there is, which is what makes its absence a defect
rather than a missing nicety.

**The snapshot ceiling stopped being detectable when the divisor became per
shard.** `rotation.rs` states it as a cycle divided by one shard's count falling
below the runtime's tick. The process still serves at most one snapshot per
10 ms tick, so N rotations draw on one budget: 31 shards of 100 instruments on
a five-second cycle each derive a 50 ms tick, five times the process tick and
comfortable by that statement, while the process is asked for 620 snapshots a
second and can send 100. The spec gains the sum the condition actually is, and
task 16 says why it is counted rather than refused — at load there is no
published count to divide.

**A per-shard `ManifestSummary` reaches the recorder.** `Instrument Count` and
`Manifest Seq` are the channel's, which is the specification's definition and
the point of task 3; `dz-recorder-relower` kept one manifest per archive and
compared it against the union of every channel's definitions. Nothing on the
wire moved, so no subscriber is affected, and the publisher side is not the
thing to fix — a process-wide count would be false on every channel it went out
on. Task 15 moves the check to the channel and argues why the channel rather
than the channel instance.
`take_unknown_shards` collected the distinct names and nothing called it, so a
venue naming a shard this document has no channel for had those instruments
declined with nothing an operator could see. The mechanism was task 2's; the
caller was not written.

`Publisher::take_unknown_shards` drains the registry and the tick loop writes
one line per distinct name, beside the lines it already writes for a dropped
fan-out member and a refused snapshot. The line carries the name offered **and**
the names configured, because a misspelling is only visible as the pair, and it
says the instruments were declined rather than leaving that to be inferred. The
exit report gains the two counts.

**The two counters stay mapped to no metric family.** The normative set is
closed by the playbook, this repository does not own it, and a series added here
would be one nobody declared. What is corrected is the justification. `Counts`
said the signal already existed as `refdata_instruments_current{channel_id}`
sitting at 0 for that shard's channel — true only where the venue misnames
*every* offer for a shard. One that misnames some of them leaves the gauge
non-zero with those instruments unpublished, and no series in the closed set
tells that from a channel holding fewer instruments. For that case the line is
the only signal there is, which is what makes writing it obligatory rather than
decorative.

Asserted:
`a_shard_the_document_has_no_channel_for_is_named_once_however_often_it_is_offered`
covers the drain — two instruments under one misspelling produce one name, and a
second poll re-offering both produces none —
`the_unknown_shard_line_names_the_offer_and_the_configured_shards` and
`a_publisher_with_no_named_shard_still_names_what_it_has` cover the line.

**The `eprintln!` itself has no test that can fail**, and this is said rather
than covered: no test in the workspace runs `tick_loop` and none captures
stderr. The drain was put on `Publisher` for that reason — it is the furthest
along that path a test reaches.
…ll is counted

The rotation divisor became per shard, which is right and is what makes each
channel's `[[feed]] snapshot_cycle` mean what it says. The serving rate did not:
the tick body takes at most one periodic snapshot, so N shards draw on one
budget of one per 10 ms tick while the demand adds up.

`rotation.rs` stated the ceiling as a cycle divided by *one shard's* count
falling below the runtime's tick. That was the whole of it with one rotation.
With N it misses the case shards introduce — 31 shards of 100 instruments on a
five-second cycle each derive a 50 ms tick, five times the process tick and
comfortable by that reading, while the process is asked for 620 snapshots a
second and can send 100. Every channel laps in 31 seconds and `snapshot_cycle`
reads as honoured.

The note now states the condition as the sum it is, `schedule_share` is that
term for one shard in integer arithmetic against `WHOLE_SNAPSHOT_CAPACITY`, and
`Publisher::tick` sums it and counts the ticks it cannot meet. A shard with
nothing published asks for nothing rather than for `tick`'s clamped set of one,
so a long document of empty channels does not add up to a shortfall nobody is
having. `run::TICK` becomes public because it is the serving rate: it is the one
number in the tick body that depends on the caller's interval, and it has to be,
since every other cadence there is a debt read off the clock.

**Counted, not refused.** The divisor is the published count, so at load there
is nothing to divide; a refusal on the first tick that could compute it would
darken a publisher that is already sending, over a shortfall that degrades into
a slower lap and never into a wrong answer, and whose remedy is a configuration
edit an operator has to be told about rather than one the process can make. The
tick loop names it on the decade schedule `worth_a_line` already states and the
exit report names the total.

Asserted: `every_shards_derived_tick_can_be_comfortable_while_the_process_is_not`
and `one_shard_below_the_process_tick_is_still_caught` on the arithmetic,
`a_share_is_the_process_tick_over_the_shards_own_derived_tick` and
`a_shard_with_nothing_published_asks_for_nothing` on its two edges, and
`cycles_that_are_achievable_per_shard_and_not_together_are_counted` on a
publisher — with `cycles_the_process_can_serve_are_not_counted` as the control,
without which the count passes against a counter that increments on every tick.
`ManifestSummary`'s `Instrument Count` and `Manifest Seq` describe one channel's
published set. `ArchivedRefdata` kept one manifest for a whole archive, chosen
by highest `Manifest Seq`, and `finalise` compared it against the union of every
channel's definitions. Over a capture of one channel that is the same
comparison; over a capture of two there is no arrangement in which it is right.
Both directions of the error are reachable at once — a complete channel reports
`ReferenceDataIncomplete` because the union exceeds its count, and a genuinely
short channel with the lower `Manifest Seq` reports nothing — and which one is
seen depends on nothing but how the archive is arranged.

The publisher side is not the thing to fix. A process-wide `Instrument Count`
would be a false statement on every channel it went out on, which is the whole
point of the change this reviews.

So: a manifest per `Channel ID`, the symbols each channel defined beside it, and
`finalise` a loop comparing each channel's count against its own definitions.
`Caveat::ReferenceDataIncomplete` names the channel — without that, two channels
short by the same numbers are one value and `push_once` collapses them into one
line identifying neither. `declared_instrument_count` takes a `Channel ID`: the
counts of several channels do not reduce to one number, and their sum is the
size of a union of disjoint published sets, which no manifest states.

The symbol table stays one for the archive, because the re-lowering resolves a
symbol without knowing which channel carried it, and splitting it would mean
resolving against a channel a payload archive cannot name.

**Per channel, not per channel instance.** `by_symbol` already unions the
redundant paths of a channel and raises `ScaleRestated` where they disagree, so
the reconstruction this check guards resolves from that union; keyed on the
instance it would raise a caveat against a path whose refdata window was shorter
even where the union covers the set and nothing is declined — a statement about
capture coverage dressed as one about the archive. The key is the datagram
header's `Channel ID` and not the summary's own field, because the header is the
only channel identity a definition has, and keying the two halves of one
comparison on two different fields would compare sets that are not the same
channel's.

**Not deferred**, having considered it: the honest alternative was to call this
its own piece of work and meanwhile refuse to run over an archive covering more
than one channel. That is right where the fix is large. Both call sites already
hold the `Channel ID`, the map is keyed on a `u8` and `finalise` becomes a loop,
so refusing would have cost a public error variant, a runner that stops on
captures it can read perfectly well, and the same work again later.

Asserted: `each_channels_manifest_is_checked_against_its_own_definitions` over
an archive whose complete channel holds the higher `Manifest Seq` and whose
short one holds the lower, so the union comparison is wrong on both channels at
once; `a_channel_with_no_manifest_of_its_own_borrows_no_other_channels_count`
for the channel nothing can be said about. `pack_on` and `refdata_datagrams_on`
are what let a fixture hold two channels at all — every fixture published on one
until now, which is why the suite passed over this.
#97 landed the same spec and plan from a head that predates two corrections, so
both files arrive as add/add conflicts. This branch's copies are the later ones
and are resolved as such — every line `main` carries is present here, replaced
rather than dropped:

- the task count, thirteen against the sixteen this branch now has;
- the bullet claiming the break is recorded in a changelog, corrected here to
  the tag message and the release notes, because there is no changelog in the
  tree and no crate carries its own version;
- the promise that an unknown shard name is *logged* once per distinct value,
  which is now *remembered* once per distinct value with task 14 being what
  reaches an operator with it;
- the four-revert heading, which is a longer table here.

Verified line by line rather than assumed, and the merged tree is green: 1434
passing, fmt and clippy clean, and the repository rules pass.

@bgm-malbeclabs bgm-malbeclabs left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve on correctness. Three notes, none blocking.

Per-tick cost grows with shards times instruments. Registry::definition_tick walks self.slots — the whole process instrument table — once per shard per tick and keeps only that shard's entries. SnapshotRotation::due does the same through on_shard. At the number this PR is named for, 31 shards of two specifications, that is 31 walks of a shared table every 10 ms. It is correct and it is bounded, and the comment in the emit loop is honest about it: "Every slot is walked and only this shard's are emitted." Worth a measured number before 62 channel instances go out, since the count is the whole point of the change.

ShardConfig.channel_id is whichever block came first. In run.rs:

channel_id: config.feeds.iter().find(|feed| feed.shard == shard)

A shard carrying both specifications has two channel ids, and this takes the one whose [[feed]] block sits earlier in the document. registry.rs argues why that is harmless — the datagram builder stamps the header at push, and one composed summary is truthful on both refdata ports — and I believe it. The call site does not say the choice is arbitrary. One line of comment there, because the next person to read find will read it as the channel id.

The era file naming holds. {feed}.{shard}.era cannot collide with another feed's {feed}.era, because is_one_path_component rejects . in both names. ShardName::era_shard is the single place the default token maps to Shard::DEFAULT, which is the mapping that most wants to be in one place: a renamed era file reads as no file, resolves to the first era, and the publisher restarts on era 1 saying nothing.

Two things I would copy elsewhere. Closing the channel_id collision that already existed, rather than only the one this change makes likely. And finding task 5 by running the plan's own revert clause instead of assuming it: the revert that failed nothing was the task nobody had written, which is the only way that class of miss gets found.

@elitegreg elitegreg left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I re-verified every fix claimed on this PR against the head rather than reading the replies, and the substantive ones hold up.

schedule_share and WHOLE_SNAPSHOT_CAPACITY are the right answer to the snapshot-budget finding: the per-shard divisor made every shard's arithmetic locally comfortable while the process-wide serving rate stayed at one per tick, and count_snapshot_schedule now measures the sum against what one process can send. Making it a counted overrun rather than a startup refusal is also right, since published counts move at runtime. The exit line naming the shared budget is what an operator would actually need.

DuplicateChannelId carrying first_shard and second_shard fixes a message that would otherwise have read "the blocks for top-of-book and top-of-book" in exactly the case the refusal exists for. take_unknown_shards has a caller now, and both shard counters get exit lines, which closes the gap where a venue misnaming a shard had no signal at all.

Four notes, none of them blocking. One is bookkeeping rather than code, and two are mine and previously unraised.

Comment thread rust/adapter/dz-adapter-core/src/lib.rs Outdated
Comment thread rust/publisher/dz-publisher-runtime/src/run.rs
Comment thread rust/publisher/dz-publisher-metrics/src/metrics/refdata.rs
Comment thread rust/publisher/dz-publisher-refdata/src/registry.rs
`ListingSink::list_on` justified its version bump with "every implementor of it
is in this workspace", in two places: the crate-level documentation and the
trait's own. Implementors outside this workspace exist — a test double for a
venue's own adapter implements the sink even where the adapter under test only
calls it — so the claim was untrue in both copies, and a claim stated twice is a
claim corrected once.

The correction itself was written on `jo/egress-ttl-and-a-boundary-correction`
and moves here, because the sentence is in this branch's diff and this branch
lands first. The review thread that raised it had been resolved against a commit
a later rebase removed, so nothing on this head pointed at a fix and the audit
would have read as answered.

What the trait chose is unchanged and the wording does not pretend otherwise:
what makes the break worth asking for is not that nobody pays it but where it
lands. A compile error in a test double is found by the next `cargo test`; a
published set silently collapsed onto one channel is found by a subscriber.

Documentation only. There is no behaviour here and no mutant to kill.
…k's ceiling

Two things a review found in what the shards work wrote, both documentation.

`PortOpener` was inserted directly above `open_ports`'s doc comment with no
blank line between them, so the whole port-role argument — why a dead refdata
socket or a dead snapshot socket ends the process rather than degrading — became
the trait's documentation, and `open_ports`, which moved below `compose_feeds`,
was left with none. The trait's rendered docs opened with three paragraphs about
socket failure scope and then started again at its own summary. The block goes
back onto the function it argues about; the trait keeps its summary and the two
paragraphs on why `RouteLookup` was not the missing seam.

`Registry::definition_tick` now states the ceiling `rotation.rs` already states
for the rotation. Both walk the same shared slot table for the same reason —
the slots are one table because an `Instrument ID` is — and someone sizing a
shard count would have read one and not the other. It is written down rather
than removed, and the arithmetic is why: the cursor persists across ticks and a
shard emits its whole published set once a lap, so a lap is exactly one
revolution of the table, `N * slots` visits a lap over the process. At 31 shards
of 100 instruments on the default 30 s cycle that is 40 slot visits a tick
against a 10 ms tick, and the walk is entered only on the one tick in
twenty-four a shard owes anything.

Measuring it found the term that is actually larger, which is not the one the
review named. `shard_index` resolves the shard name by scanning the configured
shards and comparing strings, and it runs before `due` is consulted, so it costs
N comparisons on every tick rather than on the ticks something is owed. Timed
over a whole lap, 31 shards of 100 cost 1.3 microseconds a tick with 1.1 of that
the name scan; 128 shards of 100 cost 18.6 with 15.0 the name scan. Both are far
enough inside a 10 ms tick to be stated rather than optimised, and `Channel ID`
being a `u8` caps any document at 256 shards regardless. The note says which of
the two to take first if either is ever taken, so that the next reader does not
buy the smaller one.

`rotation.rs` gains the pointer back, so the two ceilings are found together.

Documentation only. No behaviour changes and there is no mutant to kill; the
1434 tests pass unchanged.
… it is now

`lib.rs` still said `Manifest Seq` describes the published set rather than a
channel, and concluded that one composed manifest is truthful on every
feed's refdata port. That was true before the shards work and is false
after it: `Registry::manifest_seq` and `Registry::published_on` each take
the shard the channel carries, so both numbers are a channel's.

The paragraph now splits the two claims that used to sit in one sentence.
`Instrument ID` identity really is the one thing there can only be one of,
and that half is unchanged. The published set is not one, and the reason is
the one the recorder's completeness check needed: summing the counts over
shards states the size of a union of disjoint published sets, which no
manifest declares and no subscriber sees. A manifest is truthful on the
refdata port it leaves by, rather than on every one of them.

Documentation only. No link was added, so the crate's existing rustdoc
warnings are unchanged.
@juan-malbeclabs

Copy link
Copy Markdown
Contributor Author

@nikw9944 — both blocking items are fixed and verified on 51ba27d, and so is every item under them. Requesting a re-review, since the CHANGES_REQUESTED is the only thing holding this now (@elitegreg approved after re-verifying the fixes against the head).

Blocking 1 — a venue naming an unconfigured shard was silent. Fixed in 9ff24ee. take_unknown_shards has a caller: run.rs:962, in the poll loop beside poll_listings. It is named there rather than left to the exit report for the reason you gave — the instruments are being declined now, and the only series that could show it is a gauge at 0, which says a channel is empty and cannot say what was asked for. The registry hands each distinct name back once, so it is a line per name and not a line per poll, which matters against an adapter that re-offers its whole set every second.

Blocking 2 — per-shard ManifestSummary breaking the recorder's completeness check. Fixed in ead302b, with 2b5d547 recording the tier in the design first. You were right to send me to relower/src/refdata.rs:321 before deciding: over an archive of two channels the union comparison was wrong in both directions at once, and which one you saw depended on nothing but how the archive was arranged.

The fix is on the recorder's side, not the publisher's — a process-wide Instrument Count would be a false statement on every channel it went out on, which is the point of the change. The recorder now keeps a manifest per Channel ID with the symbols each channel defined beside it, and finalise compares each channel's declared count against its own definitions. Caveat::ReferenceDataIncomplete names the channel, because without that two channels short by the same numbers are the same value and push_once collapses them into one line identifying neither. Per channel rather than per channel instance, because by_symbol already unions a channel's redundant paths and raises ScaleRestated where they disagree.

Minor — the rotation divisor. schedule_share and WHOLE_SNAPSHOT_CAPACITY: the per-shard divisor made every shard's arithmetic locally comfortable while the process-wide serving rate stayed at one snapshot per tick, and count_snapshot_schedule now measures the sum against what one process can send. A counted overrun rather than a startup refusal, since published counts move at runtime, and the exit line names the shared budget.

Docs — both clauses gone, in 07a74a0. dz-adapter-core/src/lib.rs:93 and sink.rs:204 no longer claim every implementor is in this workspace. grep -rn "in this workspace" rust/adapter/dz-adapter-core/ returns nothing.

Optional — Manifest Seq. Taken, in 51ba27d. You were right that the crate doc outlived the change. dz-publisher-runtime/src/lib.rs said Manifest Seq describes the published set rather than a channel, and concluded one composed manifest is truthful on every refdata port. Registry::manifest_seq and Registry::published_on each take the shard the channel carries, so both numbers are a channel's and that conclusion is false. The paragraph now splits the two claims that shared a sentence: Instrument ID identity really is the one thing there can only be one of, and stays; the published set is not one, for the reason your blocking item needed — summing the counts over shards states the size of a union of disjoint published sets, which no manifest declares and no subscriber sees.

Gates on 51ba27d: cargo fmt --all --check clean, cargo clippy -p dz-publisher-runtime --all-targets -- -D warnings clean, doc tests passing. The crate's pre-existing rustdoc warnings are unchanged — the paragraph adds no link. CI green on all 14 checks.

@nikw9944 nikw9944 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm — both remaining documentation findings are verified fixed at 51ba27d: 07a74a0 corrects the boundary clause in both lib.rs and sink.rs, and 51ba27d rewrites the runtime crate doc to describe the published set as a shard's. Nothing open.

@juan-malbeclabs
juan-malbeclabs merged commit 71c7499 into main Sep 10, 2026
14 checks passed
@juan-malbeclabs
juan-malbeclabs deleted the jo/publisher-feed-routes branch September 10, 2026 15:28
juan-malbeclabs added a commit that referenced this pull request Sep 10, 2026
**Stacked on #98**, which carries the shards work; this diff is the
upstream write. Split out of #98 at a reviewer's request: it changes
behaviour on its own and should not have to be approved in order to
approve the shards.

## The failure, which is measured rather than argued

`UpstreamSink` reaches an adapter in exactly one method —
`on_connected`, called on every successful connect — and `poll_listings`
is handed a `ListingSink` and nothing else. So an instrument discovered
mid-session can be admitted completely: minted an `Instrument ID`,
defined on the reference-data port, counted in the manifest, given a
`Manifest Seq` bump. And never subscribed, because the subscription was
composed at logon from the set the adapter held then. Nothing reports
it. The feed is healthy, the manifest says the instrument is published,
and no message for it will ever arrive.

Over thirty days on one venue, 5 instruments were listed in two
mid-session batches. A publisher reading over a websocket carried all
five inside one 17-day process lifetime, because its transport
re-subscribes on its own poll. A publisher of the same venue reading
over a session transport carried none of them until its next restart —
10.35 and 7.32 days later. Same boundary, same instruments; the
difference is whether the transport happened to have a reason of its own
to write again.

## Why neither suggested shape works

Two were offered: a second argument to `poll_listings`, or a defaulted
`listings_changed(&mut dyn UpstreamSink)`. Both are refused for one
reason that is fatal and one that follows from it.

**Neither names the connection, and the write has to.** `on_connected`
already argues this at length, because it is where the same mistake was
found before: one adapter serves every source a publisher opens, so
state that belongs to a connection has to be stored per `conn` — "an
adapter that keeps one upstream sequence cursor, or one authentication
token, or one 'have I subscribed yet' flag, is correct with one source
and wrong the moment a second is configured — and the way it is wrong is
silent". An `UpstreamSink` with no `ConnectionId` beside it is exactly
that flag, handed to the adapter by the boundary.

**And there is nowhere to send it from.** `poll_listings` runs in the
runtime's tick loop, which holds the adapter and no transport: the
drivers own the `Input`s, one per source. A sink handed to the tick loop
would be a queue whose consumer is *some* driver, and picking one is
picking the connection the adapter was supposed to name.

## The shape

One defaulted `poll_upstream(conn, out)` on the boundary, asked by each
driver on its own connection when the cadence is due, flushed through
the same rate limit a logon pays.

Named for the mechanism rather than for the listing change, because the
repair path is a second cause — a mid-session snapshot request had
nowhere to go either — and a venue's own reasons are a third. **No
reason parameter**: the adapter is the only thing that knows what it has
already written on that connection, and a reason would invite state
keyed on the reason rather than on `conn`.

**The runtime does not change. Not one line.** `poll_listings` admits,
the adapter records the handle, the driver asks, the adapter writes what
it has not written on that connection.

**This write is not deduplicated, and the doc comment says so in those
words.** Re-offering a listing is free because the sink returns the
handle already minted; this reaches a venue, and nothing here
understands a venue's bytes. Getting it wrong looks from here like a
venue rate limiting a publisher that is working perfectly.

**Two asymmetries with `on_connected`, both argued.** A refusal is
counted and the connection survives, because mid-session it is still
delivering everything subscribed at logon and dropping it would cost
those to save the one. A failed send ends the connection through the
path that already exists, and the reconnect writes the whole set again,
so nothing has to be reconciled.

`UPSTREAM_POLL` is five seconds and deliberately slower than the
runtime's one-second listing poll: that one is affordable by
construction, and this one is a venue's inbox. Its doc also states what
it does **not** do — fire on a silent connection. Bounding the receive
budget by the cadence would have made an elapsed budget mean two things,
and `Received::Idle` means exactly one.

## Testing Verification

- `cargo test --workspace` — **1418 passing, 0 failed** on this head
(the shards base is 1413; five tests are new here), over the scripted
transport and the test clock.
- `cargo fmt --all --check` and `cargo clippy --all-targets -- -D
warnings` clean. `scripts/check-public-repo-rules.sh` exits 0.
- **The centre revert corrected its own test.** Moving the call to the
connect path, beside `on_connected`, failed nothing on its first run:
the test asserted "asked once, on this connection, and the message
reached `send`", every word of which a call at logon satisfies. It now
records how many payloads the adapter had seen when it was asked and
asserts one; a connect-time ask records zero. With that, the revert
fails three tests.
- A second revert — dropping the due-ness check and asking on every
receive — fails two, because the count is then per payload.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants