products: the served catalog is insert-only and never prunes on a manifest epoch change

[armcconnell]

Summary

The /v1 product catalog is served from a map that is only ever inserted into. It never prunes on a manifest epoch change, so retired instruments accumulate for the lifetime of the process and are served as though they were live.

The reference-data supplement exists precisely so a subscriber can detect removals, and we already implement that state machine correctly — it just isn't what the API reads from.

Two parallel instrument maps

Map	Fed by	Pruned on epoch change?	Used for
RefDataState.defs (src/ingest/subscriber.rs:30)	refdata port	yes	gating quote emission
InstrumentSnapshot (src/model.rs:460)	upsert_instrument	no	/v1/products

RefDataState follows the supplement's subscriber algorithm as written: on_manifest (subscriber.rs:67) clears defs when Manifest Seq advances, and on_instrument_definition (:83) only admits definitions tagged with the current latest_seq. An instrument dropped from the publisher's active set falls out of that map within one definition cycle, which is correct.

upsert_instrument (src/ingest/processor.rs:208) only inserts. remove_instrument exists at :242, but all four call sites (:347, :714, :985, :1714) are the Source-ID-change branch of reveal_if_needed. Its own doc comment states the problem plainly:

There is no other removal path for this map in the crate, so anything that stops naming an identity … must call this explicitly, or the stale entry sits in the connect-time replay snapshot for the life of the process

Manifest-driven departure is exactly such a case, and nothing calls it.

Evidence

Measured on a live host, one market-by-price channel, ~39h process uptime:

Source	Instruments on that channel
ManifestSummary on the wire (ground truth)	5,572
/v1/products filtered to that channel	12,498
status → that channel's products	2,080

So roughly 55% of what the catalog serves for that channel is retired. Full catalog was 101,666 products and grows monotonically. Instruments that settled ~40h earlier still resolve through products get, with an empty book and an empty tape but a live-looking catalog entry.

The third number is a separate puzzle: status.products reports 2,080 for the same channel, which matches neither the manifest count nor the catalog size. Whatever it means, no field currently available to a consumer answers "is this instrument still active?"

Related: status is venue health, not instrument state

src/sinks/api.rs:376 renders the product's status as:

"status": if state.health.venue_up(i.venue.as_ref()) { "online" } else { "offline" },

That is a property of the venue, not the instrument — every product on a reachable venue reads online regardless of lifecycle. Combined with the above, a consumer sees a settled instrument as online with no field that says otherwise. Worth deciding whether an instrument-level state belongs here, since the field name already implies one.

Suggested direction

1. Derive the served catalog from RefDataState.defs rather than maintaining a second insert-only map — the pruning is already correct there, so this removes the divergence rather than adding a second mechanism to keep in sync.
2. Failing that, drive remove_instrument from the epoch transition in on_manifest.
3. Either way, reconcile the three counts above and decide whether retired instruments are dropped or retained-and-flagged. Retained-and-flagged is likely right so history queries still resolve, but then the flag has to be visible in /v1.

Requires no publisher-side change: manifest-driven removal already works correctly on the wire today.

Verified against the running revision 589021d1.