Skip to content

Commit 89ca84a

Browse files
gregoryfosterclaude
andcommitted
#119 [docs]: the fact streams keep no retention, by decision; triggers recorded
content.blobs and content.artifacts stay bounded by maxmemory alone. STREAMS.md records the XLEN triggers and the MINID trim that would follow; the fetch, replicate and persist contracts state that neither stream is a history, and that any trim spares unread facts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 4850b1a commit 89ca84a

4 files changed

Lines changed: 13 additions & 2 deletions

File tree

‎docs/STREAMS.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ what each stream carries is here:
1717

1818
- **`content.persist` is the third command loop, off unless an operator turns it on (#114 step 6)** — on `co-replicator` since 2026-09-26. Its own group (`replicator.persist`) and dedupe namespace; both outcomes (`blob_persisted`, `persist_failed`) go to **`content.artifacts`**, beside the replicate pair, so the issuer's one group sees both. It copies a blob into the permanent content-addressed store, `gs://<REPLICATOR_PERMANENT_BUCKET>/blobs/<sha256>.bin`, create-if-absent with touch off, after T3a on `blob_uri` and a check that the URI's digest is `content_fingerprint`. The store re-hashes on the create (cannobserv#492), so a corrupted temp blob is `source_corrupt` and never stored. No URL and no domain echo on either fact: the digest is the address. **`REPLICATOR_PERSIST_ENABLED` defaults off** because the loop's boot-time `XGROUP CREATE` does not retry, and a broker that has not granted the stream (broker#64) would stop the worker. Contract: [content-persist-issuer-contract.md](contracts/content-persist-issuer-contract.md).
1919

20+
- **The two fact streams have no retention, by decision (#119).** `content.blobs` and `content.artifacts` are bounded by the broker's `maxmemory` alone — nothing here trims (#106) — and consumers are promised no history ([fetch contract](contracts/content-fetch-issuer-contract.md#what-replicator-does-not-guarantee)). At 2026-09-28's ~28 `content.blobs` facts a day and a 5.92 MB broker of 512 MB, the bound is years away. **Revisit when a trigger fires:** `rcli XLEN content.blobs` past 25k, `content.artifacts` past 10k, replicate or persist fan-out at production scale, or a broker#61 memory finding naming either. **The trim would then be `XTRIM MINID`** below every group's settled position (the lower of last-delivered and lowest pending id), and on `content.blobs` below `now − REPLICATOR_BLOB_TTL_SECONDS` too — a fact past `blob_expires_at` names reaped bytes. That takes `+xtrim` and `+xinfo|groups` from broker, and `test_nothing_here_trims_a_stream` narrowing to refuse only `MAXLEN`, which no ACL can tell apart. Not archiver#267's `XDEL` by id: Replicator consumes neither stream, so no fact has a closed state it can see.
21+
2022
- **The alias is a selector and the guards are allow-lists.** `credentials_alias` names a binding an operator provisioned on *this host* (`src/worker/aliases.py`, from `REPLICATOR_REPLICATION_ALIASES_FILE`); unset means nothing is provisioned and everything is refused, which is the current state of every host and the safe default under T5. Guard order is load-bearing and asserted: alias → provider → the alias's own writer → destination → source, so every refusal happens **before any credential is touched** (T1) — observable for the first time now that there is a driver call to precede. **The writers are keyed by alias, not by provider** (CR #26): `AsyncGcsDriver` takes a bucket in its constructor and never sees another, so a driver *is* a bucket and the key has to be whatever selects one — keyed by provider, two `gcs` bindings collapsed onto a single driver and a command could land in a bucket its binding never named, outside the very T3 root `validate_destination` had just checked. A binding whose driver cannot be built (ADC resolves in that constructor) is **skipped and logged, never raised**, because `load_alias_table` promises one line earlier that a replicate misconfiguration will not take down a worker whose actual job is `content.fetch`; the alias is then refused `provider_disabled`, whose remedy is the operator act that fixes it. The blob reaches the driver as a **seekable binary stream** from `BlobStore.open_stream`, not as bytes or a path: a path would make the provider copy something already on disk, bytes would pull a whole artifact into memory, and seekable is required because the driver reads the local md5 only on the 412 path, after the failed create has moved the position. **`blob_uri` is never resolved as a path** — the fingerprint is extracted, validated as 64 lowercase hex, and compared against `uri_for()` of each store this host reads — the temp store, and since #114 the permanent store if `REPLICATOR_PERMANENT_BUCKET` names one — so the only string reaching storage is one a store built, and the bytes come from the store that matched. That is T3a, and it is sharper than the destination guard it was added beside: `file:///etc/replicator/co-pypi-reader.json` as a `blob_uri` would otherwise publish this host's GCS reader key to a permanent, public, undeletable store. `invalid_source` and `blob_expired` stay distinct because only the second is fixed by fetching again. Two charter invariants enforce the rest: the alias is a key and never a value, and no payload field feeds a credential-shaped parameter.
2123

2224
- **The failure fact is a seam, not a call in the loop.** `FailureReporter` is injected exactly as `Handler` is, so `loop.py` stays ignorant of `content.blobs` and `blobs_topic` stays a defaulted argument a live-broker test can move. The loop owns the *decision* (`terminal` is "did this hit the delivery ceiling", which only the loop knows); the reporter owns the *publish*. `_close()` publishes then dead-letters, so **fact-before-ack** cannot be got wrong one call site at a time — `dead_letter` acks inside itself, and a fact published after it is lost outright on a crash. A failed fact-publish is **swallowed**, deliberately asymmetric with the byte path's `_publish`: there raising prevents an orphan blob, here the DLQ entry is already the durable record and raising would burn the delivery ceiling to reach the same DLQ minutes later.

‎docs/contracts/content-fetch-issuer-contract.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -375,8 +375,11 @@ Two corollaries, both about values you have stored:
375375
design — that is what makes MUST-1 work.
376376
- **No retention guarantee on `content.blobs`.** Replicator never trims it — `BusPublish` takes no
377377
`MAXLEN` and nothing here issues `XTRIM`, which `tests/test_broker_keyspace.py` holds (#106) —
378-
so whatever policy applies is the broker operator's, not part of this contract. It is not an
379-
archive to reconcile against later.
378+
so the broker's `maxmemory` is its only bound, by decision (#119). It is not an archive to
379+
reconcile against later: a group created after a fact was published is promised nothing
380+
about it, and a `blob_available` past its `blob_expires_at` names bytes already reaped. The one
381+
promise is negative: **if Replicator ever trims, it trims only facts every existing consumer
382+
group has read and acknowledged**, never by length.
380383

381384
---
382385

‎docs/contracts/content-persist-issuer-contract.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,8 @@ the write time.** For the earliest time the bytes were kept, take the minimum ov
7878
**P3 — Branch on `terminal`, then treat an unknown `reason` as opaque.** Every fact this service emits
7979
today is terminal. A failure that is still retrying publishes nothing, so silence means "still trying".
8080
**Keep a reaper** that re-issues under a fresh `command_id`, as the fetch contract's MUST-6 requires.
81+
`content.artifacts` is not a history either: its retention terms are the replicate contract's
82+
(#119), so read its facts as they arrive.
8183

8284
**P4 — Send the raw-bytes digest and the URI the fetch fact gave you, verbatim.** Do not rebuild
8385
`blob_uri` from a bucket name. The URI a store minted is the only one T3a accepts.

‎docs/contracts/content-replicate-issuer-contract.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -341,6 +341,10 @@ revision, its timestamp, or the fingerprint — rather than relying on a provide
341341
⚙ **R3 — do not treat `public_url` as stable across occasions.** Each replication occasion yields its
342342
own artifact and its own URL; the registry row records which.
343343

344+
⚙ **`content.artifacts` is not a history (#119)** — `content.blobs`'
345+
[retention terms](content-fetch-issuer-contract.md#what-replicator-does-not-guarantee) apply;
346+
a fact your group never read is MUST-6's, and the reaper its backstop.
347+
344348
---
345349

346350
## What Replicator refuses

0 commit comments

Comments
 (0)