Skip to content

Commit 5855ce7

Browse files
frristclaude
andcommitted
docs(architecture): align §3 with the per-key version-tree design
Point §3's versioned-storage section at docs/s3-versioning.md, replace the composite-key bullet and manifest diagram with the ObjectLeaf shape, update the schema comment, and drop the resolved open question and the versioning entry from the deferred list. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent dc8d099 commit 5855ce7

1 file changed

Lines changed: 21 additions & 54 deletions

File tree

docs/architecture.md

Lines changed: 21 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -107,54 +107,25 @@ non-obvious parts:
107107

108108
**Versioning.** Buckets carry a versioning state — `Unversioned` (default), `Enabled`, or
109109
`Suspended`.
110-
- Versions of a key are grouped and ordered **newest-first** in the MST by a composite key, so the
111-
current version is the head leaf and version-scoped reads are direct seeks — see *Versioned key
112-
encoding* below.
110+
- Versions of a key are grouped under its `ObjectLeaf`: the current version inline (single-seek
111+
reads), noncurrent versions in a per-key sub-MST ordered **newest-first**, version-scoped reads
112+
direct seeks — see *Versioned storage* below.
113113
- `ListObjects` (V1/V2) returns only the current, non-delete-marked version per key (one head per
114114
key, skip to the next). `ListObjectVersions` walks all leaves. Delete markers are skipped by the
115115
former, surfaced by the latter (with the `x-amz-delete-marker` response header).
116116
- Overwrite depends on state: `Enabled` creates a new version and retains the prior one;
117117
`Suspended`/`Unversioned` replaces the `null` version in place and drives the superseded data's
118118
delete through the reference index ([§5](#5-the-data-layer)).
119119

120-
**Versioned key encoding (`invertedVersionId`).** Because the MST iterates forward only, the current
121-
version is made the head leaf of a key's group — reachable with a single seek — by storing versions
122-
under the composite key `escape(objectKey) ++ TERM ++ invertedVersionId`, whose trailing component
123-
sorts **newest-first**:
124-
- **ordinal** — each version gets a per-bucket monotonic sequence (`buckets.next_version_seq`,
125-
advanced atomically in the commit path; gaps from retried commits are harmless). It is not
126-
wall-clock, which avoids clock skew and cross-instance disagreement.
127-
- **`invertedVersionId`** — the ordinal's numeric inverse (`0xFFFF…FFFF − seq`) rendered as a
128-
**fixed-width 16-char lowercase hex** string. Hex is order-preserving, NUL-free, and valid UTF-8;
129-
the inverse makes the newest version (largest seq) the lexicographically smallest token, so it
130-
sorts first within the key's group.
131-
- **`escape(objectKey) ++ TERM`** — an order-preserving, self-terminating encoding so a key's version
132-
group stays contiguous and is never entered by a prefix-sharing neighbor (`a.png` vs `a.png2`):
133-
escape `0x01` inside the key as `0x01 0x02`, then terminate with `TERM = 0x01 0x01`, which sorts
134-
below any continuation (both bytes are non-NUL and valid UTF-8).
135-
- **client `versionId`** — the `x-amz-version-id` is the `invertedVersionId` token itself, so a
136-
version-scoped request reconstructs the exact key and resolves in a direct O(log n) seek, no scan.
137-
138-
Reads follow from the layout: current = the first leaf of the group; version-scoped = the
139-
reconstructed key; `ListObjectVersions` walks the group (already newest-first); `ListObjects` reads
140-
each head and seeks past the group to the next key. The token + terminator costs ~18 bytes (more if a
141-
key itself contains `0x01`). `MaxKeyBytes` is currently 1024 to match S3's key limit; it can be
142-
raised (e.g. to ~1056) so the version overhead does not shrink the usable S3 key length, at the cost
143-
of a slightly larger maximum leaf.
144-
145-
> **Design decision (for review).** Two choices are baked into this encoding; the defaults below are
146-
> chosen but flagged for the team to ratify.
147-
>
148-
> - **Composite key (chosen) vs. per-key version index.** Keying the MST by `(key, version)` keeps all
149-
> versions in one sorted space and amortizes well for many-versioned keys. The alternative keys the
150-
> MST by object key alone, with the leaf pointing at an explicit newest-first version index — which
151-
> removes the inversion, the escaping, and the key-budget pressure, but rewrites and grows that
152-
> index block on every new version. Revisit if hot keys accumulate very many versions.
153-
> - **`versionId` as a direct locator (chosen) vs. opaque.** Setting `versionId = invertedVersionId`
154-
> gives a scan-free, version-scoped lookup, but it **leaks version ordering and approximate write
155-
> count** (a smaller id is newer). AWS-style opacity would require a random id plus a
156-
> `versionId → seq` side map (or an HMAC of the seq), losing the direct-locator property. We keep the
157-
> direct locator; the opacity trade-off is left for the team to weigh.
120+
**Versioned storage (the per-key version tree).** The full design is
121+
**[`s3-versioning.md`](./s3-versioning.md)**. In brief: the top MST is keyed by the **plain object
122+
key**; each leaf is an `ObjectLeaf` `{current, prev, nullSeq}` where `current` is the head version
123+
(single-seek reads) and `prev` is a per-key sub-MST of noncurrent versions keyed newest-first by
124+
the inverted per-bucket ordinal (`seq`, from `buckets.next_version_seq`). `seq` (ordering,
125+
internal) and `version_id` (identity, the client handle — a ULID token, or `"null"`) are
126+
**separate fields**, so the null version's replace-in-place semantics fall out instead of needing
127+
a sentinel. Object keys need no escaping and no terminator, and fit `MaxKeyBytes` as-is; the
128+
seq-derived token reveals write ordering, which is accepted (S3 ids are opaque to clients).
158129

159130
**ETags.** The ETag is MD5-based, never the sha256 content digest. A whole-object ETag is the MD5 of
160131
the body; a multipart object's ETag is `hex(md5(concat of the N part MD5s)) + "-N"` (matching
@@ -212,11 +183,16 @@ depends on the single- vs multi-shard split in [§8](#8-retrieval-addressing-whe
212183

213184
```shell
214185
MST (bucket)
215-
leaf key = compositeKey("photos/cat.jpg", invertedVersionId)
186+
leaf key = "photos/cat.jpg" (plain object key)
216187
217-
└──▶ CID ──▶ ObjectManifest ← one dag-cbor block, catalog plane
188+
└──▶ CID ──▶ ObjectLeaf ← per-key version group (s3-versioning.md §2)
189+
├ current { seq, versionId, manifest CID } (the head version, inline)
190+
├ prev CID of the per-key sub-MST of noncurrent versions (nil if none)
191+
└ nullSeq a noncurrent null version's seq (0 = none)
192+
193+
ObjectManifest ← one dag-cbor block, catalog plane
218194
├ key "photos/cat.jpg"
219-
├ versionId "v_01J8QX" (MST key holds an inverted form)
195+
seq/versionId "01J8QX…" (identity; cached on the leaf's current)
220196
├ created "2026-06-17T"… (last-modified)
221197
├ etag "9b2cf…-3" (md5-of-md5s-N if multipart, else md5 hex)
222198
├ contentType + http headers + user metadata
@@ -586,9 +562,6 @@ The first cut keeps digest-before-upload; `allocate-by-size` is a same-rack fast
586562
rotation.
587563
- **Compaction trigger policy**, the **`allocate-by-size`** security model, and the **batch**
588564
allocate/accept wire format.
589-
- **Versioned key encoding** ([§3](#3-the-s3-layer)) — ratify composite-key vs. per-key version index, and
590-
direct-locator vs. opaque `versionId` (the direct locator leaks version ordering / approximate
591-
write count); and whether to raise `MaxKeyBytes` so versioning doesn't shrink the usable S3 key.
592565
593566
---
594567
@@ -661,7 +634,7 @@ CREATE TABLE ingot.buckets (
661634
CHECK (versioning IN ('unversioned','enabled','suspended')),
662635
root_cid bytea, -- committed MST root (locally durable)
663636
forge_root_cid bytea, -- MST root durable on Forge (lags root_cid)
664-
next_version_seq bigint NOT NULL DEFAULT 0, -- per-bucket version ordinal (§3, invertedVersionId)
637+
next_version_seq bigint NOT NULL DEFAULT 0, -- per-bucket version ordinal (§3, the version seq)
665638
created_at timestamptz NOT NULL DEFAULT now()
666639
);
667640
CREATE INDEX buckets_space_idx ON ingot.buckets (space);
@@ -779,12 +752,6 @@ The storage / delete / retrieval core is built and tested:
779752
780753
These are intentional simplifications of the target topology, not bugs:
781754
782-
- **S3 versioning is not implemented.** Buckets are effectively `Unversioned`: an overwrite replaces
783-
in place and drives the prior data through the reference index. The `versionId` machinery in
784-
[§3](#3-the-s3-layer) (composite key, `invertedVersionId`, `ListObjectVersions`, delete markers, `next_version_seq`)
785-
is **not** built. The schema is designed toward it — `blob_refs.version_id` carries the constant
786-
`"null"` sentinel, `buckets.versioning`/`next_version_seq` and `ObjectManifest.deleteMarker` are
787-
reserved — so adding versioning later needs no manifest/MST-key migration.
788755
- **Deployed next to Piri.** The same-rack topology of [§10](#10-deployment-topology--the-digest-before-upload-cost) is assumed; digest-before-upload is
789756
kept and `allocate-by-size` is out of scope.
790757
- **No Ingot-side aggregation.** Ingot ships each body blob to Piri as-is; `min_aggregate_size`,

0 commit comments

Comments
 (0)