@@ -107,54 +107,25 @@ non-obvious parts:
107107
108108** Versioning.** Buckets carry a versioning state — ` Unversioned ` (default), ` Enabled ` , or
109109` Suspended ` .
110- - Versions of a key are grouped and ordered ** newest-first ** in the MST by a composite key, so the
111- current version is the head leaf and version-scoped reads are direct seeks — see * Versioned key
112- encoding * below.
110+ - Versions of a key are grouped under its ` ObjectLeaf ` : the current version inline (single-seek
111+ reads), noncurrent versions in a per-key sub-MST ordered ** newest-first ** , version-scoped reads
112+ direct seeks — see * Versioned storage * below.
113113- ` ListObjects ` (V1/V2) returns only the current, non-delete-marked version per key (one head per
114114 key, skip to the next). ` ListObjectVersions ` walks all leaves. Delete markers are skipped by the
115115 former, surfaced by the latter (with the ` x-amz-delete-marker ` response header).
116116- Overwrite depends on state: ` Enabled ` creates a new version and retains the prior one;
117117 ` Suspended ` /` Unversioned ` replaces the ` null ` version in place and drives the superseded data's
118118 delete through the reference index ([ §5] ( #5-the-data-layer ) ).
119119
120- ** Versioned key encoding (` invertedVersionId ` ).** Because the MST iterates forward only, the current
121- version is made the head leaf of a key's group — reachable with a single seek — by storing versions
122- under the composite key ` escape(objectKey) ++ TERM ++ invertedVersionId ` , whose trailing component
123- sorts ** newest-first** :
124- - ** ordinal** — each version gets a per-bucket monotonic sequence (` buckets.next_version_seq ` ,
125- advanced atomically in the commit path; gaps from retried commits are harmless). It is not
126- wall-clock, which avoids clock skew and cross-instance disagreement.
127- - ** ` invertedVersionId ` ** — the ordinal's numeric inverse (` 0xFFFF…FFFF − seq ` ) rendered as a
128- ** fixed-width 16-char lowercase hex** string. Hex is order-preserving, NUL-free, and valid UTF-8;
129- the inverse makes the newest version (largest seq) the lexicographically smallest token, so it
130- sorts first within the key's group.
131- - ** ` escape(objectKey) ++ TERM ` ** — an order-preserving, self-terminating encoding so a key's version
132- group stays contiguous and is never entered by a prefix-sharing neighbor (` a.png ` vs ` a.png2 ` ):
133- escape ` 0x01 ` inside the key as ` 0x01 0x02 ` , then terminate with ` TERM = 0x01 0x01 ` , which sorts
134- below any continuation (both bytes are non-NUL and valid UTF-8).
135- - ** client ` versionId ` ** — the ` x-amz-version-id ` is the ` invertedVersionId ` token itself, so a
136- version-scoped request reconstructs the exact key and resolves in a direct O(log n) seek, no scan.
137-
138- Reads follow from the layout: current = the first leaf of the group; version-scoped = the
139- reconstructed key; ` ListObjectVersions ` walks the group (already newest-first); ` ListObjects ` reads
140- each head and seeks past the group to the next key. The token + terminator costs ~ 18 bytes (more if a
141- key itself contains ` 0x01 ` ). ` MaxKeyBytes ` is currently 1024 to match S3's key limit; it can be
142- raised (e.g. to ~ 1056) so the version overhead does not shrink the usable S3 key length, at the cost
143- of a slightly larger maximum leaf.
144-
145- > ** Design decision (for review).** Two choices are baked into this encoding; the defaults below are
146- > chosen but flagged for the team to ratify.
147- >
148- > - ** Composite key (chosen) vs. per-key version index.** Keying the MST by ` (key, version) ` keeps all
149- > versions in one sorted space and amortizes well for many-versioned keys. The alternative keys the
150- > MST by object key alone, with the leaf pointing at an explicit newest-first version index — which
151- > removes the inversion, the escaping, and the key-budget pressure, but rewrites and grows that
152- > index block on every new version. Revisit if hot keys accumulate very many versions.
153- > - ** ` versionId ` as a direct locator (chosen) vs. opaque.** Setting ` versionId = invertedVersionId `
154- > gives a scan-free, version-scoped lookup, but it ** leaks version ordering and approximate write
155- > count** (a smaller id is newer). AWS-style opacity would require a random id plus a
156- > ` versionId → seq ` side map (or an HMAC of the seq), losing the direct-locator property. We keep the
157- > direct locator; the opacity trade-off is left for the team to weigh.
120+ ** Versioned storage (the per-key version tree).** The full design is
121+ ** [ ` s3-versioning.md ` ] ( ./s3-versioning.md ) ** . In brief: the top MST is keyed by the ** plain object
122+ key** ; each leaf is an ` ObjectLeaf ` ` {current, prev, nullSeq} ` where ` current ` is the head version
123+ (single-seek reads) and ` prev ` is a per-key sub-MST of noncurrent versions keyed newest-first by
124+ the inverted per-bucket ordinal (` seq ` , from ` buckets.next_version_seq ` ). ` seq ` (ordering,
125+ internal) and ` version_id ` (identity, the client handle — a ULID token, or ` "null" ` ) are
126+ ** separate fields** , so the null version's replace-in-place semantics fall out instead of needing
127+ a sentinel. Object keys need no escaping and no terminator, and fit ` MaxKeyBytes ` as-is; the
128+ seq-derived token reveals write ordering, which is accepted (S3 ids are opaque to clients).
158129
159130** ETags.** The ETag is MD5-based, never the sha256 content digest. A whole-object ETag is the MD5 of
160131the body; a multipart object's ETag is ` hex(md5(concat of the N part MD5s)) + "-N" ` (matching
@@ -212,11 +183,16 @@ depends on the single- vs multi-shard split in [§8](#8-retrieval-addressing-whe
212183
213184``` shell
214185MST (bucket)
215- leaf key = compositeKey( " photos/cat.jpg" , invertedVersionId )
186+ leaf key = " photos/cat.jpg" (plain object key )
216187 │
217- └──▶ CID ──▶ ObjectManifest ← one dag-cbor block, catalog plane
188+ └──▶ CID ──▶ ObjectLeaf ← per-key version group (s3-versioning.md §2)
189+ ├ current { seq, versionId, manifest CID } (the head version, inline)
190+ ├ prev CID of the per-key sub-MST of noncurrent versions (nil if none)
191+ └ nullSeq a noncurrent null version' s seq (0 = none)
192+
193+ ObjectManifest ← one dag-cbor block, catalog plane
218194 ├ key "photos/cat.jpg"
219- ├ versionId " v_01J8QX …" (MST key holds an inverted form )
195+ ├ seq/ versionId "01J8QX …" (identity; cached on the leaf ' s current )
220196 ├ created " 2026-06-17T" … (last-modified)
221197 ├ etag " 9b2cf…-3" (md5-of-md5s-N if multipart, else md5 hex)
222198 ├ contentType + http headers + user metadata
@@ -586,9 +562,6 @@ The first cut keeps digest-before-upload; `allocate-by-size` is a same-rack fast
586562 rotation.
587563- ** Compaction trigger policy** , the ** ` allocate-by-size` ** security model, and the ** batch**
588564 allocate/accept wire format.
589- - ** Versioned key encoding** ([§3](# 3-the-s3-layer)) — ratify composite-key vs. per-key version index, and
590- direct-locator vs. opaque ` versionId` (the direct locator leaks version ordering / approximate
591- write count); and whether to raise ` MaxKeyBytes` so versioning doesn' t shrink the usable S3 key.
592565
593566---
594567
@@ -661,7 +634,7 @@ CREATE TABLE ingot.buckets (
661634 CHECK (versioning IN (' unversioned' ,' enabled' ,' suspended' )),
662635 root_cid bytea, -- committed MST root (locally durable)
663636 forge_root_cid bytea, -- MST root durable on Forge (lags root_cid)
664- next_version_seq bigint NOT NULL DEFAULT 0, -- per-bucket version ordinal (§3, invertedVersionId )
637+ next_version_seq bigint NOT NULL DEFAULT 0, -- per-bucket version ordinal (§3, the version seq )
665638 created_at timestamptz NOT NULL DEFAULT now()
666639);
667640CREATE INDEX buckets_space_idx ON ingot.buckets (space);
@@ -779,12 +752,6 @@ The storage / delete / retrieval core is built and tested:
779752
780753These are intentional simplifications of the target topology, not bugs:
781754
782- - ** S3 versioning is not implemented.** Buckets are effectively ` Unversioned` : an overwrite replaces
783- in place and drives the prior data through the reference index. The ` versionId` machinery in
784- [§3](# 3-the-s3-layer) (composite key, `invertedVersionId`, `ListObjectVersions`, delete markers, `next_version_seq`)
785- is ** not** built. The schema is designed toward it — ` blob_refs.version_id` carries the constant
786- ` " null" ` sentinel, ` buckets.versioning` /` next_version_seq` and ` ObjectManifest.deleteMarker` are
787- reserved — so adding versioning later needs no manifest/MST-key migration.
788755- ** Deployed next to Piri.** The same-rack topology of [§10](# 10-deployment-topology--the-digest-before-upload-cost) is assumed; digest-before-upload is
789756 kept and ` allocate-by-size` is out of scope.
790757- ** No Ingot-side aggregation.** Ingot ships each body blob to Piri as-is; ` min_aggregate_size` ,
0 commit comments