Skip to content

Commit 84d7769

Browse files
rupurtrupurt
committed
Plan per-stream retention delivery slice
• [MSN] 1 mission driving 1 epic forward • [EXC] 1 story executing across 1 voyage • [HLT] 1 warnings, no structural errors detected
1 parent a232dfa commit 84d7769

12 files changed

Lines changed: 464 additions & 0 deletions

File tree

‎.keel/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,12 @@
201201
|--------|--------|
202202
| [Deliver Hosted Transport Robustness Improvements](epics/VHRmIhDsm/voyages/VHRmIjGvL/) | done |
203203

204+
### [Add Per-Stream Retention Policies And Retained Frontier Semantics](epics/VHUAlZWZG/) (active)
205+
206+
| Voyage | Status |
207+
|--------|--------|
208+
| [Deliver Per-Stream Retention Configuration And Visibility](epics/VHUAlZWZG/voyages/VHUApus0L/) | in-progress |
209+
204210
### [Research Branch-Aware Materialization And Processing](epics/VDd0u3PFg/) (done)
205211

206212
| Voyage | Status |

‎.keel/epics/VHUAlZWZG/PRD.md‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# Add Per-Stream Retention Policies And Retained Frontier Semantics - Product Requirements
2+
3+
## Problem Statement
4+
5+
Transit needs explicit per-stream retention controls for age- and size-bounded history, plus stream status surfaces that show configured retention and the retained frontier, without changing the default append-only behavior.
6+
7+
## Goals & Objectives
8+
9+
| ID | Goal | Success Metric | Target |
10+
|----|------|----------------|--------|
11+
| GOAL-01 | Let operators configure per-stream retention explicitly while keeping Transit’s default history-unbounded behavior unchanged. | Targeted tests and CLI proof coverage show `stream create`, `streams list`, and stream status surfaces preserve `retention = none` by default and expose explicit age/size retention when configured. | Voyage `VHUApus0L` planned |
12+
13+
## Users
14+
15+
| Persona | Description | Primary Need |
16+
|---------|-------------|--------------|
17+
| Transit Operator | Engineer running Transit as shared infrastructure for applications with bounded-retention requirements. | A first-class way to set per-stream age or size retention without accidentally changing every stream’s replay contract. |
18+
| Downstream Application Engineer | Engineer creating streams that should retain history indefinitely by default but may require bounded replay for selected workloads. | Stream creation and inspection surfaces that make retention policy explicit and predictable. |
19+
| Transit Maintainer | Engineer responsible for preserving shared-engine semantics and operator clarity. | A retention design that stays distinct from compaction and surfaces retained-frontier behavior explicitly. |
20+
21+
## Scope
22+
23+
### In Scope
24+
25+
- [SCOPE-01] Per-stream retention metadata with optional `max_age_days` and `max_bytes`, defaulting to no retention when unset.
26+
- [SCOPE-02] Stream creation surfaces that accept `--retention-max-age-days` and `--retention-max-bytes`.
27+
- [SCOPE-03] Shared-engine retention enforcement that removes oldest eligible rolled segments while preserving the active segment.
28+
- [SCOPE-04] Stream inspection surfaces that expose configured retention and the retained frontier via `retained_start_offset` or equivalent earliest-available status.
29+
- [SCOPE-05] Documentation and proof coverage that explain bounded replay and distinguish retention from compaction.
30+
31+
### Out of Scope
32+
33+
- [SCOPE-06] Key-aware log compaction, tombstones, or Kafka-style latest-value retention.
34+
- [SCOPE-07] Any global default retention window such as `30 days` applied implicitly to all streams.
35+
- [SCOPE-08] Partial trimming or in-place rewrite of the active segment.
36+
- [SCOPE-09] Selective subject erasure or GDPR workflow claims beyond coarse-grained retention controls.
37+
38+
## Requirements
39+
40+
### Functional Requirements
41+
42+
<!-- BEGIN FUNCTIONAL_REQUIREMENTS -->
43+
| ID | Requirement | Goals | Priority | Rationale |
44+
|----|-------------|-------|----------|-----------|
45+
| FR-01 | Add per-stream retention metadata that can represent `none`, optional maximum age in days, and optional maximum retained bytes, with no retention applied when the policy is absent. | GOAL-01 | must | Retention must be explicit and opt-in so Transit does not silently change its default replay semantics. |
46+
| FR-02 | Expose retention policy at stream creation time through CLI and supporting engine/protocol surfaces using `--retention-max-age-days` and `--retention-max-bytes`. | GOAL-01 | must | Operators need a practical way to configure retention when streams are created. |
47+
| FR-03 | Enforce retention in the shared engine by removing oldest eligible rolled segments under age and/or size limits while preserving append-only behavior for the active segment. | GOAL-01 | must | The retention model should bound stored history without inventing a compaction semantic. |
48+
| FR-04 | Surface configured retention in `transit streams list` and expose the retained frontier through stream status using `retained_start_offset` or an equivalent earliest-available field. | GOAL-01 | must | Once replay becomes bounded, operators need explicit visibility into policy and the earliest retained offset. |
49+
| FR-05 | Publish proof coverage and operator guidance that explain bounded replay, retained-frontier semantics, and the distinction between retention and compaction. | GOAL-01 | must | This feature changes operator expectations about replay availability and needs explicit guidance. |
50+
<!-- END FUNCTIONAL_REQUIREMENTS -->
51+
52+
### Non-Functional Requirements
53+
54+
<!-- BEGIN NON_FUNCTIONAL_REQUIREMENTS -->
55+
| ID | Requirement | Goals | Priority | Rationale |
56+
|----|-------------|-------|----------|-----------|
57+
| NFR-01 | The default retention posture must remain `none`; unconfigured streams must preserve today’s history-unbounded behavior. | GOAL-01 | must | A hidden default retention window would silently change Transit’s core replay contract. |
58+
| NFR-02 | Retention must remain shared-engine semantics that behave the same in embedded and hosted modes. | GOAL-01 | must | This repository does not allow server-only storage semantics to bypass the shared engine model. |
59+
| NFR-03 | Retention must stay semantically distinct from compaction: no key-aware collapse, no latest-value projection, and no in-place mutation of retained records. | GOAL-01 | must | Operators need a clear contract that bounded replay does not imply Kafka-style compaction. |
60+
<!-- END NON_FUNCTIONAL_REQUIREMENTS -->
61+
62+
## Verification Strategy
63+
64+
| Area | Method | Evidence |
65+
|------|--------|----------|
66+
| Retention creation surface | Targeted CLI and engine tests for retention metadata plus create-time flags | Story-level evidence under voyage `VHUApus0L` |
67+
| Retention enforcement | Targeted shared-engine tests for age/size trimming and retained frontier behavior | Story-level evidence under voyage `VHUApus0L` |
68+
| Operator visibility | CLI proof coverage and docs review for list/status retention fields and bounded replay guidance | Story-level evidence under voyage `VHUApus0L` |
69+
70+
## Assumptions
71+
72+
| Assumption | Impact if Wrong | Validation |
73+
|------------|-----------------|------------|
74+
| Retention can be delivered by trimming oldest rolled segments without introducing key-aware compaction semantics. | The epic could drift into a broader storage rewrite or compaction design. | Validate in SDD and story contracts that the active segment remains untouched and retained history stays append-only. |
75+
| Stream status surfaces are sufficient to communicate replay bounds if they expose retention policy and retained frontier explicitly. | Operators may still misread bounded replay as silent data loss. | Validate through proof/docs coverage and user-facing field naming. |
76+
77+
## Open Questions & Risks
78+
79+
| Question/Risk | Owner | Status |
80+
|---------------|-------|--------|
81+
| Should the public field name be `earliest_offset` or `retained_start_offset` on the status surface? | Epic owner | Open |
82+
| How should cursor validation behave once a cursor falls behind the retained frontier? | Epic owner | Open |
83+
| What deterministic enforcement point should own retention trimming: append, recovery, explicit maintenance step, or a combination? | Epic owner | Open |
84+
85+
## Success Criteria
86+
87+
<!-- BEGIN SUCCESS_CRITERIA -->
88+
- [ ] Streams default to no retention unless an explicit policy is configured.
89+
- [ ] `transit stream create` exposes `--retention-max-age-days` and `--retention-max-bytes`.
90+
- [ ] `transit streams list` exposes configured `retention_age` and `retention_bytes`.
91+
- [ ] Stream status exposes the earliest retained frontier via `retained_start_offset` or equivalent.
92+
- [ ] Shared-engine retention enforcement trims oldest eligible rolled segments under configured limits without changing the active append path into compaction.
93+
<!-- END SUCCESS_CRITERIA -->

‎.keel/epics/VHUAlZWZG/README.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
---
2+
# system-managed
3+
id: VHUAlZWZG
4+
created_at: 2026-04-21T20:10:17
5+
# authored
6+
title: Add Per-Stream Retention Policies And Retained Frontier Semantics
7+
index: 30
8+
mission: VHUAhwLlS
9+
---
10+
11+
# Add Per-Stream Retention Policies And Retained Frontier Semantics
12+
13+
> Transit needs explicit per-stream retention controls for age- and size-bounded history, plus stream status surfaces that show configured retention and the retained frontier, without changing the default append-only behavior.
14+
15+
## Documents
16+
17+
| Document | Description |
18+
|----------|-------------|
19+
| [PRD.md](PRD.md) | Product requirements and success criteria |
20+
| `PRESS_RELEASE.md` (optional) | Working-backwards artifact for large user-facing launches; usually skip for incremental/refactor/architecture-only work |
21+
22+
## Voyages
23+
24+
<!-- BEGIN GENERATED -->
25+
**Progress:** 0/1 voyages complete, 0/3 stories done
26+
| Voyage | Status | Stories |
27+
|--------|--------|---------|
28+
| [Deliver Per-Stream Retention Configuration And Visibility](voyages/VHUApus0L/) | in-progress | 0/3 |
29+
<!-- END GENERATED -->

‎.keel/epics/VHUAlZWZG/voyages/VHUApus0L/README.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
---
2+
# system-managed
3+
id: VHUApus0L
4+
status: in-progress
5+
epic: VHUAlZWZG
6+
created_at: 2026-04-21T20:10:34
7+
# authored
8+
title: Deliver Per-Stream Retention Configuration And Visibility
9+
index: 1
10+
updated_at: 2026-04-21T20:15:13
11+
started_at: 2026-04-21T20:15:22
12+
---
13+
14+
# Deliver Per-Stream Retention Configuration And Visibility
15+
16+
> Let operators configure per-stream retention with no default policy, enforce age/size-based retention in the shared engine, and surface retention policy plus retained start offset in list and status outputs.
17+
18+
## Documents
19+
20+
<!-- BEGIN DOCUMENTS -->
21+
| Document | Description |
22+
|----------|-------------|
23+
| [SRS.md](SRS.md) | Requirements and verification criteria |
24+
| [SDD.md](SDD.md) | Architecture and implementation details |
25+
<!-- END DOCUMENTS -->
26+
27+
## Stories
28+
29+
<!-- BEGIN GENERATED -->
30+
**Progress:** 0/3 stories complete
31+
32+
| Title | Type | Status |
33+
|-------|------|--------|
34+
| [Add Per-Stream Retention Metadata And Create-Time Surface](../../../../stories/VHUAuqNpn/README.md) | feat | in-progress |
35+
| [Enforce Retention And Surface Retained Frontier Status](../../../../stories/VHUAuquph/README.md) | feat | backlog |
36+
| [Publish Retention Proof Coverage And Operator Guidance](../../../../stories/VHUAurAqP/README.md) | feat | backlog |
37+
<!-- END GENERATED -->

‎.keel/epics/VHUAlZWZG/voyages/VHUApus0L/SDD.md‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
# Deliver Per-Stream Retention Configuration And Visibility - Software Design Description
2+
3+
> Let operators configure per-stream retention with no default policy, enforce age/size-based retention in the shared engine, and surface retention policy plus retained start offset in list and status outputs.
4+
5+
**SRS:** [SRS.md](SRS.md)
6+
7+
## Overview
8+
9+
This voyage adds explicit per-stream retention policy to the shared engine, threads that policy through stream creation and inspection surfaces, and exposes the retained frontier so operators understand when replay is bounded. The design keeps Transit’s current default semantics by making retention opt-in and enforcing it through whole-segment lifecycle management rather than compaction.
10+
11+
## Context & Boundaries
12+
13+
- In scope:
14+
- retention metadata for streams
15+
- create-time CLI/protocol surface for age- and size-based retention
16+
- shared-engine trimming of oldest rolled segments
17+
- list and status fields for configured retention and retained frontier
18+
- proof and docs explaining bounded replay
19+
- Out of scope:
20+
- Kafka-style compaction and tombstones
21+
- global default retention windows
22+
- in-place rewrite of active or retained records
23+
- selective privacy erasure semantics above coarse-grained retention
24+
25+
```
26+
┌────────────────────────────────────────────────────────────┐
27+
│ This Voyage │
28+
│ │
29+
│ CLI / Protocol -> Shared Engine Policy -> Status UX │
30+
│ create flags retention trimming list/status │
31+
│ │
32+
└────────────────────────────────────────────────────────────┘
33+
↑ ↑
34+
Stream authors Operators / proofs
35+
```
36+
37+
## Dependencies
38+
39+
| Dependency | Type | Purpose | Version/API |
40+
|------------|------|---------|-------------|
41+
| `transit-core` storage state and manifest model | Internal | Stores retention metadata, evaluates eligible rolled segments, and computes retained-frontier status. | Current workspace crate API |
42+
| `transit-cli` command surface | Internal | Accepts retention flags and renders retention/frontier status. | Current workspace crate API |
43+
| Hosted server protocol surfaces | Internal | Carries stream status and create-time retention options for downstream clients. | Current workspace crate API |
44+
45+
## Key Decisions
46+
47+
| Decision | Choice | Rationale |
48+
|----------|--------|-----------|
49+
| Default retention posture | `none` | Preserves existing Transit replay semantics unless a stream is explicitly opted into bounded history. |
50+
| Retention granularity | Whole rolled segments only | Keeps enforcement deterministic and append-only without introducing record-level compaction semantics. |
51+
| Active segment behavior | Never partially trimmed | Avoids rewriting hot append state and preserves the current append path contract. |
52+
| Frontier visibility | Expose configured retention plus `retained_start_offset` or equivalent earliest-available field | Once replay is bounded, operators need explicit visibility into where retained history begins. |
53+
54+
## Architecture
55+
56+
The design threads one policy through three layers:
57+
58+
- Stream creation layer:
59+
- accepts optional retention flags
60+
- stores policy alongside stream metadata
61+
- Shared engine lifecycle layer:
62+
- evaluates retention eligibility against rolled segments only
63+
- trims oldest eligible segments under age and/or size limits
64+
- recomputes earliest retained offset after enforcement
65+
- Operator visibility layer:
66+
- includes retention policy in `streams list`
67+
- includes retained frontier in stream status
68+
- documents bounded replay and cursor fallout
69+
70+
## Components
71+
72+
- Retention policy model:
73+
- purpose: represent `none`, `max_age_days`, and `max_bytes`
74+
- interface: stream metadata and create surfaces
75+
- behavior: absent policy means no trimming
76+
- Retention evaluator:
77+
- purpose: determine which oldest rolled segments are eligible to drop
78+
- interface: shared engine storage lifecycle
79+
- behavior: applies age and size limits without touching the active segment
80+
- Frontier reporter:
81+
- purpose: compute earliest retained replay position
82+
- interface: status/list surfaces
83+
- behavior: surfaces retained-frontier state explicitly so bounded replay is visible
84+
- Proof/docs surface:
85+
- purpose: explain semantics and prove expected behavior
86+
- interface: CLI proof path and public docs
87+
- behavior: distinguishes retention from compaction and describes retained-frontier consequences
88+
89+
## Interfaces
90+
91+
- Stream creation:
92+
- add optional `--retention-max-age-days <days>`
93+
- add optional `--retention-max-bytes <bytes>`
94+
- Streams list:
95+
- add `retention_age`
96+
- add `retention_bytes`
97+
- Stream status:
98+
- add `retained_start_offset` or equivalent earliest-retained field
99+
- include configured retention policy so the frontier is interpretable
100+
- Internal engine/state:
101+
- persist retention policy with stream metadata
102+
- expose retained-frontier status alongside `next_offset`, active-head, and manifest data
103+
104+
## Data Flow
105+
106+
- Operator creates a stream with no retention flags:
107+
- stream metadata persists `retention = none`
108+
- list/status surfaces report no retention and unbounded replay semantics remain
109+
- Operator creates a stream with age and/or size retention:
110+
- stream metadata persists the configured policy
111+
- append/maintenance lifecycle evaluates oldest rolled segments against the policy
112+
- eligible oldest rolled segments are removed
113+
- retained frontier advances to the earliest remaining offset
114+
- list/status surfaces report configured retention and new retained frontier
115+
- Operator inspects the stream:
116+
- `streams list` shows retention policy
117+
- status surface shows earliest retained offset so replay bounds are explicit
118+
119+
## Error Handling
120+
121+
| Error Condition | Detection | Response | Recovery |
122+
|-----------------|-----------|----------|----------|
123+
| Invalid retention input such as zero/negative age or bytes | CLI/parser validation or config validation | Reject stream creation/update request with explicit validation error | Operator corrects the supplied policy |
124+
| Size policy smaller than current active segment footprint | Retention evaluator sees no eligible rolled segments to drop | Preserve active segment and report retained frontier based on remaining history | Retention catches up after future segment rolls create eligible old segments |
125+
| Cursor or replay request falls behind retained frontier | Status/frontier comparison during read/cursor validation | Return explicit out-of-retention error or earliest-retained guidance | Operator resets the cursor/read start to the retained frontier |
126+
| Operator misreads retention as compaction | Proof/docs review and status field naming | Publish explicit documentation and proof coverage | Maintain vocabulary separation between retention and compaction |

0 commit comments

Comments
 (0)