Skip to content

Commit b26b12b

Browse files
keithahkeithah
authored
Phase 17/18: reliability + observability pass (#33)
* docs(planning): re-scope phases 17-18 to lean reliability Rename Phase 17/18 around write-mode reliability and observability, add 17-02/18-02 plans and summaries, and align state/project focus to single-replica pragmatic hardening. * fix(write): reduce secret-scan false positives and improve evidence logs Scan regex secrets against staged additions per file to allow safe secret removals, improve no-change refusal guidance, and normalize evidence/runbook fields for easier deliveryId-based operations.
1 parent ddbfba3 commit b26b12b

16 files changed

Lines changed: 463 additions & 43 deletions

File tree

‎.planning/PROJECT.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,8 @@ When a PR is opened or `@kodiai` is mentioned, the bot responds with accurate, a
3131
### Active
3232

3333
- [ ] Code modification via @mention (branch creation, commit, push) with guardrails
34-
- [ ] Durable idempotency/locking (reduce race risk beyond marker checks)
35-
- [ ] Expand operator evidence capture (GitHub delivery metadata access + richer log correlation tooling)
34+
- [ ] Write-mode reliability polish (clearer failures, safer retries, reduced false positives)
35+
- [ ] Expand operator evidence capture (delivery correlation, queue/failure metrics, grep-ready runbooks)
3636

3737
### Out of Scope
3838

‎.planning/STATE.md‎

Lines changed: 13 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -5,28 +5,28 @@
55
See: .planning/PROJECT.md (updated 2026-02-09)
66

77
**Core value:** When a PR is opened or @kodiai is mentioned, the bot responds with accurate, actionable code feedback without requiring any workflow setup in the target repo.
8-
**Current focus:** Planning next milestone (fresh requirements + roadmap).
8+
**Current focus:** v0.2 execution: write-mode reliability + observability (no distributed infra).
99

1010
## Current Position
1111

12-
**Current Phase:** 21
13-
**Current Phase Name:** polish
12+
**Current Phase:** 18
13+
**Current Phase Name:** observability-verification
1414
**Total Phases:** 21
1515
**Current Plan:** 2
16-
**Total Plans in Phase:** 4
17-
**Status:** In progress
18-
**Progress:** [####### ] 75%
16+
**Total Plans in Phase:** 2
17+
**Status:** Complete
18+
**Progress:** [##########] 100%
1919

20-
**Last Activity:** 2026-02-10
21-
**Last Activity Description:** Phase 21: wrote xbmc/xbmc write-flow smoke doc; improved write guardrail refusal details; manual smoke run pending
20+
**Last Activity:** 2026-02-11
21+
**Last Activity Description:** Completed 18-02 observability + verification pass (log consistency + runbook/query templates)
2222
**Paused At:** None
2323

2424
Milestone: v0.2 (planning)
2525

2626
## Performance Metrics
2727

2828
**Velocity:**
29-
- Total plans completed: 36
29+
- Total plans completed: 38
3030
- Average duration: 3min
3131
- Total execution time: 244min
3232

@@ -61,8 +61,10 @@ Milestone: v0.2 (planning)
6161
| Phase 14-write-mode-foundations P01 | 15 min | 3 tasks | 4 files |
6262
| Phase 15-write-pipeline P01 | 20 min | 3 tasks | 6 files |
6363
| Phase 16-write-guardrails P01 | 25 min | 3 tasks | 6 files |
64-
| Phase 17-durability-locking P01 | 20 min | 3 tasks | 2 files |
65-
| Phase 18-ops-evidence P01 | 15 min | 3 tasks | 4 files |
64+
| Phase 17-write-mode-reliability P01 | 20 min | 3 tasks | 2 files |
65+
| Phase 17-write-mode-reliability P02 | 15 min | 3 tasks | 4 files |
66+
| Phase 18-observability-verification P01 | 15 min | 3 tasks | 4 files |
67+
| Phase 18-observability-verification P02 | 12 min | 3 tasks | 4 files |
6668
| Phase 19-write-confirmation P01 | 15 min | 3 tasks | 4 files |
6769
| Phase 20-next-improvements P01 | 30 min | 5 tasks | 8 files |
6870

.planning/phases/17-durability-locking/17-01-PLAN.md renamed to .planning/phases/17-write-mode-reliability/17-01-PLAN.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
phase: 17-durability-locking
2+
phase: 17-write-mode-reliability
33
plan: 01
44
type: execute
55
wave: 1
@@ -24,7 +24,7 @@ must_haves:
2424
---
2525

2626
<objective>
27-
Strengthen durability for write-mode by adding idempotency and lightweight locking so redeliveries and retries do not create duplicate branches/PRs.
27+
Strengthen write-mode reliability by adding idempotency and lightweight in-process locking so redeliveries and retries do not create duplicate branches/PRs.
2828
</objective>
2929

3030
<tasks>

.planning/phases/17-durability-locking/17-01-SUMMARY.md renamed to .planning/phases/17-write-mode-reliability/17-01-SUMMARY.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
---
2-
phase: 17-durability-locking
2+
phase: 17-write-mode-reliability
33
plan: 01
44
subsystem: write-mode
5-
tags: [write-mode, idempotency, locking, durability]
5+
tags: [write-mode, idempotency, locking, reliability]
66

77
# Dependency graph
88
requires:
@@ -37,7 +37,7 @@ duration: 20 min
3737
completed: 2026-02-10
3838
---
3939

40-
# Phase 17 Plan 01: Durability + Locking Summary
40+
# Phase 17 Plan 01: Write-Mode Reliability Summary
4141

4242
**Made write-mode idempotent and retry-safe by keying write outputs to the triggering comment and reusing an existing PR when the same request is delivered again.**
4343

‎.planning/phases/17-write-mode-reliability/17-02-PLAN.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
---
2+
phase: 17-write-mode-reliability
3+
plan: 02
4+
type: execute
5+
wave: 2
6+
depends_on:
7+
- 17-write-mode-reliability/17-01-SUMMARY.md
8+
- 16-write-guardrails/16-01-SUMMARY.md
9+
files_modified:
10+
- src/handlers/mention.ts
11+
- src/handlers/mention.test.ts
12+
- src/jobs/workspace.ts
13+
- docs/runbooks/mentions.md
14+
autonomous: true
15+
---
16+
17+
<objective>
18+
Improve write-mode reliability for private single-replica usage by focusing on user-visible clarity and safer behavior under expected failures, without adding distributed infrastructure.
19+
</objective>
20+
21+
<tasks>
22+
23+
<task type="auto">
24+
<name>Task 1: failure UX tightening for write-mode</name>
25+
<files>src/handlers/mention.ts src/handlers/mention.test.ts</files>
26+
<action>
27+
Ensure write failures produce short, actionable refusal/error responses that clearly include:
28+
- stable reason code
29+
- triggering path/rule when available
30+
- smallest safe next action (or explicit no-safe-bypass)
31+
Keep responses concise and aligned with current decision-only style.
32+
</action>
33+
<verify>
34+
- Unit tests cover deny path, allowlist mismatch, secret detection, and generic write failures.
35+
</verify>
36+
</task>
37+
38+
<task type="auto">
39+
<name>Task 2: false-positive tuning pass (no policy widening)</name>
40+
<files>src/jobs/workspace.ts src/handlers/mention.test.ts</files>
41+
<action>
42+
Review guardrail checks for obvious false-positive edge cases and tighten detection behavior without reducing safety defaults.
43+
Add tests for any adjusted behavior.
44+
</action>
45+
<verify>
46+
- bun test
47+
</verify>
48+
</task>
49+
50+
<task type="auto">
51+
<name>Task 3: operator quick-reference updates</name>
52+
<files>docs/runbooks/mentions.md</files>
53+
<action>
54+
Add a short troubleshooting section mapping common write failure reason codes to immediate operator actions.
55+
</action>
56+
</task>
57+
58+
</tasks>
59+
60+
<verification>
61+
- bun test
62+
- bunx tsc --noEmit
63+
</verification>
64+
65+
<output>
66+
Create `.planning/phases/17-write-mode-reliability/17-02-SUMMARY.md`.
67+
</output>

‎.planning/phases/17-write-mode-reliability/17-02-SUMMARY.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
---
2+
phase: 17-write-mode-reliability
3+
plan: 02
4+
subsystem: write-mode
5+
tags: [reliability, guardrails, ux]
6+
7+
# Dependency graph
8+
requires:
9+
- phase: 17-write-mode-reliability
10+
provides: idempotent write-mode baseline
11+
- phase: 16-write-guardrails
12+
provides: policy enforcement baseline
13+
provides:
14+
- Secret regex scanning reduced false positives by evaluating staged additions only
15+
- Write-policy refusal UX includes explicit no-changes next action
16+
- Runbook reason-code quick map for faster operator triage
17+
affects: [mentions, write-mode, runbooks]
18+
19+
# Metrics
20+
duration: 15 min
21+
completed: 2026-02-11
22+
---
23+
24+
# Phase 17 Plan 02: Write-Mode Reliability Summary
25+
26+
Improved write-mode reliability with safer secret-scan behavior and clearer operator/user guidance.
27+
28+
## What changed
29+
30+
- Secret regex scanning now evaluates staged additions per file (not whole patch), so removing old secret-like lines no longer triggers false-positive refusals.
31+
- Added write-policy `no-changes` refusal guidance with an explicit next action.
32+
- Added runbook quick map from refusal reason codes to immediate operator actions.
33+
- Added regression test proving secret-like content removal is allowed.
34+
35+
Files changed:
36+
37+
- `src/jobs/workspace.ts`
38+
- `src/handlers/mention.ts`
39+
- `src/handlers/mention.test.ts`
40+
- `docs/runbooks/mentions.md`
41+
42+
## Verification
43+
44+
- `bun test`
45+
- `bunx tsc --noEmit`

.planning/phases/18-ops-evidence/18-01-PLAN.md renamed to .planning/phases/18-observability-verification/18-01-PLAN.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
---
2-
phase: 18-ops-evidence
2+
phase: 18-observability-verification
33
plan: 01
44
type: execute
55
wave: 1
66
depends_on:
7-
- 17-durability-locking/17-01-SUMMARY.md
7+
- 17-write-mode-reliability/17-01-SUMMARY.md
88
files_modified:
99
- src/handlers/mention.ts
1010
- src/handlers/review.ts
@@ -20,7 +20,7 @@ must_haves:
2020
---
2121

2222
<objective>
23-
Improve operator evidence and debugging UX by standardizing a single evidence bundle log line for each execution and publish.
23+
Improve observability and verification UX by standardizing a single evidence bundle log line for each execution and publish.
2424
</objective>
2525

2626
<tasks>

.planning/phases/18-ops-evidence/18-01-SUMMARY.md renamed to .planning/phases/18-observability-verification/18-01-SUMMARY.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
phase: 18-ops-evidence
2+
phase: 18-observability-verification
33
plan: 01
44
subsystem: ops
55
tags: [ops, evidence, logging, runbooks]
@@ -19,7 +19,7 @@ duration: 15 min
1919
completed: 2026-02-10
2020
---
2121

22-
# Phase 18 Plan 01: Ops Evidence Bundle Summary
22+
# Phase 18 Plan 01: Observability + Verification Summary
2323

2424
Added a single structured "Evidence bundle" log line for:
2525

‎.planning/phases/18-observability-verification/18-02-PLAN.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
---
2+
phase: 18-observability-verification
3+
plan: 02
4+
type: execute
5+
wave: 2
6+
depends_on:
7+
- 18-observability-verification/18-01-SUMMARY.md
8+
- 17-write-mode-reliability/17-02-SUMMARY.md
9+
files_modified:
10+
- src/handlers/mention.ts
11+
- src/handlers/review.ts
12+
- docs/runbooks/mentions.md
13+
- docs/smoke/xbmc-kodiai-write-flow.md
14+
autonomous: true
15+
---
16+
17+
<objective>
18+
Strengthen observability and release verification for v0.2 by improving grepability and adding lightweight production smoke evidence capture, without introducing external state systems.
19+
</objective>
20+
21+
<tasks>
22+
23+
<task type="auto">
24+
<name>Task 1: structured log consistency pass</name>
25+
<files>src/handlers/mention.ts src/handlers/review.ts</files>
26+
<action>
27+
Ensure evidence and failure logs use consistent key names for delivery correlation and PR context.
28+
Keep existing evidence bundle format stable where already used.
29+
</action>
30+
<verify>
31+
- Unit tests continue passing
32+
</verify>
33+
</task>
34+
35+
<task type="auto">
36+
<name>Task 2: runbook query snippets for top operator flows</name>
37+
<files>docs/runbooks/mentions.md</files>
38+
<action>
39+
Add concise query snippets for:
40+
- finding write-mode evidence by deliveryId
41+
- finding refusal reason patterns
42+
- finding recheck/rereview request outcomes
43+
</action>
44+
</task>
45+
46+
<task type="manual">
47+
<name>Task 3: post-deploy smoke evidence capture template</name>
48+
<files>docs/smoke/xbmc-kodiai-write-flow.md</files>
49+
<action>
50+
Add a tiny template section to record smoke run evidence (PR links + deliveryIds + outcomes) so each release has a repeatable verification artifact.
51+
</action>
52+
</task>
53+
54+
</tasks>
55+
56+
<verification>
57+
- bun test
58+
- bunx tsc --noEmit
59+
</verification>
60+
61+
<output>
62+
Create `.planning/phases/18-observability-verification/18-02-SUMMARY.md`.
63+
</output>

‎.planning/phases/18-observability-verification/18-02-SUMMARY.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
phase: 18-observability-verification
3+
plan: 02
4+
subsystem: ops
5+
tags: [observability, verification, runbooks]
6+
7+
# Dependency graph
8+
requires:
9+
- phase: 18-observability-verification
10+
provides: evidence bundle baseline
11+
- phase: 17-write-mode-reliability
12+
provides: current write-mode refusal behavior
13+
provides:
14+
- Evidence logs include consistent owner/repoName/repo context fields
15+
- Mentions runbook includes query snippets for refusals and rereview outcomes
16+
- Smoke doc has release evidence capture template
17+
affects: [mentions, review, ops]
18+
19+
# Metrics
20+
duration: 12 min
21+
completed: 2026-02-11
22+
---
23+
24+
# Phase 18 Plan 02: Observability + Verification Summary
25+
26+
Completed the observability/verification pass with consistent log context and reusable operator templates.
27+
28+
## What changed
29+
30+
- Added consistent repo context fields (`owner`, `repoName`, existing `repo`) to write/review evidence bundle logs.
31+
- Expanded mentions runbook with grep-ready queries for refusal reasons and rereview outcomes.
32+
- Added release evidence capture template to the xbmc/kodiai smoke doc.
33+
34+
Files changed:
35+
36+
- `src/handlers/mention.ts`
37+
- `src/handlers/review.ts`
38+
- `docs/runbooks/mentions.md`
39+
- `docs/smoke/xbmc-kodiai-write-flow.md`
40+
41+
## Verification
42+
43+
- `bun test`
44+
- `bunx tsc --noEmit`

0 commit comments

Comments
 (0)