Phase 17/18: reliability + observability pass

.planning/PROJECT.md

@@ -31,8 +31,8 @@ When a PR is opened or `@kodiai` is mentioned, the bot responds with accurate, a
 ### Active
 
 - [ ] Code modification via @mention (branch creation, commit, push) with guardrails
-- [ ] Durable idempotency/locking (reduce race risk beyond marker checks)
-- [ ] Expand operator evidence capture (GitHub delivery metadata access + richer log correlation tooling)
+- [ ] Write-mode reliability polish (clearer failures, safer retries, reduced false positives)
+- [ ] Expand operator evidence capture (delivery correlation, queue/failure metrics, grep-ready runbooks)
 
 ### Out of Scope
 

.planning/STATE.md

@@ -5,28 +5,28 @@
 See: .planning/PROJECT.md (updated 2026-02-09)
 
 **Core value:** When a PR is opened or @kodiai is mentioned, the bot responds with accurate, actionable code feedback without requiring any workflow setup in the target repo.
-**Current focus:** Planning next milestone (fresh requirements + roadmap).
+**Current focus:** v0.2 execution: write-mode reliability + observability (no distributed infra).
 
 ## Current Position
 
-**Current Phase:** 21
-**Current Phase Name:** polish
+**Current Phase:** 18
+**Current Phase Name:** observability-verification
 **Total Phases:** 21
 **Current Plan:** 2
-**Total Plans in Phase:** 4
-**Status:** In progress
-**Progress:** [#######   ] 75%
+**Total Plans in Phase:** 2
+**Status:** Complete
+**Progress:** [##########] 100%
 
-**Last Activity:** 2026-02-10
-**Last Activity Description:** Phase 21: wrote xbmc/xbmc write-flow smoke doc; improved write guardrail refusal details; manual smoke run pending
+**Last Activity:** 2026-02-11
+**Last Activity Description:** Completed 18-02 observability + verification pass (log consistency + runbook/query templates)
 **Paused At:** None
 
 Milestone: v0.2 (planning)
 
 ## Performance Metrics
 
 **Velocity:**
-- Total plans completed: 36
+- Total plans completed: 38
 - Average duration: 3min
 - Total execution time: 244min
 
@@ -61,8 +61,10 @@ Milestone: v0.2 (planning)
 | Phase 14-write-mode-foundations P01 | 15 min | 3 tasks | 4 files |
 | Phase 15-write-pipeline P01 | 20 min | 3 tasks | 6 files |
 | Phase 16-write-guardrails P01 | 25 min | 3 tasks | 6 files |
-| Phase 17-durability-locking P01 | 20 min | 3 tasks | 2 files |
-| Phase 18-ops-evidence P01 | 15 min | 3 tasks | 4 files |
+| Phase 17-write-mode-reliability P01 | 20 min | 3 tasks | 2 files |
+| Phase 17-write-mode-reliability P02 | 15 min | 3 tasks | 4 files |
+| Phase 18-observability-verification P01 | 15 min | 3 tasks | 4 files |
+| Phase 18-observability-verification P02 | 12 min | 3 tasks | 4 files |
 | Phase 19-write-confirmation P01 | 15 min | 3 tasks | 4 files |
 | Phase 20-next-improvements P01 | 30 min | 5 tasks | 8 files |
 

.planning/phases/17-durability-locking/17-01-PLAN.md → .planning/phases/17-write-mode-reliability/17-01-PLAN.md

@@ -1,5 +1,5 @@
 ---
-phase: 17-durability-locking
+phase: 17-write-mode-reliability
 plan: 01
 type: execute
 wave: 1
@@ -24,7 +24,7 @@ must_haves:
 ---
 
 <objective>
-Strengthen durability for write-mode by adding idempotency and lightweight locking so redeliveries and retries do not create duplicate branches/PRs.
+Strengthen write-mode reliability by adding idempotency and lightweight in-process locking so redeliveries and retries do not create duplicate branches/PRs.
 </objective>
 
 <tasks>

.planning/phases/17-durability-locking/17-01-SUMMARY.md → .planning/phases/17-write-mode-reliability/17-01-SUMMARY.md

@@ -1,8 +1,8 @@
 ---
-phase: 17-durability-locking
+phase: 17-write-mode-reliability
 plan: 01
 subsystem: write-mode
-tags: [write-mode, idempotency, locking, durability]
+tags: [write-mode, idempotency, locking, reliability]
 
 # Dependency graph
 requires:
@@ -37,7 +37,7 @@ duration: 20 min
 completed: 2026-02-10
 ---
 
-# Phase 17 Plan 01: Durability + Locking Summary
+# Phase 17 Plan 01: Write-Mode Reliability Summary
 
 **Made write-mode idempotent and retry-safe by keying write outputs to the triggering comment and reusing an existing PR when the same request is delivered again.**
 

.planning/phases/17-write-mode-reliability/17-02-PLAN.md

@@ -0,0 +1,67 @@
+---
+phase: 17-write-mode-reliability
+plan: 02
+type: execute
+wave: 2
+depends_on:
+  - 17-write-mode-reliability/17-01-SUMMARY.md
+  - 16-write-guardrails/16-01-SUMMARY.md
+files_modified:
+  - src/handlers/mention.ts
+  - src/handlers/mention.test.ts
+  - src/jobs/workspace.ts
+  - docs/runbooks/mentions.md
+autonomous: true
+---
+
+<objective>
+Improve write-mode reliability for private single-replica usage by focusing on user-visible clarity and safer behavior under expected failures, without adding distributed infrastructure.
+</objective>
+
+<tasks>
+
+<task type="auto">
+  <name>Task 1: failure UX tightening for write-mode</name>
+  <files>src/handlers/mention.ts src/handlers/mention.test.ts</files>
+  <action>
+  Ensure write failures produce short, actionable refusal/error responses that clearly include:
+  - stable reason code
+  - triggering path/rule when available
+  - smallest safe next action (or explicit no-safe-bypass)
+  Keep responses concise and aligned with current decision-only style.
+  </action>
+  <verify>
+  - Unit tests cover deny path, allowlist mismatch, secret detection, and generic write failures.
+  </verify>
+</task>
+
+<task type="auto">
+  <name>Task 2: false-positive tuning pass (no policy widening)</name>
+  <files>src/jobs/workspace.ts src/handlers/mention.test.ts</files>
+  <action>
+  Review guardrail checks for obvious false-positive edge cases and tighten detection behavior without reducing safety defaults.
+  Add tests for any adjusted behavior.
+  </action>
+  <verify>
+  - bun test
+  </verify>
+</task>
+
+<task type="auto">
+  <name>Task 3: operator quick-reference updates</name>
+  <files>docs/runbooks/mentions.md</files>
+  <action>
+  Add a short troubleshooting section mapping common write failure reason codes to immediate operator actions.
+  </action>
+</task>
+
+</tasks>
+
+<verification>
+- bun test
+- bunx tsc --noEmit
+</verification>
+
+<output>
+Create `.planning/phases/17-write-mode-reliability/17-02-SUMMARY.md`.
+</output>

.planning/phases/17-write-mode-reliability/17-02-SUMMARY.md

@@ -0,0 +1,45 @@
+---
+phase: 17-write-mode-reliability
+plan: 02
+subsystem: write-mode
+tags: [reliability, guardrails, ux]
+
+# Dependency graph
+requires:
+  - phase: 17-write-mode-reliability
+    provides: idempotent write-mode baseline
+  - phase: 16-write-guardrails
+    provides: policy enforcement baseline
+provides:
+  - Secret regex scanning reduced false positives by evaluating staged additions only
+  - Write-policy refusal UX includes explicit no-changes next action
+  - Runbook reason-code quick map for faster operator triage
+affects: [mentions, write-mode, runbooks]
+
+# Metrics
+duration: 15 min
+completed: 2026-02-11
+---
+
+# Phase 17 Plan 02: Write-Mode Reliability Summary
+
+Improved write-mode reliability with safer secret-scan behavior and clearer operator/user guidance.
+
+## What changed
+
+- Secret regex scanning now evaluates staged additions per file (not whole patch), so removing old secret-like lines no longer triggers false-positive refusals.
+- Added write-policy `no-changes` refusal guidance with an explicit next action.
+- Added runbook quick map from refusal reason codes to immediate operator actions.
+- Added regression test proving secret-like content removal is allowed.
+
+Files changed:
+
+- `src/jobs/workspace.ts`
+- `src/handlers/mention.ts`
+- `src/handlers/mention.test.ts`
+- `docs/runbooks/mentions.md`
+
+## Verification
+
+- `bun test`
+- `bunx tsc --noEmit`

.planning/phases/18-ops-evidence/18-01-PLAN.md → .planning/phases/18-observability-verification/18-01-PLAN.md

@@ -1,10 +1,10 @@
 ---
-phase: 18-ops-evidence
+phase: 18-observability-verification
 plan: 01
 type: execute
 wave: 1
 depends_on:
-  - 17-durability-locking/17-01-SUMMARY.md
+  - 17-write-mode-reliability/17-01-SUMMARY.md
 files_modified:
   - src/handlers/mention.ts
   - src/handlers/review.ts
@@ -20,7 +20,7 @@ must_haves:
 ---
 
 <objective>
-Improve operator evidence and debugging UX by standardizing a single evidence bundle log line for each execution and publish.
+Improve observability and verification UX by standardizing a single evidence bundle log line for each execution and publish.
 </objective>
 
 <tasks>

.planning/phases/18-ops-evidence/18-01-SUMMARY.md → .planning/phases/18-observability-verification/18-01-SUMMARY.md

@@ -1,5 +1,5 @@
 ---
-phase: 18-ops-evidence
+phase: 18-observability-verification
 plan: 01
 subsystem: ops
 tags: [ops, evidence, logging, runbooks]
@@ -19,7 +19,7 @@ duration: 15 min
 completed: 2026-02-10
 ---
 
-# Phase 18 Plan 01: Ops Evidence Bundle Summary
+# Phase 18 Plan 01: Observability + Verification Summary
 
 Added a single structured "Evidence bundle" log line for:
 

.planning/phases/18-observability-verification/18-02-PLAN.md

@@ -0,0 +1,63 @@
+---
+phase: 18-observability-verification
+plan: 02
+type: execute
+wave: 2
+depends_on:
+  - 18-observability-verification/18-01-SUMMARY.md
+  - 17-write-mode-reliability/17-02-SUMMARY.md
+files_modified:
+  - src/handlers/mention.ts
+  - src/handlers/review.ts
+  - docs/runbooks/mentions.md
+  - docs/smoke/xbmc-kodiai-write-flow.md
+autonomous: true
+---
+
+<objective>
+Strengthen observability and release verification for v0.2 by improving grepability and adding lightweight production smoke evidence capture, without introducing external state systems.
+</objective>
+
+<tasks>
+
+<task type="auto">
+  <name>Task 1: structured log consistency pass</name>
+  <files>src/handlers/mention.ts src/handlers/review.ts</files>
+  <action>
+  Ensure evidence and failure logs use consistent key names for delivery correlation and PR context.
+  Keep existing evidence bundle format stable where already used.
+  </action>
+  <verify>
+  - Unit tests continue passing
+  </verify>
+</task>
+
+<task type="auto">
+  <name>Task 2: runbook query snippets for top operator flows</name>
+  <files>docs/runbooks/mentions.md</files>
+  <action>
+  Add concise query snippets for:
+  - finding write-mode evidence by deliveryId
+  - finding refusal reason patterns
+  - finding recheck/rereview request outcomes
+  </action>
+</task>
+
+<task type="manual">
+  <name>Task 3: post-deploy smoke evidence capture template</name>
+  <files>docs/smoke/xbmc-kodiai-write-flow.md</files>
+  <action>
+  Add a tiny template section to record smoke run evidence (PR links + deliveryIds + outcomes) so each release has a repeatable verification artifact.
+  </action>
+</task>
+
+</tasks>
+
+<verification>
+- bun test
+- bunx tsc --noEmit
+</verification>
+
+<output>
+Create `.planning/phases/18-observability-verification/18-02-SUMMARY.md`.
+</output>

.planning/phases/18-observability-verification/18-02-SUMMARY.md

@@ -0,0 +1,44 @@
+---
+phase: 18-observability-verification
+plan: 02
+subsystem: ops
+tags: [observability, verification, runbooks]
+
+# Dependency graph
+requires:
+  - phase: 18-observability-verification
+    provides: evidence bundle baseline
+  - phase: 17-write-mode-reliability
+    provides: current write-mode refusal behavior
+provides:
+  - Evidence logs include consistent owner/repoName/repo context fields
+  - Mentions runbook includes query snippets for refusals and rereview outcomes
+  - Smoke doc has release evidence capture template
+affects: [mentions, review, ops]
+
+# Metrics
+duration: 12 min
+completed: 2026-02-11
+---
+
+# Phase 18 Plan 02: Observability + Verification Summary
+
+Completed the observability/verification pass with consistent log context and reusable operator templates.
+
+## What changed
+
+- Added consistent repo context fields (`owner`, `repoName`, existing `repo`) to write/review evidence bundle logs.
+- Expanded mentions runbook with grep-ready queries for refusal reasons and rereview outcomes.
+- Added release evidence capture template to the xbmc/kodiai smoke doc.
+
+Files changed:
+
+- `src/handlers/mention.ts`
+- `src/handlers/review.ts`
+- `docs/runbooks/mentions.md`
+- `docs/smoke/xbmc-kodiai-write-flow.md`
+
+## Verification
+
+- `bun test`
+- `bunx tsc --noEmit`