amikos-tech
diff --git a/‎.planning/REQUIREMENTS.md‎
Lines changed: 5 additions & 5 deletions b/‎.planning/REQUIREMENTS.md‎
Lines changed: 5 additions & 5 deletions
diff --git a/‎.planning/ROADMAP.md‎
Lines changed: 7 additions & 4 deletions b/‎.planning/ROADMAP.md‎
Lines changed: 7 additions & 4 deletions
diff --git a/‎.planning/STATE.md‎
Lines changed: 27 additions & 14 deletions b/‎.planning/STATE.md‎
Lines changed: 27 additions & 14 deletions
diff --git a/‎.planning/phases/25-error-body-truncation/25-01-PLAN.md‎
Lines changed: 201 additions & 0 deletions b/‎.planning/phases/25-error-body-truncation/25-01-PLAN.md‎
Lines changed: 201 additions & 0 deletions
@@ -20,8 +20,8 @@
 
 ### Error Handling
 
-- [ ] **ERR-01**: Shared `SanitizeErrorBody` utility truncates HTTP error bodies to a safe display length with `[truncated]` suffix
-- [ ] **ERR-02**: All embedding providers use `SanitizeErrorBody` for error message construction instead of raw `string(respData)`
+- [x] **ERR-01**: Shared `SanitizeErrorBody` utility truncates HTTP error bodies to a safe display length with `[truncated]` suffix
+- [x] **ERR-02**: All embedding providers use `SanitizeErrorBody` for error message construction instead of raw `string(respData)`
 
 ### Provider Enhancement
 
@@ -67,8 +67,8 @@
 | EFL-01 | Phase 23 | Complete |
 | EFL-02 | Phase 24 | Pending |
 | EFL-03 | Phase 24 | Pending |
-| ERR-01 | Phase 25 | Pending |
-| ERR-02 | Phase 25 | Pending |
+| ERR-01 | Phase 25 | Complete |
+| ERR-02 | Phase 25 | Complete |
 | TLA-01 | Phase 26 | Pending |
 | TLA-02 | Phase 26 | Pending |
 | TLA-03 | Phase 26 | Pending |
@@ -84,4 +84,4 @@
 
 ---
 *Requirements defined: 2026-04-08*
-*Last updated: 2026-04-11 after Phase 23 completion*
+*Last updated: 2026-04-13 after Phase 25 completion*
@@ -26,7 +26,7 @@ See: [v0.4.1 Archived Roadmap](milestones/v0.4.1-ROADMAP.md)
 - [x] **Phase 22: WithGroupBy Validation** - WithGroupBy(nil) returns an error instead of silently skipping grouping (completed 2026-04-10)
 - [x] **Phase 23: ORT EF Leak Fix** - Default ORT EF is properly closed when CreateCollection finds an existing collection (completed 2026-04-11)
 - [x] **Phase 24: GetOrCreateCollection EF Safety** - GetOrCreateCollection does not pass closed EFs to CreateCollection fallback (completed 2026-04-12)
-- [ ] **Phase 25: Error Body Truncation** - Embedding provider error messages truncate raw HTTP bodies to safe display lengths
+- [x] **Phase 25: Error Body Truncation** - Embedding provider error messages truncate raw HTTP bodies to safe display lengths (completed 2026-04-13)
 - [ ] **Phase 26: Twelve Labs Async Embedding** - Twelve Labs provider handles async task responses for long-running media
 - [ ] **Phase 27: Download Stack Consolidation** - default_ef download code uses shared downloadutil instead of its own HTTP implementation
 - [ ] **Phase 28: Morph Test Fix** - Morph EF integration test handles upstream 404 gracefully
@@ -109,10 +109,13 @@ Plans:
   1. A shared SanitizeErrorBody utility exists that truncates bodies exceeding a safe display length and appends a `[truncated]` suffix
   2. All embedding providers use SanitizeErrorBody when constructing error messages from HTTP responses
   3. Error messages from providers with large error bodies are readable (not multi-KB dumps)
-**Plans**: TBD
+**Plans**: 4 plans
 
 Plans:
-- [ ] 25-01: TBD
+- [x] 25-01-PLAN.md — Create the shared sanitizer contract with panic recovery and normalize Perplexity/OpenRouter
+- [x] 25-02-PLAN.md — Migrate the representative raw-body providers and add OpenAI/Baseten regressions
+- [x] 25-03-PLAN.md — Finish batch A with Cloudflare/Cohere/HF/Jina and preserve Cloudflare mixed formatting
+- [x] 25-04-PLAN.md — Finish batch B, sanitize Twelve Labs structured errors, and run the full embedding sweep/lint
 
 ### Phase 26: Twelve Labs Async Embedding
 **Goal**: Twelve Labs provider handles async task responses for long-running audio and video embeddings
@@ -196,7 +199,7 @@ Phase 24 depends on Phase 23. Phase 26 depends on Phase 25. Phase 29 depends on
 | 22. WithGroupBy Validation | v0.4.2 | 1/1 | Complete    | 2026-04-10 |
 | 23. ORT EF Leak Fix | v0.4.2 | 1/1 | Complete    | 2026-04-11 |
 | 24. GetOrCreateCollection EF Safety | v0.4.2 | 1/1 | Complete    | 2026-04-12 |
-| 25. Error Body Truncation | v0.4.2 | 0/0 | Not started | - |
+| 25. Error Body Truncation | v0.4.2 | 4/4 | Complete   | 2026-04-13 |
 | 26. Twelve Labs Async Embedding | v0.4.2 | 0/0 | Not started | - |
 | 27. Download Stack Consolidation | v0.4.2 | 0/0 | Not started | - |
 | 28. Morph Test Fix | v0.4.2 | 0/0 | Not started | - |
 
@@ -2,15 +2,15 @@
 gsd_state_version: 1.0
 milestone: v0.4.2
 milestone_name: Bug Fixes and Robustness
-status: planning
-stopped_at: Phase 24 completed
-last_updated: "2026-04-12T15:44:20.760Z"
-last_activity: 2026-04-12
+status: ready
+stopped_at: Phase 25 shipped — PR #506
+last_updated: "2026-04-13T18:25:33.287Z"
+last_activity: "2026-04-13 -- Phase 25 shipped — PR #506"
 progress:
   total_phases: 11
-  completed_phases: 5
-  total_plans: 6
-  completed_plans: 6
+  completed_phases: 6
+  total_plans: 10
+  completed_plans: 10
   percent: 100
 ---
 
@@ -27,25 +27,38 @@ See: .planning/PROJECT.md (updated 2026-04-10)
 
 Phase: 30
 Plan: Not started
-Status: Ready for Phase 30 planning
-Last activity: 2026-04-12
+Status: Phase 25 shipped — PR #506
+Last activity: 2026-04-13 -- Phase 25 shipped — PR #506
 
 Progress: [██████████] 100%
 
 ## Performance Metrics
 
 **Velocity:**
 
-- Total plans completed: 6
-- Average duration: --
-- Total execution time: 0 hours
+- Total plans completed: 10
+- Average duration: 23 min
+- Total execution time: 23 min
 
 ## Accumulated Context
 
 ### Decisions
 
 Decisions are logged in PROJECT.md Key Decisions table.
 
+- [Phase 25]: Kept ReadLimitedBody and MaxResponseBodySize unchanged so transport safety and display safety stay separate concerns.
+- [Phase 25]: Sanitized OpenRouter's parsed error.message as body-derived text instead of trusting structured JSON fields to remain short.
+- [Phase 25]: Left ERR-02 pending because 25-01 only normalizes Perplexity/OpenRouter; later Phase 25 plans still migrate the remaining providers.
+- [Phase 25]: Kept provider-specific error wording intact and changed only raw-body segments to use SanitizeErrorBody(...)
+- [Phase 25]: Used OpenAI and Baseten as the representative long-body regressions for the first raw-body provider batch
+- [Phase 25]: Left ERR-02 pending because the remaining provider batches still belong to Plans 25-03 and 25-04
+- [Phase 25]: Cloudflare keeps parsed embeddings.Errors intact while sanitizing only the appended raw-body tail.
+- [Phase 25]: Cloudflare's mixed-format contract is enforced with a focused httptest regression instead of a source-only check.
+- [Phase 25]: Cohere's default embed model moved to embed-english-v3.0 after the retired v2.0 default blocked live ef verification on April 13, 2026.
+- [Phase 25]: Kept the batch-B provider edits mechanical by changing only the body-derived error segment and preserving existing status and endpoint wording. — This completed ERR-02 without widening the rollout into provider-specific wording changes.
+- [Phase 25]: Treated Twelve Labs parsed apiErr.Message values as body-derived text and sanitized them the same way as the raw fallback path. — The review-approved policy for parsed provider error text needed to stay consistent across OpenRouter and Twelve Labs.
+- [Phase 25]: Used a temporary authless DOCKER_CONFIG and a one-time ollama/ollama:latest pre-pull to unblock the required ollama ef verification. — The host Docker credsStore helper timed out on public-image pulls; isolating Docker config restored the intended verification path without repository changes.
+
 ### Roadmap Evolution
 
 - Phase 21.1 inserted after Phase 21: RRF cloud integration test coverage including arithmetic compositions (URGENT) — post-fix cloud coverage gap for Phase 21 arithmetic methods
@@ -57,5 +70,5 @@ Decisions are logged in PROJECT.md Key Decisions table.
 
 ## Session
 
-**Last Date:** 2026-04-12T14:47:20.000Z
-**Stopped At:** Phase 24 completed
+**Last Date:** 2026-04-13T08:13:19.127Z
+**Stopped At:** Phase 25 shipped — PR #506
@@ -0,0 +1,201 @@
+---
+phase: 25-error-body-truncation
+plan: 01
+type: execute
+wave: 1
+depends_on: []
+files_modified:
+  - pkg/commons/http/utils.go
+  - pkg/commons/http/utils_test.go
+  - pkg/embeddings/perplexity/perplexity.go
+  - pkg/embeddings/perplexity/perplexity_test.go
+  - pkg/embeddings/openrouter/openrouter.go
+  - pkg/embeddings/openrouter/openrouter_test.go
+autonomous: true
+requirements:
+  - ERR-01
+  - ERR-02
+
+must_haves:
+  truths:
+    - "`pkg/commons/http/utils.go` defines `SanitizeErrorBody(body []byte)` as the shared display-layer sanitizer, with whitespace trimming, rune-safe truncation, and the exact `[truncated]` suffix."
+    - "`SanitizeErrorBody(...)` follows `CLAUDE.md` panic-prevention guidance: it recovers from internal panics and returns the best sanitized result available instead of propagating a panic or surfacing an unsanitized full body."
+    - "`MaxResponseBodySize`, `ReadLimitedBody(...)`, and other transport guards remain unchanged so this plan affects display safety only."
+    - "`pkg/embeddings/perplexity/perplexity.go` and `pkg/embeddings/openrouter/openrouter.go` stop owning local truncation behavior and delegate to `chttp.SanitizeErrorBody(...)`."
+    - "OpenRouter treats parsed `error.message` as body-derived text and sanitizes it before returning it."
+    - "Focused tests in `pkg/commons/http/utils_test.go`, `pkg/embeddings/perplexity/perplexity_test.go`, and `pkg/embeddings/openrouter/openrouter_test.go` pin UTF-8-safe truncation and the `[truncated]` contract."
+  artifacts:
+    - path: "pkg/commons/http/utils.go"
+      provides: "Shared panic-safe error-body sanitizer"
+      contains: "func SanitizeErrorBody("
+    - path: "pkg/commons/http/utils_test.go"
+      provides: "Shared helper contract coverage for trim, rune-safe truncation, and suffix policy"
+      contains: "TestSanitizeErrorBody"
+    - path: "pkg/embeddings/openrouter/openrouter.go"
+      provides: "OpenRouter structured-error parsing routed through the shared sanitizer"
+      contains: "SanitizeErrorBody([]byte(apiErr.Error.Message))"
+    - path: "pkg/embeddings/perplexity/perplexity.go"
+      provides: "Perplexity HTTP error formatting normalized onto the shared sanitizer"
+      contains: "SanitizeErrorBody(respData)"
+  key_links:
+    - from: "pkg/commons/http/utils.go:SanitizeErrorBody"
+      to: "pkg/embeddings/perplexity/perplexity.go and pkg/embeddings/openrouter/openrouter.go"
+      via: "Shared display sanitizer replaces provider-local truncation helpers"
+      pattern: "SanitizeErrorBody"
+    - from: "pkg/commons/http/utils_test.go"
+      to: "pkg/embeddings/perplexity/perplexity_test.go and pkg/embeddings/openrouter/openrouter_test.go"
+      via: "Focused regressions keep the `[truncated]` contract aligned across the shared helper and both local precedents"
+      pattern: "[truncated]"
+---
+
+<objective>
+Establish the shared `SanitizeErrorBody(...)` contract in `pkg/commons/http` and normalize the two existing local precedents (`Perplexity` and `OpenRouter`) onto it.
+
+Purpose: this plan isolates the risky semantics change first: one shared helper, one shared suffix contract, one explicit policy for parsed OpenRouter message strings, and one CLAUDE.md-compliant panic-recovery rule. The later provider-audit plans can then be mechanical migrations instead of contract design work.
+
+Output: shared panic-safe sanitizer, updated helper/provider tests, and normalized Perplexity/OpenRouter error formatting.
+</objective>
+
+<execution_context>
+@/Users/tazarov/.codex/get-shit-done/workflows/execute-plan.md
+@/Users/tazarov/.codex/get-shit-done/templates/summary.md
+</execution_context>
+
+<context>
+@.planning/PROJECT.md
+@.planning/ROADMAP.md
+@.planning/REQUIREMENTS.md
+@.planning/STATE.md
+@.planning/phases/25-error-body-truncation/25-RESEARCH.md
+@.planning/phases/25-error-body-truncation/25-REVIEWS.md
+@.planning/phases/25-error-body-truncation/25-VALIDATION.md
+@pkg/commons/http/utils.go
+@pkg/commons/http/utils_test.go
+@pkg/embeddings/perplexity/perplexity.go
+@pkg/embeddings/perplexity/perplexity_test.go
+@pkg/embeddings/openrouter/openrouter.go
+@pkg/embeddings/openrouter/openrouter_test.go
+@CLAUDE.md
+
+<interfaces>
+From `pkg/commons/http/utils.go`:
+```go
+const MaxResponseBodySize = 200 * 1024 * 1024
+
+func ReadLimitedBody(r io.Reader) ([]byte, error) { ... }
+```
+
+From `pkg/embeddings/perplexity/perplexity.go`:
+```go
+const maxErrorBodyChars = 512
+
+func sanitizeErrorBody(body []byte) string { ... }
+```
+
+From `pkg/embeddings/openrouter/openrouter.go`:
+```go
+func parseAPIError(body []byte) string {
+	var apiErr apiErrorResponse
+	if err := json.Unmarshal(body, &apiErr); err == nil && apiErr.Error.Message != "" {
+		return apiErr.Error.Message
+	}
+	...
+}
+```
+</interfaces>
+</context>
+
+<tasks>
+
+<task type="auto" tdd="true">
+  <name>Task 1: Pin the shared sanitizer contract in tests before changing production code</name>
+  <files>pkg/commons/http/utils_test.go, pkg/embeddings/perplexity/perplexity_test.go, pkg/embeddings/openrouter/openrouter_test.go</files>
+  <behavior>
+    - `SanitizeErrorBody(nil)` and `SanitizeErrorBody([]byte(""))` return `""`
+    - leading/trailing whitespace is trimmed before truncation checks
+    - long values truncate by rune count, not bytes
+    - truncated values append the exact suffix `[truncated]`
+    - OpenRouter's parsed `error.message` path is treated as body-derived text and is expected to surface the shared sanitized contract
+    - compile-fail RED is acceptable for the brand-new helper because `SanitizeErrorBody(...)` does not exist yet
+  </behavior>
+  <action>
+Add or update focused tests first:
+
+- in `pkg/commons/http/utils_test.go`, add a `TestSanitizeErrorBody` table that covers nil/empty input, trim behavior, short-body passthrough, long ASCII truncation, and UTF-8-safe truncation
+- in `pkg/embeddings/perplexity/perplexity_test.go`, replace old `...(truncated)` expectations with `[truncated]`
+- in `pkg/embeddings/openrouter/openrouter_test.go`, update the raw fallback truncation assertion and add/strengthen a structured JSON case so an overlong `error.message` is expected to be sanitized too
+
+Use the focused command below as the RED/GREEN loop. The first RED run may fail to compile because the shared helper is not defined yet; that compile-fail is acceptable and should not be worked around by weakening the tests.
+  </action>
+  <verify>
+    <automated>cd /Users/tazarov/GolandProjects/chroma-go && go test -tags=ef ./pkg/commons/http ./pkg/embeddings/openrouter ./pkg/embeddings/perplexity</automated>
+  </verify>
+  <done>`utils_test.go`, `perplexity_test.go`, and `openrouter_test.go` all pin the exact `[truncated]` contract and the new structured-message policy before implementation changes begin.</done>
+</task>
+
+<task type="auto" tdd="true">
+  <name>Task 2: Implement the shared helper with panic recovery and normalize Perplexity/OpenRouter onto it</name>
+  <files>pkg/commons/http/utils.go, pkg/embeddings/perplexity/perplexity.go, pkg/embeddings/openrouter/openrouter.go</files>
+  <behavior>
+    - `SanitizeErrorBody(...)` is the single shared display sanitizer used by both providers
+    - `ReadLimitedBody(...)` and `MaxResponseBodySize` remain unchanged
+    - the helper recovers from panics and returns the best-effort sanitized result accumulated so far; if panic happens before the final value is assigned, the fallback path still enforces the same trim + rune-limit + `[truncated]` contract
+    - OpenRouter sanitizes both parsed `error.message` and raw fallback body text because both originate from the HTTP body
+  </behavior>
+  <action>
+In `pkg/commons/http/utils.go`, add `func SanitizeErrorBody(body []byte) (result string)` with this exact contract:
+
+- trim whitespace
+- truncate by rune count at the shared 512-rune display limit
+- append the exact suffix `[truncated]` when truncation occurs
+- include a `defer` panic-recovery block per `CLAUDE.md`; recovery must return the best `result` accumulated so far and, if `result` is still empty, it must re-apply the same trim + rune-limit + `[truncated]` contract to the fallback string rather than returning `strings.TrimSpace(string(body))` raw
+
+Then normalize the local precedents:
+
+- remove Perplexity's bespoke truncation helper and route the HTTP error path through `chttp.SanitizeErrorBody(respData)`
+- keep OpenRouter's JSON parse, but sanitize `apiErr.Error.Message` via `chttp.SanitizeErrorBody([]byte(apiErr.Error.Message))`; the raw fallback path must also use the shared helper
+
+Do not change transport-level helpers or widen scope into other providers in this plan.
+  </action>
+  <verify>
+    <automated>cd /Users/tazarov/GolandProjects/chroma-go && go test -tags=ef ./pkg/commons/http ./pkg/embeddings/openrouter ./pkg/embeddings/perplexity && make lint</automated>
+  </verify>
+  <done>The shared helper exists, follows the panic-recovery rule, Perplexity/OpenRouter delegate to it, and the focused helper/provider test set passes green.</done>
+</task>
+
+</tasks>
+
+<threat_model>
+## Trust Boundaries
+
+| Boundary | Description |
+|----------|-------------|
+| Provider HTTP response body -> returned Go error | Third-party APIs control both body size and body content; the SDK decides how much is surfaced to callers |
+
+## STRIDE Threat Register
+
+| Threat ID | Category | Component | Disposition | Mitigation Plan |
+|-----------|----------|-----------|-------------|-----------------|
+| T-25-01 | I (Information Disclosure) | `pkg/commons/http/utils.go` and provider error formatting | mitigate | Centralize body sanitization and stop returning provider bodies verbatim |
+| T-25-02 | T (Tampering) | local provider truncation helpers | mitigate | Replace Perplexity/OpenRouter local implementations with one shared helper |
+| T-25-03 | D (Denial of Service) | error/log readability | mitigate | Keep transport guard separate and add rune-safe display truncation with `[truncated]` |
+| T-25-04 | D (Denial of Service) | sanitizer runtime behavior | mitigate | Add panic recovery so sanitization failures degrade to partial results instead of crashing library callers |
+</threat_model>
+
+<verification>
+1. `go test -tags=ef ./pkg/commons/http ./pkg/embeddings/openrouter ./pkg/embeddings/perplexity` passes
+2. `pkg/commons/http/utils.go` contains `func SanitizeErrorBody(`
+3. `pkg/commons/http/utils.go` contains a `recover()` path for best-effort partial return behavior
+4. `pkg/embeddings/perplexity/perplexity.go` and `pkg/embeddings/openrouter/openrouter.go` both contain `SanitizeErrorBody`
+</verification>
+
+<success_criteria>
+- Phase 25 has a shared sanitizer contract anchored in `pkg/commons/http`
+- The helper follows the repo's no-panic rule and degrades to partial output on panic
+- Perplexity and OpenRouter no longer drift on suffix or truncation behavior
+- The shared helper and both local precedents are covered by focused green tests
+</success_criteria>
+
+<output>
+After completion, create `.planning/phases/25-error-body-truncation/25-01-SUMMARY.md`
+</output>