Merge pull request #110 from SpillwaveSolutions/feat/phase5-storage-abstraction

feat: Phase 5 — Storage Backend Abstraction Layer

Author: RichardHightower

.claude/settings.local.json

@@ -57,7 +57,23 @@
       "Bash(/Users/richardhightower/clients/spillwave/src/doc-serve/.claude/skills/installing-local/remove.sh:*)",
       "Bash(git rev-parse:*)",
       "Bash(bash:*)",
-      "Bash(poetry --version:*)"
+      "Bash(poetry --version:*)",
+      "Bash(node:*)",
+      "Bash(poetry run black:*)",
+      "Bash(gh pr create:*)",
+      "Bash(gh pr edit:*)",
+      "Bash(gh auth status:*)",
+      "Bash(gh auth switch:*)",
+      "Bash(.venv/bin/pip install:*)",
+      "Bash(.venv/bin/pytest:*)",
+      "Skill(gsd:new-milestone)",
+      "Bash(poetry show:*)",
+      "Bash(poetry run python:*)",
+      "Bash(task:*)",
+      "Bash(gh run list:*)",
+      "Bash(git checkout:*)",
+      "Bash(git pull:*)",
+      "Skill(gsd:plan-phase)"
     ]
   },
   "sandbox": {

.github/workflows/pr-qa-gate.yml

@@ -16,6 +16,7 @@ jobs:
   qa-gate:
     name: Quality Assurance Gate
     runs-on: ubuntu-latest
+    environment: ci-testing
 
     steps:
       - name: Free up disk space
@@ -121,6 +122,10 @@ jobs:
           echo "✓ Versions aligned: $SERVER_VERSION"
 
       - name: Run tests with coverage (Server)
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+          COHERE_API_KEY: ${{ secrets.COHERE_API_KEY }}
         run: task server:pr-qa-gate
 
       - name: Run tests with coverage (CLI)

.github/workflows/provider-e2e.yml

@@ -76,6 +76,7 @@ jobs:
   provider-tests:
     name: Provider Tests - ${{ matrix.provider }}
     runs-on: ubuntu-latest
+    environment: ci-testing
     if: github.event_name == 'push' || contains(github.event.pull_request.labels.*.name, 'test-providers')
 
     strategy:

.planning/REQUIREMENTS.md

@@ -109,13 +109,13 @@ Which phases cover which requirements. Updated during roadmap creation.
 
 | Requirement | Phase | Status |
 |-------------|-------|--------|
-| STOR-01 | Phase 5 | Pending |
-| STOR-02 | Phase 5 | Pending |
-| STOR-03 | Phase 5 | Pending |
-| STOR-04 | Phase 5 | Pending |
-| CONF-01 | Phase 5 | Pending |
-| CONF-02 | Phase 5 | Pending |
-| CONF-03 | Phase 5 | Pending |
+| STOR-01 | Phase 5 | Done |
+| STOR-02 | Phase 5 | Done |
+| STOR-03 | Phase 5 | Done |
+| STOR-04 | Phase 5 | Done |
+| CONF-01 | Phase 5 | Done |
+| CONF-02 | Phase 5 | Done |
+| CONF-03 | Phase 5 | Done |
 | PGVEC-01 | Phase 6 | Pending |
 | PGVEC-02 | Phase 6 | Pending |
 | PGVEC-03 | Phase 6 | Pending |

.planning/ROADMAP.md

@@ -36,11 +36,11 @@
   3. ChromaBackend adapter passes all storage protocol operations without breaking existing functionality
   4. Environment variable AGENT_BRAIN_STORAGE_BACKEND overrides config file selection
   5. Server startup validates backend configuration and fails fast on misconfiguration
-**Plans**: TBD
+**Plans**: 2 plans
 
 Plans:
-- [ ] 05-01: TBD
-- [ ] 05-02: TBD
+- [x] 05-01-PLAN.md — StorageBackendProtocol definition, StorageConfig YAML schema, backend factory
+- [x] 05-02-PLAN.md — ChromaBackend adapter, service refactor to protocol, integration verification
 
 #### Phase 6: PostgreSQL Backend Implementation
 **Goal**: Implement PostgreSQL backend with pgvector for vector search, tsvector for full-text search, and proper async connection pooling
@@ -106,7 +106,7 @@ Phases execute in numeric order: 5 → 6 → 7 → 8
 | 2. Pluggable Providers | v3.0 | 4/4 | Complete | 2026-02-09 |
 | 3. Schema-Based GraphRAG | v3.0 | 2/2 | Complete | 2026-02-10 |
 | 4. Provider Integration Testing | v3.0 | 2/2 | Complete | 2026-02-10 |
-| 5. Storage Abstraction | v5.0 | 0/? | Not started | - |
+| 5. Storage Abstraction | v5.0 | 2/2 | Complete | 2026-02-10 |
 | 6. PostgreSQL Backend | v5.0 | 0/? | Not started | - |
 | 7. Testing & CI | v5.0 | 0/? | Not started | - |
 | 8. Plugin & Documentation | v5.0 | 0/? | Not started | - |
@@ -155,4 +155,4 @@ Feature 101: AST-aware code ingestion, code summaries
 
 ---
 *Roadmap created: 2026-02-07*
-*Last updated: 2026-02-10 — v5.0 roadmap created with 4 phases covering 34 requirements*
+*Last updated: 2026-02-10 — Phase 5 complete (2/2 plans, verified)*

.planning/STATE.md

@@ -2,29 +2,29 @@
 
 **Last Updated:** 2026-02-10
 **Current Milestone:** v5.0 PostgreSQL Backend
-**Status:** Ready to plan Phase 5
+**Status:** Phase 5 complete — ready for Phase 6
 
 ## Current Position
 
-Phase: 5 of 8 (Storage Backend Abstraction Layer)
-Plan: Ready to plan (no plans created yet)
-Status: Ready to plan
-Last activity: 2026-02-10 — v5.0 roadmap created with 4 phases covering 34 requirements
+Phase: 5 of 8 (Storage Backend Abstraction Layer) — COMPLETE
+Plan: 2 of 2 complete
+Status: Verified — 11/11 must-haves, 5/5 success criteria
+Last activity: 2026-02-10 — Phase 5 verified and complete
 
-Progress: [████░░░░░░] 50% (v3.0 milestone complete, v5.0 milestone started)
+Progress: [██░░░░░░░░] 25% (Phase 5 complete, Phases 6-8 remaining)
 
 ## Project Reference
 
 See: .planning/PROJECT.md (updated 2026-02-10)
 
 **Core value:** Developers can semantically search their entire codebase and documentation through a single, fast, local-first API that understands code structure and relationships
-**Current focus:** Phase 5 - Storage Backend Abstraction Layer
+**Current focus:** Phase 6 - PostgreSQL Backend Implementation
 
 ## Milestone Summary
 
 ```
 v3.0 Advanced RAG:     [██████████] 100% (shipped 2026-02-10)
-v5.0 PostgreSQL:       [░░░░░░░░░░]   0% (4 phases, 34 requirements)
+v5.0 PostgreSQL:       [██░░░░░░░░]  25% (Phase 5 complete, 3 phases remaining)
 ```
 
 ## Performance Metrics
@@ -44,58 +44,67 @@ v5.0 PostgreSQL:       [░░░░░░░░░░]   0% (4 phases, 34 requi
 | Phase 4: Provider Integration Testing | 2 | Complete |
 
 **v5.0 milestone:**
-- Total plans: TBD (to be determined during phase planning)
-- Expected phases: 4 (Phases 5-8)
-- Requirements coverage: 34/34 mapped (100%)
+- Total plans: TBD (Phase 5: 2 plans, Phases 6-8: TBD during planning)
+- Phase 5: 2/2 plans complete (05-01, 05-02)
+- Average duration: ~10 minutes per plan
+- Requirements coverage: 34/34 mapped (100%), 7/34 done
 
-*Updated after v5.0 roadmap creation*
+**Phase 5 Metrics:**
+
+| Plan | Duration | Tasks | Tests Added | Status |
+|------|----------|-------|-------------|--------|
+| 05-01 | 8 min | 3/3 | +33 | Complete |
+| 05-02 | 11 min | 3/3 | +20 | Complete |
 
 ## Accumulated Context
 
 ### From v3.0 Advanced RAG
-- Pluggable provider pattern (YAML config) works well — reuse for backend selection
-- Storage layer currently tightly coupled to ChromaDB — will need abstraction
-- 505 tests passing, 70% coverage — must maintain through refactor
+- Pluggable provider pattern (YAML config) works well — reused for backend selection
+- 559 tests passing, 70% coverage after Phase 5 refactor
 - Existing architecture: ChromaDB (vectors), disk BM25 (keyword), SimplePropertyGraphStore (graph)
 - Dual-layer validation pattern (startup warning + runtime error) proven effective
 
-### Decisions
-
-Recent decisions from PROJECT.md affecting v5.0 work:
+### From Phase 5 (Storage Abstraction)
+- StorageBackendProtocol defines 11 async methods (initialize, upsert, vector_search, keyword_search, etc.)
+- ChromaBackend wraps VectorStoreManager + BM25IndexManager via composition
+- BM25 scores normalized to 0-1 (per-query max normalization)
+- Backend factory: env var > YAML > default("chroma")
+- Services accept both old-style (vector_store, bm25_manager) and new (storage_backend) constructors
 
-- v3.0: Skill + CLI over MCP — User preference: simpler, less context overhead
-- v3.0: Dual-layer validation (startup warning + indexing error) — Warns on startup, blocks only when data integrity at risk
-- v3.0: CI matrix with conditional API key checks — Tests skip gracefully, config tests always run
-- v5.0: PostgreSQL as optional dual backend — ChromaDB remains default for local-first simplicity
-
-### Pending Todos
+### Decisions
 
-None yet (v5.0 milestone just started).
+- v3.0: Skill + CLI over MCP — simpler, less context overhead
+- v3.0: Dual-layer validation (startup warning + indexing error)
+- v3.0: CI matrix with conditional API key checks
+- v5.0: PostgreSQL as optional dual backend — ChromaDB remains default
+- 05-01: Protocol over ABC — structural subtyping, no inheritance required
+- 05-01: Normalize scores to 0-1 range — consistent across backends
+- 05-01: Singleton factory pattern — matches existing pattern
+- 05-02: Adapter pattern — composition over code movement
+- 05-02: BM25 rebuild stays in IndexingService — full-corpus operation
+- 05-02: Per-query BM25 normalization — divide by max score
+- 05-02: Backward-compatible constructors — preserves 505+ test patterns
 
 ### Blockers/Concerns
 
-**Phase 5 (Storage Abstraction):**
-- Must design async-first protocol before implementation to avoid sync/async mixing pitfalls
-- ChromaDB is synchronous; must wrap in asyncio.to_thread() for protocol compliance
-
 **Phase 6 (PostgreSQL Implementation):**
 - LlamaIndex llama-index-vector-stores-postgres version compatibility with existing llama-index-core ^0.14.0 needs validation
 - Connection pool sizing must be tuned for concurrent load (research shows default 10 may be insufficient)
 - HNSW index build on large corpora (100k+ docs) can take hours and consume 64GB+ memory
 
 **Phase 7 (Testing & CI):**
 - CI must support PostgreSQL service container without breaking existing ChromaDB-only tests
-- Score normalization between ChromaDB BM25 (0-10 range) and PostgreSQL ts_rank (0-1 range) needs tuning
+- Score normalization between ChromaDB BM25 and PostgreSQL ts_rank needs tuning
 
 **Phase 8 (Plugin & Documentation):**
 - Plugin must guide users through backend selection without overwhelming with complexity
 - Documentation must clarify no auto-migration tool (users rebuild from source)
 
 ## Session Continuity
 
-Last session: 2026-02-10 (roadmap creation)
-Stopped at: Created v5.0 roadmap with 4 phases, validated 100% requirement coverage (34/34 requirements mapped)
-Resume file: None (start with /gsd:plan-phase 5)
+Last session: 2026-02-10 (Phase 5 execution + verification)
+Stopped at: Phase 5 complete and verified. Ready for Phase 6 planning.
+Resume file: Run `/gsd:plan-phase 6` to start Phase 6 (PostgreSQL Backend Implementation)
 
 ---
 *State updated: 2026-02-10*

.planning/config.json

@@ -14,4 +14,4 @@
     "migrated_at": "2026-02-07",
     "legacy_path": ".speckit/"
   }
-}
+}