fix(ce): reconcile embedder vector dimensionality (HashingEmbedder 256 vs index default 1536)

[nndn]

Context

The bundled OSS HashingEmbedder emits 256 dims (adapters/outbound/intelligence/local_embedder.py:38 — _DEFAULT_DIMENSIONS = 256), but every vector-index creator defaults to 1536 (OpenAI text-embedding-3-small) via getattr(self._embedder, "dimensions", 1536):

• adapters/outbound/graph/neo4j_writer.py:80
• adapters/outbound/graph/falkordb_writer.py:233 (plus _ensure_indexes_sync(..., embedding_dim: int = 1536) at :238)
• adapters/outbound/graph/cypher.py:759 — ensure_canonical_indexes(driver, *, embedding_dim: int = 1536) (comment at :770 cites text-embedding-3-small)

This is consistent when an embedder is wired (the getattr reads .dimensions == 256). The hazard is the fallback path: the writers/backends accept embedder=None:

• adapters/outbound/graph/neo4j_writer.py:42
• adapters/outbound/graph/falkordb_writer.py:194
• adapters/outbound/graph/backends/embedded_backend.py:44

If ensure_indexes runs with no embedder it creates a 1536-dim index, while embedding writes (once an embedder is wired) emit 256-dim vectors — the index width mismatches the stored data and semantic reads silently degrade to lexical fallback. Swapping in sentence-transformers (the embeddings extra in pyproject.toml) introduces a third width (384/768/…), compounding the drift.

Acceptance criteria

• Index creation and embedding writes share one source-of-truth dimension on both Neo4j and FalkorDB.
• The 1536 magic default is removed — derive the width from the configured embedder, or fail closed when no embedder is wired (rather than guessing 1536).
• A test asserts the created index dimension == the embedder's .dimensions across the embedded / FalkorDB / sentence-transformers paths.

Provenance

Carry-forward from the cg-05 backends PR (folds the original PR-12 embedding-claim-query + PR-13 FalkorDB backend). Deferred deliberately; tracked here rather than blocking the PR.

[linear-code]

POT-1701

[coderabbitai]

Coding Plan

Summary

• Remove the hardcoded 1536 dimension from ensure_canonical_indexes and make the dimension an explicit parameter
• Inject embedder into Neo4jGraphWriter to provide the dimension source-of-truth
• When no embedder is wired, skip vector index creation entirely (fail closed) rather than guessing wrong
• Leave FalkorDB unchanged since it already intentionally skips vector indexes
• Add tests verifying the dimension flows correctly from embedder to index DDL

Design Choices

Design Choice 1: How should "fail closed" behave when no embedder is wired?

Options Considered:

1. Raise an error if vector index is needed but no embedder is provided
2. Skip vector index creation entirely and rely on lexical fallback (matching existing pattern for unsupported Neo4j editions)

Chosen Option: 2

Rationale: Skip vector index with a warning log — this matches the existing graceful degradation pattern in ensure_canonical_indexes and keeps the system functional for users who haven't configured an embedder yet.

Design Choice 2: Should FalkorDB writer changes be included?

Options Considered:

1. Modify FalkorDBGraphWriter to accept embedder parameter for future use
2. Leave FalkorDB unchanged since it intentionally skips vector indexes

Chosen Option: 2

Rationale: Leave FalkorDB unchanged — the vector index is intentionally omitted and reads already use token-overlap fallback. Adding embedder plumbing would be dead code until FalkorDB supports vector indexes.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Plumb Embedder Dimensions Through Index Creation

This phase establishes the single source-of-truth for embedding dimensions by modifying the index creation path to accept dimensions from a wired embedder rather than using a hardcoded default.

Task 1: Modify ensure_canonical_indexes to Accept Embedding Dimension

Remove the hardcoded 1536 default and require callers to explicitly provide the dimension or signal "no vector index needed."

• Update function signature in adapters/outbound/graph/cypher.py to accept embedding_dim: int | None parameter
• When embedding_dim is None, skip the vector index creation block entirely (log at INFO level that vector index is skipped due to no embedder)
• When embedding_dim is provided, use that value in the vector.dimensions option of the Cypher DDL
• Update the docstring to reflect the new behavior: dimension is required for vector index, omitting it skips the index

Task 2: Add Embedder Parameter to Neo4jGraphWriter

Enable the writer to receive an embedder instance and extract its dimensions for index creation.

• Add optional embedder: EmbedderPort | None = None parameter to Neo4jGraphWriter.__init__ in adapters/outbound/graph/neo4j_writer.py
• Store embedder as instance attribute self._embedder
• In ensure_indexes method, extract dimension via self._embedder.dimensions if self._embedder else None
• Pass the extracted dimension to ensure_canonical_indexes

Task 3: Update Bootstrap Wiring for Embedder Injection

Ensure composition roots pass through any configured embedder to the Neo4j writer.

• Update build_ingestion_server in bootstrap/ingestion_server.py to accept optional embedder parameter and pass it to Neo4jGraphWriter
• The bootstrap currently has no embedder to inject (no concrete implementation exists), so the parameter remains optional and defaults to None
• This creates the seam for future embedder implementations to be wired in

🤖 Prompt for AI agents

This phase removes the hardcoded 1536 vector dimension and replaces it with an
explicit, embedder-sourced dimension plumbed through the index creation path.

**Task 1: Modify `ensure_canonical_indexes` in
`adapters/outbound/graph/cypher.py`**
- Update the function signature to accept a new parameter: `embedding_dim: int |
None`
- When `embedding_dim` is `None`, skip the vector index creation block entirely
and log at INFO level that the vector index is being skipped because no embedder
is configured
- When `embedding_dim` is provided, use its value in the `vector.dimensions`
option of the Cypher DDL (replacing the hardcoded 1536)
- Update the function's docstring to reflect this new behavior

**Task 2: Modify `Neo4jGraphWriter` in
`adapters/outbound/graph/neo4j_writer.py`**
- Add an optional `embedder: EmbedderPort | None = None` parameter to `__init__`
- Store it as `self._embedder`
- In the `ensure_indexes` method, compute the dimension as
`self._embedder.dimensions if self._embedder else None`
- Pass this computed dimension to `ensure_canonical_indexes`

**Task 3: Update `build_ingestion_server` in `bootstrap/ingestion_server.py`**
- Add an optional `embedder` parameter (defaulting to `None`) to the function
- Forward this `embedder` argument when instantiating `Neo4jGraphWriter`
- No concrete embedder implementation is expected yet; this creates the wiring
seam for future use

Phase 2: Testing and Verification

This phase adds tests that verify the dimension consistency contract: when an embedder is wired, the created index uses the embedder's dimensions.

Task 1: Create Test Embedder Stub

Provide a minimal embedder implementation for testing that conforms to EmbedderPort.

• Create a _TestEmbedder class in the test file that implements EmbedderPort protocol
• Implement name property returning a test identifier
• Implement dimensions property returning a configurable value (e.g., 256 to match planned HashingEmbedder)
• Implement embed and embed_many methods returning zero vectors of correct dimension (sufficient for index creation tests)

Task 2: Add Dimension Consistency Test for Neo4j Writer

Verify that the writer uses the embedder's dimensions when creating indexes.

• Create test file tests/unit/test_neo4j_writer.py (currently missing)
• Add test that instantiates Neo4jGraphWriter with a test embedder having dimensions=256
• Mock the Neo4j driver to capture the Cypher statements
• Assert the vector index DDL contains vector.dimensions: 256

Task 3: Add Test for No-Embedder Path

Verify that when no embedder is wired, the vector index is skipped rather than defaulting to 1536.

• Add test that instantiates Neo4jGraphWriter with embedder=None
• Call ensure_indexes with mocked driver
• Assert the vector index DDL (CREATE VECTOR INDEX claim_fact_embeddings) is NOT executed
• Assert the non-vector indexes (entity_group_key, claim_group_name, etc.) ARE created

🤖 Prompt for AI agents

This phase adds unit tests to `tests/unit/test_neo4j_writer.py` (a new file)
verifying the dimension consistency contract introduced in Phase 1.

**Task 1: Create `_TestEmbedder` stub class in the test file**
- Implement the `EmbedderPort` protocol with:
  - `name` property returning a test identifier string
- `dimensions` property returning a configurable integer (use 256 to match the
planned HashingEmbedder)
- `embed` and `embed_many` methods returning zero vectors of the correct
dimension

**Task 2: Test — embedder dimensions flow to vector index DDL**
- Instantiate `Neo4jGraphWriter` with a `_TestEmbedder` configured with
`dimensions=256`
- Mock the Neo4j driver to capture all executed Cypher statements
- Call `ensure_indexes` and assert the captured DDL includes `vector.dimensions:
256`

**Task 3: Test — no embedder skips vector index creation**
- Instantiate `Neo4jGraphWriter` with `embedder=None`
- Mock the Neo4j driver to capture all executed Cypher statements
- Call `ensure_indexes` and assert that `CREATE VECTOR INDEX
claim_fact_embeddings` is NOT among the executed statements
- Assert that the non-vector index DDL statements (e.g., entity_group_key,
claim_group_name) ARE still executed

Research

The context-engine follows a hexagonal architecture with domain ports, adapters, and bootstrap wiring. The EmbedderPort protocol exists in domain/ports/embedder.py with a .dimensions property, but no concrete embedder implementation is wired anywhere. The ensure_canonical_indexes function in cypher.py hardcodes 1536 for the vector index dimension. Writers (Neo4jGraphWriter, FalkorDBGraphWriter) accept no embedder parameter. FalkorDB intentionally skips vector indexes entirely (documented in code). The embedded/in-memory backends create no indexes and use token-overlap fallback for semantic search.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

This phase removes the hardcoded 1536 vector dimension and replaces it with an
explicit, embedder-sourced dimension plumbed through the index creation path.

**Task 1: Modify `ensure_canonical_indexes` in
`adapters/outbound/graph/cypher.py`**
- Update the function signature to accept a new parameter: `embedding_dim: int |
None`
- When `embedding_dim` is `None`, skip the vector index creation block entirely
and log at INFO level that the vector index is being skipped because no embedder
is configured
- When `embedding_dim` is provided, use its value in the `vector.dimensions`
option of the Cypher DDL (replacing the hardcoded 1536)
- Update the function's docstring to reflect this new behavior

**Task 2: Modify `Neo4jGraphWriter` in
`adapters/outbound/graph/neo4j_writer.py`**
- Add an optional `embedder: EmbedderPort | None = None` parameter to `__init__`
- Store it as `self._embedder`
- In the `ensure_indexes` method, compute the dimension as
`self._embedder.dimensions if self._embedder else None`
- Pass this computed dimension to `ensure_canonical_indexes`

**Task 3: Update `build_ingestion_server` in `bootstrap/ingestion_server.py`**
- Add an optional `embedder` parameter (defaulting to `None`) to the function
- Forward this `embedder` argument when instantiating `Neo4jGraphWriter`
- No concrete embedder implementation is expected yet; this creates the wiring
seam for future use
===============================================================================

Task: 2

This phase adds unit tests to `tests/unit/test_neo4j_writer.py` (a new file)
verifying the dimension consistency contract introduced in Phase 1.

**Task 1: Create `_TestEmbedder` stub class in the test file**
- Implement the `EmbedderPort` protocol with:
  - `name` property returning a test identifier string
- `dimensions` property returning a configurable integer (use 256 to match the
planned HashingEmbedder)
- `embed` and `embed_many` methods returning zero vectors of the correct
dimension

**Task 2: Test — embedder dimensions flow to vector index DDL**
- Instantiate `Neo4jGraphWriter` with a `_TestEmbedder` configured with
`dimensions=256`
- Mock the Neo4j driver to capture all executed Cypher statements
- Call `ensure_indexes` and assert the captured DDL includes `vector.dimensions:
256`

**Task 3: Test — no embedder skips vector index creation**
- Instantiate `Neo4jGraphWriter` with `embedder=None`
- Mock the Neo4j driver to capture all executed Cypher statements
- Call `ensure_indexes` and assert that `CREATE VECTOR INDEX
claim_fact_embeddings` is NOT among the executed statements
- Assert that the non-vector index DDL statements (e.g., entity_group_key,
claim_group_name) ARE still executed

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!

[AdityaPainuli]

@nndn I'd like to pick this one up, but want to confirm scope first since #1002 already covers a chunk of it.

From what I can see, #1002 handles the FalkorDB side: per-record embedding_dim, mis-sized embedding detection, and the index drop + rebuild on dimension change. What's left of this issue after that merges:

• cypher.py ensure_canonical_indexes still defaults embedding_dim to 1536
• neo4j_writer.py still falls back to 1536 via getattr(self._embedder, "dimensions", 1536)
• the embedder=None path still creates a 1536-dim index instead of failing closed

My plan: Neo4j side only. Derive the dimension from the wired embedder, skip vector index creation with a warning when no embedder is wired (matches the existing graceful-degradation pattern in ensure_canonical_indexes), remove the 1536 magic default, and add a test asserting created index dim == embedder .dimensions. Leaving FalkorDB alone since you're handling it in #1002.

Does that split make sense, or do you want the FalkorDB fallback cleanup bundled here too once #1002 lands?