[DRAFT] [FEATURE] Pluggable external chunk text store to reduce Neptune Analytics memory usage

[mykola-pereyma]

⚠️ DRAFT — This issue is under discussion and not yet ready for implementation.

Problem Statement

ChunkGraphBuilder stores raw chunk text as a graph property:

'chunk.value = params.text'

At retrieval time, chunk text is accessed at these specific locations:

• keyword_vss_provider.py:81 — RETURN c.value AS content (explicit content fetch for keyword search)
• statement_enhancement.py:103 — node.node.metadata['chunk']['value'] (LLM synthesis context)
• traversal_based_base_retriever.py:136 — only when include_chunk_details=True (opt-in, defaults to False)
• update_chunk_metadata.py:18 — chunk.metadata.pop('value', None) (moves value from metadata dict)

Note: The main traversal-based retriever does NOT fetch chunk text by default (include_chunk_details=False). The memory cost is incurred regardless — Neptune keeps all node properties in-memory even when not queried.

For 5M chunks averaging 500 tokens (~2KB text), this consumes ~10GB of Neptune Analytics graph memory for text that is only needed at final answer synthesis.

Key insight: The traversal-based retriever already sets value: NULL in its projection — the architecture already treats chunk text as a late-binding fetch. This confirms external storage is architecturally aligned.

Proposed Solution

1. ChunkStore protocol (backend-agnostic interface):

class ChunkStore(ABC):
    @abstractmethod
    def get(self, chunk_id: str) -> str: ...
    @abstractmethod
    def put(self, chunk_id: str, text: str) -> None: ...
    @abstractmethod
    def get_batch(self, chunk_ids: List[str]) -> Dict[str, str]: ...

2. Reference implementations:

   • S3ChunkStore — S3 objects keyed by chunk_id, batch get via ThreadPoolExecutor
   • InGraphChunkStore — current behavior (reads chunk.value), for backward compat
   • Interface supports any object store (DynamoDB, Redis, local FS, third-party)

3. ChunkStoreFactory — registration pattern similar to GraphStoreFactory, allowing custom backends.

4. Write path change (ChunkGraphBuilder): when external store configured, skip chunk.value = params.text, call chunk_store.put(chunk_id, text).

5. Read path changes:

   • keyword_vss_provider.py:81: replace RETURN c.value AS content → ChunkStore.get_batch(chunk_ids)
   • update_chunk_metadata.py:18-21: resolve from ChunkStore
   • statement_enhancement.py:103: resolve from ChunkStore
   • get_chunks_query in chunk_utils.py:51: strip value from node_result, fetch from ChunkStore

Performance Impact

Metric	Current (in-graph)	Proposed (S3 store)
Graph memory (5M chunks)	+10GB	+250MB (refs only)
Chunk retrieval latency	<1ms (in-memory graph)	5-15ms (S3 GET)
Monthly cost (NA memory)	~$500/month for 10GB	~$5/month S3 storage
Retrieval batch (20 chunks)	<1ms	15-30ms (parallel S3 GET)
Graph traversal speed	Slower (large node properties evict cache)	Faster (smaller nodes, better cache utilization)

The 15-30ms retrieval overhead is negligible vs LLM synthesis latency (1-5s). Net effect: faster graph traversal + lower cost.

Why smaller nodes improve traversal: Neptune Analytics loads full node properties into memory pages. Large text values (~2KB per node) reduce the effective working set for graph traversal queries that never access the text. Removing text increases the ratio of structural data in cache, improving traversal query performance.

Acceptance Criteria

• ChunkStore is a pluggable interface — supports any object store (S3, DynamoDB, Redis, local FS, third-party)
• ChunkStoreFactory with registration pattern (like GraphStoreFactory)
• InGraphChunkStore as default — existing behavior unchanged
• S3ChunkStore supports batch retrieval with configurable parallelism
• All 4 retrieval read paths route through ChunkStore when configured
• Graph memory reduction measured (before/after on benchmark)
• No change to retrieval quality (same chunks selected, same text returned)

Alternatives Considered

• Neptune Analytics memory scaling — provision more memory. Doesn't solve the cost problem; linear cost increase for data that's rarely accessed.
• Compress chunk text in-graph — reduces memory but adds CPU overhead on every read and doesn't enable pluggable backends.