Status: deferred architecture backlog (frozen after Stabilization Gate S5,
2026-07-14 — ADR-0042)
Scope: production-grade capabilities that should be designed for, but not
implemented before the hypothesis-validation CLI proof.
This document extends roadmap-context-core.md.
Lab-ready / CLI proof path is already shipped — see .project/README.md
and docs/lab-gate.md. Use this file for deferred
production gates only.
Owner for all rows: core steward. Reopen only with measured blocker + ADR.
| Layer | Status after S5 | Notes |
|---|---|---|
| L01 Threat / injection | partial closed | Trust + pack separation + adversarial fixtures (C5); ML classifier deferred |
| L02 Fine-grained ACL | frozen-deferred | Project isolation shipped; OIDC/ACL later |
| L03 Index lifecycle | closed | C1 tombstones + lifecycle explain |
| L03A Snapshot reuse / COW | partial / frozen | Export/import (C2); simhash COW deferred |
| L04 Query AST | minimal subset shipped | Operator layer (phrase/AND-OR-NOT/~morph/lang:) — ADR-0043; general DSL & field syntax stay deferred per ADR-0041 |
| L04A Focus / memory tiers | partial | FocusProfile shipped; deep memory tiers deferred |
| L05 Snippets | closed | C4 / ADR-0033 |
| L05A Language adapters | partial+ | testkit + en thin + ru rule-based (ADR-0043); dictionary engines external |
| L05B Lexicon resources | partial | curated JSON; TEI external |
| L05C Event adapters | partial | NDJSON thin; richer exports external |
| L06 Relevance feedback | frozen-deferred | |
| L07 Claim/citation graph | forever-defer in core | ADR-0040 |
| L08 Distributed jobs | frozen-deferred | Single-node scheduler shipped (C8) |
| L09 Tool sandbox | partial | Side-effect ask baseline (C6); full sandbox deferred |
| L10 SLOs / capacity | frozen-deferred | |
| L11 Crawler governance | frozen-deferred | |
| L12 Multi-modal / binary | partial | HTML/PDF thin; OCR/DOCX deferred |
| L13 Privacy / encryption | frozen-deferred | Soft delete/export hooks minimal (C7) |
| L14 Multi-tenant governance | frozen-deferred | Isolation design only |
| L15 API/SDK | partial | v1 freeze + changelog; OpenAPI deferred |
| L15A Lab workbench/DSL | frozen-deferred | Outside core |
| L16 Production review gates | frozen-deferred | Product/ops process |
The layers below are intentionally deferred. They are the gates that turn a working context engine into a reliable private search system, distributed agent runtime, and long-lived production platform.
Do not implement these layers early unless a proof-of-concept chunk is blocked by their absence.
For each future layer:
- Define the contract early.
- Keep the interface narrow.
- Add tests before adding infrastructure.
- Implement the smallest adapter first.
- Promote to distributed or optimized form only after measurement.
The engine will ingest untrusted text from files, web pages, logs, tool outputs, and user-authored documents. Any source can try to influence agent behavior. Without an explicit threat model, retrieval can become a prompt-injection delivery mechanism.
- Threat model document covering source poisoning, prompt injection, cross-project leakage, tool exfiltration, web page attacks, and malicious documents.
- Source trust levels:
trusted,project,external,untrusted,quarantined. - Instruction/data separation in context packs.
- Prompt-injection classifier hooks for external content.
- Policy that forbids retrieved source text from overriding system, developer, project, or tool policies.
- Quarantine path for suspicious sources.
- Tests with adversarial documents.
- A malicious source can be retrieved as evidence without gaining tool permissions or overriding runtime policy.
- The context pack clearly labels untrusted content.
- Prompt-injection fixtures are part of the regression suite.
Project-level isolation is enough for the first PoC. Production systems need permissions below the project boundary: source, artifact, chunk, tool output, agent run, report, and background result.
For sensitive observation, accessibility, health, education, or human-behavior streams, this layer and Layer 13 are pre-production gates, not optional post-launch hardening. They remain outside the first local CLI proof.
- ACL model for
Project,Source,Artifact,Chunk,ContextPack,AgentRun,ToolCall, andEvaluation. - Permission filters applied before candidate merge, rerank, model calls, and tool execution.
- Access decisions stored in traces.
- Denied-candidate counters for debugging.
- Tenant/project/user/service-account identities.
- Contract tests proving no cross-project or cross-user leakage.
- A user cannot retrieve, rerank, summarize, or indirectly expose a chunk they cannot read.
- Permission checks happen outside the LLM.
- Retrieval traces show which filters were applied.
Indexes decay. Sources move, chunks change, embeddings get replaced, parsers are upgraded, and old vectors become stale. Without lifecycle rules, search results will become unreliable and hard to debug.
- Index state machine:
new,indexing,ready,degraded,stale,rebuilding,failed,archived. - Tombstones for deleted sources and chunks.
- Stale chunk cleanup.
- Orphan artifact cleanup.
- Parser/chunker/enricher/embedding/sparse/graph version coexistence.
- Zero-downtime reindex with old/new index side by side.
- Backfill jobs.
- Rebuild and repair CLI commands.
- Index health report.
- Reindexing a project does not make search unavailable.
- Deleted sources cannot appear in new context packs.
- The system can explain which index version produced a result.
ADR-0012 defines IndexSnapshot as the sync unit so local and cloud retrieval can
share the same ranking inputs. The first PoC should prove local snapshot commit
and search. Production-grade replication, reuse, and cross-device parity require
additional safety gates.
- Snapshot export/import CLI and service APIs.
- Sparse index bundle hashes in manifest records.
- VectorStore point export or snapshot restore validation by
snapshot_id. - Local pull command that verifies
source_merkle_root,chunk_set_hash, bundle hashes, and engine versions before flippingactive_snapshot_id. - Incremental segment sync for sparse bundles.
- Snapshot retention and garbage collection.
- Simhash over chunk multisets for near-duplicate snapshot discovery.
- Copy-on-write seed for project templates or near-identical workspaces.
- Merkle content proofs before any cross-user or cross-team index reuse.
- Local and cloud search can prove they are using the same snapshot and engine versions.
- A corrupted or partial snapshot cannot become active.
- Cross-user reuse is impossible until content proofs and ACL filters are tested.
S4 / C10: the ADR-0041 reopen condition was met by the lexical-search product mandate; a minimal operator subset (phrase / AND-OR-NOT / ~morph / lang:) shipped via ADR-0043 with golden tests and trace payloads (docs/search-operators.md). General DSL, field syntax, proximity, and wildcards remain deferred.
Natural language retrieval is not enough for a private search system. Users need precise search when investigating facts, legal text, scientific sources, logs, or configuration.
- Query parser for:
AND,OR,NOT- exact phrases
- proximity search
- field filters
- source filters
- date/time filters
- language filters
- citation filters
- fuzzy flags
- wildcard flags
- Query AST with validation.
- Conversion from query AST to retriever calls.
- Explanation of how a query was interpreted.
- Safe fallback when a query cannot be parsed.
- Advanced queries are deterministic.
- Query parsing is test-covered with golden inputs.
- Query interpretation is visible in retrieval traces.
When a project has too many documents, messages, tasks, logs, and tool outputs, accuracy drops unless the runtime can focus retrieval before calling a model. Focus control is a neutral core concern because every downstream product needs it, regardless of domain.
FocusProfilepersistence and inspection.- Memory tiers:
- raw artifacts;
- indexed chunks;
- active task context;
- accepted decisions;
- summaries with source refs;
- negative assumptions;
- rejected candidates.
- Focus constraints for:
- source types;
- source trust levels;
- freshness windows;
- exactness level;
- citation strictness;
- token/context budget;
- allowed tools and subagents;
- explicitly irrelevant areas.
- Focus inspector for debugging why a context pack used some sources and ignored others.
- Focus regression tests for large corpora.
- Large corpora can be searched through a bounded focus profile.
- The runtime can explain which focus constraints shaped retrieval.
- A model call never receives broad corpus context without an explicit focus decision.
Search quality is not only ranking. Users and agents need exact evidence presentation: where the match happened, why it matched, and what surrounding context is safe to include.
- Offset-preserving snippets.
- Term highlighting.
- Lemma/stem match highlighting.
- Citation span display.
- Before/after context windows.
- Page, section, heading, paragraph, and line labels.
- Quote-safe extraction for scientific/legal text.
- Snippet checksums tied to source versions.
- Search results can show exact source snippets without re-tokenizing differently from the index.
- Highlighting is stable across repeated runs.
- Snippets preserve provenance.
The core must support Russian, English, German, Spanish, French, Hindi, Indic languages, and future languages without becoming a language-specific NLP engine. Each language has different morphology, scripts, dictionaries, segmentation rules, and ambiguity patterns. These differences affect retrieval quality, snippets, citations, chunking, query expansion, and context-pack evidence.
- Public language adapter contract for:
- normalization;
- tokenization;
- lexeme/lemma/wordform analysis;
- morphology generation;
- query expansion;
- ambiguity reporting;
- adapter capability reporting.
context-lang-testkitwith shared contract tests.- Adapter lifecycle states:
experimental,supported,deprecated,retired. - Versioned analyzer, dictionary, normalizer, tokenizer, generator, and
feature-schema metadata in
IndexSnapshot, retrieval traces, and proof artifacts. - Morphology ambiguity resolution with explicit candidate lists and selected analysis reason.
- German compound splitting and separable-verb handling.
- Spanish and French accent, elision, contraction, agreement, and clitic handling.
- Hindi Devanagari normalization, postposition/oblique-case handling, gender and number agreement, and compound-verb support.
- Indic script normalization, transliteration boundaries, segmentation, and dictionary-resource policy.
- Language-specific golden eval sets for lemma recall, inflection coverage, expansion precision, false expansion rate, snippet stability, and citation span accuracy.
- A language adapter can be released, upgraded, deprecated, and tested without changing core domain models.
- Adapter output preserves source offsets and snippet/citation provenance.
- Ambiguity is explicit and traceable; no adapter silently collapses competing analyses.
- Query expansion explains every generated term and can be disabled by
FocusProfile. - Re-indexing with a new adapter or dictionary version produces a new
IndexSnapshotinstead of mutating past evidence.
Lexemes and morphology are not enough for real context management. Dictionaries, historical corpora, regional vocabularies, slang, scientific terminology, legal terms, and community lexicons need evidence about meaning, source, region, period, register, and authority. The same wordform can point to different senses, concepts, and usage traditions.
- Neutral contracts for
Sense,Concept,Attestation,Variant,MultiwordExpression,Register,DialectRegion,TimePeriod, andLexiconSource. - TEI dictionary import/export research path for entries, senses, examples, citations, etymology, and historical dictionaries.
- SKOS/ISO 25964 compatibility path for thesauri, controlled vocabularies, concept schemes, labels, and mapping relations.
- Historical dictionary support with spelling variants, edition/version metadata, and diachronic search.
- Regional, dialect, community, slang, and ethnogroup lexicon support.
- Source-authority scoring for dictionary, corpus, thesaurus, glossary, and community vocabulary resources.
- Attestation storage with original quote, source span, date, region, register, confidence, and source authority.
- Sense disambiguation and concept graph integration.
- Licensing and resource governance for dictionaries, corpora, thesauri, glossaries, and community resources.
- Import provenance so generated lexicon records can always be traced to a source file, citation, API response, or manual curation event.
- Original source text is never replaced by normalized or canonical text.
- Senses are not collapsed into lemmas.
- Concepts are not collapsed into labels.
- Every attestation has a source span, quote, and source authority metadata.
- Historical, regional, register, and source-authority filters are traceable in
retrieval and
ContextPackevidence. - TEI/SKOS/dictionary adapters can be added without changing core domain models.
- Dictionary and corpus licenses are recorded before resources are used in indexing, retrieval, or training data generation.
Logs, messages, device telemetry, scientific observations, accessibility input, and human-interaction streams are all time-oriented source corpora. Their schemas differ, but they need the same ingestion guarantees. Treating them as runtime trace events would mix source truth with engine operations and make replay ambiguous.
Plan Chunk 08A defines the minimal core contracts. This layer covers production-grade adapter lifecycle and scale after the proof loop.
- Adapter capability descriptor for event identity, ordering, late events, temporal precision, schema evolution, and checkpoint support.
- Stable event ids and idempotent ingest.
- Separate
occurred_at,observed_at(when applicable), andingested_at. - Deterministic event-window chunking and checksums.
- Source schema id, producer id/version, and migration metadata.
- Late/out-of-order event handling without rewriting historical source bytes.
- Cursor/checkpoint adapters for pull sources and offset adapters for streams.
- Generic temporal retrieval filters and event-window explain data.
- Derived artifact lineage from multiple event/source inputs.
- Contract tests shared by telemetry, message, audit-log, and scientific observation adapters.
- Re-ingesting the same stable events does not duplicate source truth.
- A late event produces a deterministic new source/snapshot version.
- Temporal retrieval explains which event-time window was applied.
- Derived aggregates retain source/artifact lineage after the creating
AgentRunis no longer loaded. - Runtime
tracing.Eventremains reserved for engine execution. - Device, person, reaction, capability, clinical, and product-session schemas remain outside the core.
High-quality context management improves through traces: which evidence was selected, ignored, opened, rejected, verified, or corrected. This feedback is the foundation for better reranking and future custom retrieval models.
These are feedback about runtime/retrieval outcomes. They do not replace adapter-owned source observation events from Layer 05C.
- Feedback events:
- result opened
- result ignored
- context pack accepted
- context pack rejected
- verifier failure
- tool result used
- user correction
- generated spec accepted
- Offline eval dataset generation from traces.
- Retrieval-quality dashboards.
- Reranker training export.
- Embedding training export.
- Privacy-aware trace retention.
- Every important user/agent feedback signal is captured as an event.
- Eval datasets can be generated without exposing raw private content by default.
- Feedback can improve retrieval without changing source truth.
S4 / C9: in-core edge store forever-deferred — ADR-0040. Claim graphs remain consumer projections until a measured blocker reopens core.
Scientific, legal, linguistic, and operational corpora contain claims that can conflict, expire, or depend on source authority. A plain vector index cannot represent this.
- Claim extraction model.
- Claim-to-source graph.
- Citation graph.
- Contradiction edges.
- Supersession edges.
- Version/edition awareness.
- Trust scoring by source type and verification status.
- Claim verifier for generated summaries and specs.
- "Evidence says" vs "model infers" separation.
- The engine can represent two conflicting sources without flattening them into one answer.
- Generated factual claims can be traced to supporting or conflicting evidence.
- Superseded sources are visible but down-ranked or flagged.
Background indexing, crawling, agent runs, verification, and monitoring need a real distributed execution model before production scale.
- Job queue adapter.
- Leases and heartbeats.
- Idempotency keys.
- Retry policy.
- Dead-letter queue.
- Cancellation.
- Timeout budgets.
- Resource quotas.
- Worker capabilities.
- Shard ownership for large projects.
- Backpressure when model, vector, metadata, or artifact stores are degraded.
- A worker crash does not corrupt an index or agent run.
- Duplicate jobs do not produce duplicate side effects.
- Long-running jobs can be cancelled and inspected.
Distributed agents become dangerous when tools can affect external systems. Permission labels are not enough; the runtime needs side-effect semantics.
- Tool side-effect classes:
- read-only
- local write
- reversible write
- irreversible write
- external network
- billing-affecting
- user-visible
- credential-affecting
- admin-affecting
- Dry-run mode.
- Compensation/rollback hooks.
- Approval checkpoints.
- Network allowlists.
- Filesystem sandbox policy.
- Environment-variable and secret access policy.
- Tool causal graph.
- The runtime can explain what side effects a run may perform before it starts.
- Risky tools cannot run without policy approval.
- Tool outputs cannot silently escalate permissions.
Production reliability needs explicit budgets. Without SLOs, every optimization and incident response becomes subjective.
- SLOs for:
- ingest latency
- indexing freshness
- search p50/p95/p99
- context pack build latency
- agent startup latency
- background job completion
- VectorStore availability
- metadata store availability
- artifact store availability
- RPO/RTO targets.
- Capacity model by documents, chunks, embeddings, projects, tenants, and background jobs.
- Load tests.
- Soak tests.
- Cost model.
- The team can say whether the system is healthy.
- Performance regressions are measurable.
- Scaling decisions are driven by observed bottlenecks.
Web search and crawling introduce legal, operational, and security risks. Broad crawling should not be part of the first proof.
- Explicit URL capture before broad crawling.
- robots.txt policy.
- Rate limits.
- Per-host budgets.
- User-agent policy.
- Content-type allowlist.
- Maximum page size.
- Redirect limits.
- Archive snapshots with timestamps.
- Malware and script stripping.
- Source trust classification.
- External content cannot overwhelm the system.
- Web captures are reproducible and source-stamped.
- Crawling policy is auditable.
Private corpora often include PDFs, DOCX, scans, spreadsheets, archives, email, and images. These formats should be adapters, not core assumptions.
- PDF text extraction.
- DOCX extraction.
- Spreadsheet extraction.
- Email and mailbox extraction.
- Archive traversal with depth limits.
- OCR adapter.
- Table extraction.
- Figure/image metadata extraction.
- Binary fingerprinting.
- Extraction confidence scores.
- Extracted text has provenance back to file, page, section, table, or image.
- Low-confidence extraction is flagged.
- Binary parsers cannot escape sandbox/resource limits.
Context systems often store sensitive source material, traces, prompts, model outputs, and tool results. Retention must be explicit.
- Retention policies per artifact type.
- Encryption-at-rest adapter support.
- Key management boundary.
- PII and secret detection.
- Redacted traces for support.
- Raw trace access policy.
- Export/delete project data.
- Audit trail for data access.
- Sensitive data has retention and access policy.
- Support/debug views do not expose raw secrets by default.
- Project deletion removes or tombstones all expected data.
Single-user local PoC does not need governance. Teams and hosted deployments do.
- Tenant model.
- Team/project roles.
- Service accounts.
- Team-level policies.
- Required rules.
- Shared adapters.
- Shared model/provider budgets.
- Cross-project search policy.
- Admin audit views.
- Team policies can restrict tools, models, sources, and background runs.
- Project-level settings cannot weaken enforced tenant policy.
- Cross-project retrieval is explicit and audited.
The core should eventually support downstream products, custom companions, tools, and adapters without forks.
- Stable
pkg/contextkitAPI. - Dockerized
context-coreservice for BFF/API consumers. - HTTP/gRPC API contract for ingest, search, context pack, agent run, trace, and snapshot inspection.
- Thin client SDKs generated from or aligned with the service contract.
- Adapter contract tests.
- Tool SDK.
- Companion configuration format.
- Lab-facing JSON/DTO compatibility tests using proof artifacts from
.proofs/. - Scenario plugin contracts for source adapters, graph projections, tool packs, rule packs, and methodology packs.
- DSL schemas for FocusProfile, RetrievalPlan, ContextPackTemplate, ToolPolicy, SourceAdapterConfig, and AgentRunPolicy.
- Migration policy.
- Semantic versioning policy.
- Deprecation policy.
- Example neutral applications.
- Third-party adapters can be tested without private internals.
- Public APIs are versioned and documented.
- Breaking changes are intentional and announced.
- A downstream BFF or lab shell can consume the service/SDK without importing internal packages.
The CLI proves the engine. A browser lab proves whether humans and product BFFs can understand and operate it. This is a downstream concern, but it should feed neutral contract improvements back into the core.
- UX fixture screens for project corpus, source list, search results, snippets, FocusProfile, ContextPack, AgentRun, and trace timeline.
- DX dashboard for PostgreSQL/pgvector, optional VectorStore backends, optional sparse backends, active snapshot, source/chunk counts, language adapter versions, and integration test status.
- BFF adapter that calls the Context service or consumes
context-devJSON during local development. - DSL workbench for editing/visualizing FocusProfile, RetrievalPlan, ContextPackTemplate, ToolPolicy, SourceAdapterConfig, and AgentRunPolicy.
- Contract tests proving Lab fixture JSON and Context CLI/API JSON stay compatible.
- Lab can demonstrate ingest -> search -> context pack -> fake agent -> trace without importing Context internals.
- Any Lab-discovered requirement is translated into a neutral Context contract or explicitly kept in Lab.
- UX/DX/DSL screens do not change core semantics without an ADR.
These gates should be required before a hosted or paid system depends on the engine.
- Threat model reviewed.
- Prompt-injection tests exist.
- Cross-project leakage tests pass.
- Secret redaction tests pass.
- Tool permission tests pass.
- Index rebuild tested.
- Store outage tested.
- Worker crash tested.
- Agent cancellation tested.
- Backup/restore tested.
- Golden retrieval suite passes.
- Citation accuracy threshold defined.
- Unsupported-claim rate tracked.
- Ranking regressions visible.
- SLOs defined.
- Metrics emitted.
- Logs structured.
- Traces replayable.
- Runbooks written.
Recommended order after the current proof loop:
The order is capability-based, not a license to defer workload safety. For any product ingesting sensitive human observation or assistive data, fine-grained ACL plus privacy/encryption/retention move ahead of real-user ingestion.
- Threat model and prompt-injection fixtures.
- Fine-grained retrieval ACL.
- Index lifecycle management.
- Snapshot replication and index reuse hardening.
- Focus control and memory tiers.
- Query language and snippet/highlighting engine.
- Event/observation source adapter lifecycle when a time-oriented consumer is selected.
- Multilingual linguistic adapter lifecycle.
- Lexicon resource and attestation management.
- Relevance feedback events.
- Distributed job control.
- Tool sandbox and side-effect model.
- Operational SLOs and capacity tests.
- Claim/contradiction graph for scientific/legal corpora.
- Crawler governance.
- Privacy/encryption/retention (or immediately after ACL for sensitive workloads).
- Multi-tenant/team governance.
- Binary/multi-modal adapters.
- Stable SDK and ecosystem contracts.
- Lab-driven UX/DX/DSL workbench hardening.
- Broad web crawling.
- Cross-user index reuse.
- Simhash copy-on-write seeding and Merkle content proofs.
- Incremental sparse segment sync beyond simple snapshot export/import.
- Custom embedding-model training.
- Multi-tenant billing.
- Full sandboxed code execution.
- Marketplace/plugin ecosystem.
- Dockerized
context-coreservice and public SDKs before CLI contracts settle. - Lab-specific UX, widgets, BFF routes, or DSL screens inside Context core.
- Production-grade Russian, German, Spanish, French, Hindi, or Indic morphology engines inside Context core.
- Language-specific dictionaries, grammar rules, and broad lexicons inside the neutral core.
- TEI/SKOS importers, historical dictionaries, regional vocabularies, slang lexicons, or community lexicon resources inside the first PoC.
- Messaging catalogs, calendar/Gantt products, CRM workflows, dashboards, or methodology runtimes inside the neutral core.
- Distributed worker orchestration.
- Complex binary document extraction.
- Production-grade query language.
- Claim contradiction graph.
The first PoC should remain small enough to debug line by line. These future layers should guide interfaces and tests, not inflate the first implementation.