Ingest uploaded documents and hosted-doc links as documentable sources

[baonguyenNava]

Issue readiness state

• Reviewed by human engineer
• Issue refinement complete
• All open questions answered
• Post-refinement changes reviewed by human
• Ready for work

Goal

Today the engine can only document git repositories. Real Strata knowledge also lives outside code: design decisions, runbooks, and hand-written guides that sit in local documents or in Google Docs and Confluence pages. We want a person to be able to point the engine at one of those documents, or at a link to one, and have it documented, indexed, and cross-linked into the graph the same way a source repository is.

Decisions (from review)

• Acquired source content is written under .sources/<id>/ and is gitignored, never committed; generated docs live under docs/sources/<id>/. .sources/ stays runtime-only per existing convention.
• Each input family gets its own manifest type value and its own profile under references/profiles/: gdoc, confluence, local-doc. This is not a single shared document type.
• A shared internal acquire(source) -> (canonical_text, upstream_id) seam sits in the skill layer with per-family adapters for Google Drive MCP, Atlassian Rovo MCP, and local filesystem. Normalization, hashing, and the failure signal live once in the seam. This seam is an implementation detail and does not collapse the per-family types or profiles.
• Acquisition uses MCP servers, Google Drive and Atlassian Rovo, under the operator's local auth. MCP and an SDK adapter behind the same seam are both sanctioned mechanisms; the SDK adapter is the fallback for a family where no MCP server fits, for example local-fs.
• Acquisition and hashing live in the skill layer; the deterministic Python scripts stay network- and auth-free. compute_delta (source_delta) and lint_docs require no change for the happy path. The content hash rides the existing {id: sha} map passed via --shas, and lint_docs already tolerates a non-git source_ref.repo since it only checks the field is non-empty. Regression tests prove this.
• lint_manifest's git-URL and non-empty-ref checks become conditional on source type. Git-type sources still hard-fail on an empty ref or a non-git repo. Non-git types (gdoc, confluence, local-doc) skip the git-URL check and instead require a non-empty repo slot, a stable id or path, and a ref that is a 40-char hex hash.
• No new mid-table manifest columns. parse_manifest zips cells positionally against the fixed columns [id, type, repo, ref, subpaths, notes], so an inserted column would silently shift every existing row. The git-URL and git-ref semantics are dropped for non-git types and the existing slots are reused instead: the repo slot holds the stable upstream id, a Google fileId or Confluence pageId, or a local path; the ref slot holds a 40-char hex content hash; the human-facing URL goes in notes. Any genuinely new field is appended only after notes. No script renders source_ref.repo as a link, since INDEX and graph.json use the doc path, so repurposing the slot is safe; the doc-frontmatter schema wording widens repo: <git url> to allow an id or path per type.
• Source identity and drift: source_ref.ref holds a SHA-1 digest, exactly 40 lowercase hex chars, matched by source_delta's _SHA_RE.fullmatch, of the acquired document's canonical extracted body text, not raw export bytes. SHA-256 is disallowed since 64 chars would never compare equal and would classify the source "changed" forever. A changed hash re-documents; an unchanged hash classifies unchanged.
• The canonical extracted-text recipe, documented by the acquisition child: extract to a single text representation, normalize to LF newlines, strip per-line trailing whitespace, collapse trailing blank lines, and remove a documented deny-list of volatile nodes, comments, suggestions, and dynamic timestamp or macro content, before hashing. This deliberately narrows the original "any small change is picked up by a hash change": edits to excluded volatile nodes will not trip drift, and dynamic-macro content is accepted residual noise. This tradeoff needs ratification, carried into Open Questions below.
• No CI execution and no shared-secret or service-account auth in this epic; explicitly deferred. Runs use the invoking operator's own local Google or Confluence credentials only. A consequence: a hosted-doc-link source can only be acquired and re-documented by an operator with access, so runs are operator-dependent by design.

Success looks like

• Register a local document (markdown, docx, txt) or a Google Doc / Confluence link as a source in sources.md, without it being a git repository.
• The engine acquires the content into .sources/<id>/ (gitignored) and produces valid docs under docs/sources/<id>/ that pass lint_docs.
• Cross-linking: a prose doc earns graph edges only through reference fields (demonstrates, integrates_with, manages, related) resolved against the existing registries; it must not declare ownership fields (feature_keys, component_keys), which stay owned by the git-sourced docs, since build_graph is first-writer-wins and a prose doc declaring them would hijack ownership. Referenced keys must pre-exist in the registries or lint_docs hard-fails. There is no minimum edge count; a legitimately standalone doc may connect to nothing.
• Zero-edge visibility: build_graph gains an isolated-node report so any doc node with no inbound or outbound edge is surfaced in the run output, analogous to the existing feature_key_gaps and component_key_gaps reports, so a disconnected ingested doc is never silently indexed as a dead end. This is new code.
• Source identity and drift behave as decided above: a 40-hex SHA-1 of canonical text, a changed hash re-documents, and an unchanged hash classifies unchanged.
• Acquisition-failure signal: a failed acquisition, whether link unreachable, auth missing or expired, permission revoked, or source deleted, emits a signal distinct from changed. The skill omits the failed id from the --shas map, writes an ACQUISITION_FAILED <id> <reason> line to the run log, leaves the prior doc intact, does not re-document on that path, and the run exits non-zero. A failure is never classifiable as unchanged nor silently absorbed into ordinary changed re-documentation.
• Regression guardrail: existing git-repo sources are unaffected. The current sources.md still passes lint_manifest, lint_docs, and build_graph, their drift classification is identical to today, the manifest and any source_delta changes are backward-compatible, and the type-conditional lint_manifest change still hard-fails a git-type source with an empty ref or non-git repo.
• First dogfooding source: the epic names one concrete first source, a specific Google Doc registered as type gdoc (the exact doc or fileId to be named by the author), and success is demonstrated end-to-end: it acquires to .sources/<id>/, produces docs/sources/<id>/, appears in docs/INDEX.md and docs/graph.json, and the isolated-node report either shows a real edge or visibly lists it as zero-edge; a re-run with no content change classifies it unchanged.

Out of scope

• Arbitrary web-page URLs and PDF ingestion. This epic covers local files and Google Doc / Confluence links only.
• A web upload UI or endpoint. Ingestion is a sources.md row, consistent with how every existing source is registered.
• Rewriting or consolidating the existing Confluence content itself (tracked separately by Consolidate Strata documentation in Confluence #8).
• Browser-observed UI and design-pattern documentation (tracked separately by Spike browser-observed UI and design-pattern documentation #14).
• Rewriting, summarizing, or normalizing the acquired source prose. This epic acquires and documents content as-is; rewrites are tracked by Consolidate Strata documentation in Confluence #8.
• CI execution of hosted-source acquisition and any shared-secret or service-account auth. Deferred; local operator credentials only.
• Confluence support is single page per manifest row. Page trees are out of scope: a tree would fan one manifest row to N docs and break the one-id / one-ref / one-hash model, since it would need per-page rows or a child-doc scheme. Deferred to a follow-up.

Open questions

• Ratify the hash-scope narrowing: confirm that excluding volatile nodes, comments, suggestions, and dynamic timestamp or macro content, from the drift hash is acceptable, accepting that edits to those nodes will not trip re-documentation. Needs sign-off from @baonguyenNava.
• Local-file stable identity: local files have no fileId or pageId. Is the manifest id the author-assigned anchor, recommended, with the repo slot holding the path, or is identity derived from the path, where a move would then silently change identity?
• Per-family canonical-text normalization: does each profile own its own normalization recipe, or is there one shared extraction contract the acquire() seam enforces?
• Google Docs export format for hashing: canonicalize from a markdown export, which has more structure and more volatile syntax, or plain text?
• Confluence ADF node ordering: does the Atlassian Rovo MCP guarantee stable child-node order across fetches, or must the seam sort to keep the hash stable?
• Should acquisition-failed become a first-class source_delta classification for report visibility, or stay log-only plus non-zero exit?
• Name the concrete first dogfooding source, the specific Google Doc or fileId, for the acceptance demo.

Child issues

[baonguyenNava]

Feedback:

Yes, acquired content should be saved in.sources, and it is getting ignored.
At the moment, the manifest is entirely focused around using GitHub repos as a source. We need to no longer hard require a repo and a ref, as we should be able to document other strata sources that aren't in code.
We should store a new type and profile per input family.
For now, this should run using local authentication, so if I run this myself, then it will use my permissions to Google Docs and Confluence. Therefore, it should use an SDK or an MCP server of some sort to connect to these sources. For now, let's not consider how this would run in CI, where we would require a shared secret. Lastly, to compute the source identity and drift, we will have to build a 40-character shop. Let's see if we can build a content hash. For example, if we pull the information from a Google Doc, then the body of that doc, we should be able to compute a content hash and then save that as a shop. Any small changes should be picked up by a hash change.

[baonguyenNava]

Refinement summary

This issue was pressure-tested for sprint readiness (product-owner versus senior-engineer debate, reconciled against the review comment). The changes:

• The five original open questions became ten Decisions (from review): acquisition location under gitignored .sources/<id>/, per-family types and profiles (gdoc, confluence, local-doc, not a shared document type), the shared skill-layer acquire() seam with per-family adapters, MCP-based local auth (with an SDK adapter as a sanctioned fallback), keeping the deterministic Python scripts network- and auth-free, type-conditional lint_manifest, reusing the existing manifest slots rather than adding columns, SHA-1-of-canonical-text as the source identity, the canonical-text and volatile-node recipe, and explicitly deferred CI / shared-secret auth.
• Success looks like grew from four bullets to eight acceptance criteria, adding the cross-link ownership rules (prose docs use reference fields, never ownership fields), the build_graph isolated-node report, the acquisition-failure signal, a backward-compatibility guardrail for existing git sources, and a named first dogfooding source.
• Out of scope gained three bullets: no rewriting of acquired prose, no CI or shared-secret auth in this epic, and single-page-only Confluence support.
• Open Questions now lists seven genuinely unresolved items, including one that needs your sign-off: the drift hash deliberately excludes volatile nodes (comments, suggestions, dynamic timestamp or macro content), which narrows the original "any small change is picked up by a hash change". Please confirm that tradeoff.
• The child-issue decomposition was rewritten to match the accepted design.

"Issue refinement complete" is now ticked. "Post-refinement changes reviewed by human" is left unticked and is awaiting your review; please read these edits and tick it once you are satisfied.

[jeffhorn-nava]

Reviewed this independently and pressure-tested it against our aims and the effort×value lens. It's a careful, thorough spec. My feedback is about scope, one security gap, and two acceptance criteria that don't hold up against the current code. Framing these as questions for us to work through, not conclusions.

Scope

This is high effort (type-conditional lint_manifest, three profiles, the acquire() seam, two MCP integrations plus an SDK fallback, a per-family hashing recipe, a new build_graph report, a regression suite) while the engine's own value is still unproven. Under our effort×value rule that's the hard-and-uncertain quadrant. Rather than decompose the full three-family epic now, I'd like to prove both the value and the riskiest mechanism cheaply first (spikes proposed at the end).

Security gap

Acquired content becomes committed content. .sources/ is gitignored, but the generated docs under docs/sources/<id>/ are committed. A Google Doc or Confluence page can carry internal-only or client-sensitive material or PII that was never written expecting to land in a git repo. I don't see a classification or redaction step anywhere in the decisions. I think we need to close this before ingesting anything hosted, not defer it.

Two acceptance criteria that don't hold against the current code (I checked build_graph.py and source_delta.py)

• The isolated-node report can never fire as written. build_graph adds a source:<id> to doc "documents" edge for every doc unconditionally (build_graph.py:45-48), so no doc node ever has zero inbound edges. The criterion "any doc node with no inbound or outbound edge" is vacuous. If the intent is "connects to no other doc," the report needs to be defined against doc-to-doc edges and ignore the trivial source edge.
• The acquisition-failure signal produces the outcome it forbids. The decision says the skill omits the failed id from --shas, and that a failure must never be "silently absorbed into ordinary changed re-documentation." But compute_delta iterates the manifest, and an already-documented source with no resolved SHA falls straight into the "no resolved SHA provided; treating as changed" branch (source_delta.py:110-112), so it re-documents. Omitting from --shas gives exactly the forbidden result. This needs a real mechanism: a distinct classification, or excluding the failed source from what's passed to compute_delta.

Suggested next step: two cheap spikes before we decompose this

I'd rather de-risk this with two small, bounded spikes than break the epic into child issues now:

• A value spike: hand-pick one real Google Doc, acquire, hash, and document it manually without the lint_manifest / profile / registry formalization, and see whether the resulting doc earns useful graph edges and actually gets read. This tests whether the outcome is worth the build.
• A feasibility spike on the drift hash: prove we can compute a stable content hash for that same Google Doc across repeated fetches, with a defined canonical-text recipe. This is the least-settled part of the spec (four open normalization questions still ride on it: per-family vs shared, export format, ADF node ordering, hash scope), and everything else depends on it.

If both come back positive, we decompose with the value proven and the core mechanism de-risked. If the value spike is negative, we've spent a day or two instead of an epic. Given the two acceptance-criteria problems above as well, I don't think we should tick "ready for work" or decompose yet.

[baonguyenNava]

I agree on not marking it "ready for work"