feat: Add authority and traceability opt-ins (COG-6279) - #4628
feat: Add authority and traceability opt-ins (COG-6279)#4628Vasilije1990 wants to merge 4 commits into
Conversation
Description: Local file paths are ingested by reference — cognee records the path and content_hash, which can verify the original but not recover it once the file moves or is deleted. An external audit flagged that a returned chunk can therefore not always be traced back to source bytes. COPY_LOCAL_FILES=true (default false, default path byte-identical) snapshots local-path originals content-addressed into cognee-managed storage at add time via the existing save_data_to_file_detailed machinery, so original_data_location stays resolvable and hash-verifiable after the source disappears. s3:// inputs remain pass-through by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Description: The ontology pass has always been annotation-only — matched entities are canonicalized and enriched, unmatched entities are retained verbatim, and nothing is ever rejected. An external audit asked whether a strict validation mode exists; it did not. ONTOLOGY_MODE=strict (default annotate, default pipeline byte-identical) drops extracted entities with neither a class match on their type nor an individual match on their name, filters edges touching dropped nodes, and logs the dropped counts. Scope is entity grounding only: no domain/range/ cardinality/disjointness reasoning, and relationship names are not checked against the ontology. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Description: An external audit found that no stored classification distinguishes a user statement from an LLM-inferred fact, that no conflict path consults provenance, and that superseded edges tagged by the opt-in functional_relationships resolver remain fully visible to every retriever. This adds one coherent supersession lifecycle: - assertion_source (user_stated / document_extracted / llm_inferred) is stamped on DataPoints under the existing COGNEE_PROVENANCE_MODE gate (user_stated for session-bridged facts), copied onto edge properties, and written as llm_inferred by cross_connect_entities and detect_contradictions. - The temporal conflict resolver ranks authority-then-recency, so an older user-stated fact beats a newer LLM-inferred one; untagged edges rank as document_extracted and resolve byte-identically to before (locked by a regression test). - Graph retrieval skips edges tagged superseded=True at both choke points (CogneeGraph projection and the hybrid entity lane). Safe by construction: the tag exists only for callers who opted into functional_relationships. contradicts edges stay visible; triplet embeddings and raw chunk text are unfiltered (documented). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
## Description Stacked on #4628 — documentation follow-up to the audit-response features. Retarget to `dev` (or let GitHub retarget automatically) once #4628 merges. An external audit validated that cognee's docs overclaimed in places and left the temporal/authority model undocumented. This PR makes the documentation state the boundaries precisely: - **CLAUDE.md**: new *Temporal Model & Assertion Authority* subsection — uni-temporal default (ingestion clock), opt-in event time via `temporal_cognify`, authority-then-recency supersession with retrieval-side filtering of `superseded` edges, dormant `valid_to`/Zep `valid_at` fields — plus the `ONTOLOGY_MODE` line in the ontology config block. - **`DataPoint.valid_to` comment**: states that no default pipeline writes it (`close_node` has no in-pipeline caller). - **`close_node.py` docstring**: application-level helper only; requires a backend with partial node updates (currently Ladybug). No behavior change; documentation and comments only. ## Test plan - [x] `pre-commit run --all-files` clean - [x] Comments/docs only — no production code touched Linear: COG-6279 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
CI didn't catch it because Two other things. Minor: Also worth correcting the PR description — |
Description
An external audit (right-to-reply issue on the article "your memory has no authority model", audited commit
fd5045f6) validated four gaps in cognee's authority model. This PR is the minimal code response — three features, each default-off or default-byte-identical, so the standard pipelines are unchanged unless opted into.1.
COPY_LOCAL_FILES— original-bytes snapshot (defaultfalse)Local file paths have always been ingested by reference: cognee records the path and
content_hash, which can verify the original but not recover it once the file moves or is deleted. With the flag on, local-path originals are snapshotted content-addressed into cognee-managed storage at add time via the existingsave_data_to_file_detailedmachinery, sooriginal_data_locationstays resolvable and hash-verifiable.s3://inputs remain pass-through by design.2.
ONTOLOGY_MODE=strict— entity-grounding validation (defaultannotate)The ontology pass has always been annotation-only: matched entities are canonicalized and enriched, unmatched entities retained verbatim. Strict mode drops extracted entities with neither a class match on their type nor an individual match on their name, filters edges touching dropped nodes, and logs the dropped counts. Scope is entity grounding only — no domain/range/cardinality/disjointness reasoning, and relationship names are not checked against the ontology (documented in the docstrings and
.env.template).3. Assertion authority + superseded-edge filtering
DataPoint.assertion_source(user_stated/document_extracted/llm_inferred) is stamped under the existingCOGNEE_PROVENANCE_MODEgate (defaultlightweight):user_statedfor session-bridged facts,document_extractedotherwise;cross_connect_entitiesanddetect_contradictionswritellm_inferred. Copied onto edge properties so the conflict resolver can see it.functional_relationshipstemporal conflict resolver now ranks authority first, then recency — an older user-stated fact beats a newer LLM inference. Untagged/pre-existing edges rank asdocument_extractedand resolve byte-identically to today (locked by a regression test).superseded=Trueat both choke points (CogneeGraphprojection and the hybrid entity lane). Safe by construction: the tag only ever exists for callers who opted intofunctional_relationships.contradictsedges stay visible; precomputed triplet embeddings and raw chunk text are unfiltered (documented).Test plan
os.remove(source)with md5 ==content_hashpre-commit run --all-filescleanLinear: COG-6279
🤖 Generated with Claude Code