feat: Presort folder pre-organization + JSON graph-model DSL (COG-6281) - #4630
feat: Presort folder pre-organization + JSON graph-model DSL (COG-6281)#4630Vasilije1990 wants to merge 3 commits into
Conversation
Graph models can now be declared in plain JSON — entities, typed fields, and relations with cardinality — instead of hand-written Pydantic classes. GraphSchemaSpec validates the document (also the safety gate before the exec-based converter) and compiles through the existing graph_schema_to_graph_model. Mirrors the frontend editor's shape (camelCase aliases), and adds identity_fields so DSL-built nodes merge across chunks and runs, which the frontend compiler never emits. Exported via cognee.low_level to keep the main SDK surface unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ingesting a messy folder today flattens it blindly: no junk filtering, no duplicate or personal-data awareness, one dataset for everything. Presort adds a read-only pre-step that scans a folder and produces a PresortReport: junk skipped, exact-duplicate clusters (lazy hashing), version candidates, potential personal data (redacted samples only), per-file already-in-cognee status derived from content hashes and pipeline_status, and proposed dataset groupings. Applying the report ingests each group through the normal pipeline with incremental loading, so re-applying is idempotent; without an LLM key it degrades to add()-only staging with a warning instead of failing. The relationships presort checks for are data, not code: a JSON graph-model spec drives which sections exist, custom relations resolve via registered detectors or an LLM fallback, and apply_graph writes the report's spec-shaped relationship graph into its own dataset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Presort rides remember() rather than adding a new SDK function: dry_run="presort" analyzes a folder, passing the report (object, dict, or saved *.presort.json) back applies it, and auto_apply=True does both in one call. Plain local-directory inputs presort automatically when no explicit dataset, session, or content_type was given — code-project directories keep the repo route, and PRESORT_FOLDERS_ENABLED=false disables the routing. add() gains a skip_connection_test passthrough so the LLM-free staging path does not trip the first-run connection probe. CLI: cognee-cli remember grows --dry-run presort, --apply, --apply-graph, --from-report, --allow-root and related flags. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| if text.startswith(("s3://", "http://", "https://", "file://")): | ||
| return False | ||
| try: | ||
| path = Path(text).expanduser() |
| if looks_like_presort_report(data): | ||
| return PresortReport.from_json(data) if isinstance(data, dict) else data | ||
| if isinstance(data, (str, Path)) and str(data).endswith(REPORT_FILE_SUFFIX): | ||
| candidate = Path(data).expanduser() |
| if dry_run is False and _should_auto_presort( | ||
| data, dataset_name, dataset_id, session_id, kwargs | ||
| ): | ||
| logger.info(f"remember: folder input detected — presorting {str(data)!r} automatically") |
| except ValueError: | ||
| raise ValueError( | ||
| f"Path {data_path!r} is outside the allowed local file roots. Set " | ||
| f"{ALLOWED_LOCAL_FILE_ROOTS_ENV}={Path(data_path).expanduser()} " |
| text = str(source) | ||
| candidate = Path(text) | ||
| try: | ||
| if candidate.is_file(): |
| candidate = Path(text) | ||
| try: | ||
| if candidate.is_file(): | ||
| text = candidate.read_text(encoding="utf-8") |
|
The part I'd want changed before this merges is that presort isn't opt-in. if dry_run is False and _should_auto_presort(data, dataset_name, dataset_id, session_id, kwargs):
dry_run = "presort"
kwargs.setdefault("auto_apply", True)Today Second thing: the new tests fail deterministically on 3.10 — 12 failures in "OS and Python Tests Ubuntu / Unit tests 3.10.x", and 3.10 is a supported version per Three smaller ones while you're in there. A folder of only dotfiles or The DSL and the test volume are good — my objection is entirely about the default and the 3.10 gate. |
3 participants
Description
Pointing cognee at a messy folder (e.g.
~/Downloads) today flattens it blindly into one dataset — no junk filtering, no duplicate or personal-data awareness, no reuse of what's already been cognified. This PR adds presort, a pre-ingestion organization step, plus a JSON graph-model DSL that both presort and end users can define relationship schemas with.Fixes COG-6281.
Presort — rides
remember(), no new top-level SDK functionAnalyze —
remember(folder, dry_run="presort")scans without touching files and returns a persistedPresortReport:.DS_Store, partial downloads, hidden/build dirs)report (1),_v2,-final, trailing dates)new/staged/cognified) from content hashes joined withpipeline_status, so nothing is assumed freshApply —
remember(report)(object, dict, or saved*.presort.json) ingests each proposed group through add→cognify→improve withincremental_loading=True(idempotent re-apply), honoring the report'sskip_duplicates/exclude_pii/apply_groups.auto_apply=Truedoes both phases in one call; outcomes ride back onreport.apply_results.Automatic for folders — a plain local directory remembered with no explicit dataset/session/content_type presorts automatically; code-project directories keep the repo route;
PRESORT_FOLDERS_ENABLED=falsedisables.Degrades without an LLM key — deterministic scan always runs,
use_llmdowngrades with a warning, apply stages withadd()only (newskip_connection_testpassthrough onadd()),apply_graphskipped with a warning.CLI —
cognee-cli remember <folder> --dry-run presort [--apply] [--apply-graph] [--use-llm] [--allow-root] [-o report.json]and--from-report report.jsonto apply later.--allow-rootappends toCOGNEE_ALLOWED_LOCAL_FILE_ROOTS(never replaces).JSON graph-model DSL (
cognee.modules.graph_models, exported viacognee.low_level)GraphSchemaSpec— entities, typed fields, relations with cardinality; snake_case with camelCase aliases so the frontend editor's JSON works unchanged. Validation is also the safety gate before the exec-based converter (identifier-only names, infra-field collision checks, size caps).graph_spec_to_json_schema— Python port of the frontendtoGraphModelSchema.ts, extended withidentity_fields(default["name"]) so DSL-built nodes actually merge across chunks/runs.graph_model_from_spec— JSON in, DataPoint-derived Pydantic class out, via the existinggraph_schema_to_graph_model.duplicate_of,version_of,belongs_to_group,contains_pii) is a DSL document; report sections follow the spec, custom relations resolve viaregister_relation_detector()or an LLM fallback, andapply_graph=Truewrites the spec-shaped relationship graph into its own dataset.Testing
staged/cognifiedand skip re-processing.New docs-facing pieces:
examples/guides/presort_downloads.py,examples/guides/graph_model_from_json.py, promptsdetect_pii.txt/extract_presort_relation.txt.🤖 Generated with Claude Code