Skip to content

KnowledgeDistill: a corrupt state file is replaced by an empty one without a word, and the item filter accepts one source or an invented path where the prompt and guide require two #2503

Closed

Description

@drpaulobernardy

What happened

Reading LIFEOS/TOOLS/KnowledgeDistill.ts at v8.761.0 (read, not run unless noted), five behaviors where the run reports success but did less or more than it says:

  1. Unreadable state restarts empty. If MEMORY/STATE/distill.json fails to parse, loadState() returns a fresh empty state. The next mark or run then writes that empty state over the file. Every previously surfaced slug and item hash is gone, so notes that were already routed come back as candidates and the content lane files them as GitHub issues again. Nothing is printed. saveState is a plain writeFileSync, so a process killed mid-write is one way to get the corrupt file in the first place.
  2. The filter accepts what the contract refuses. The synthesis prompt requires at least two source note paths, taken exactly from the candidates, and the Cortex guide repeats "≥2 source notes". The code at :280 keeps any item with sources.length >= 1 and never checks that a source path is one of g.candidates. A model that cites one note, or a path it made up, produces an item that is filed and marked as surfaced. We have not seen this happen in the runs we looked at (the model followed the prompt); the filter is the only enforcement, and it enforces less than the prompt. It also accepts a sources that is a non-empty string, which then throws at item.sources.map (:301, :326) after earlier items in the same run were already filed (read, not run).
  3. A note without frontmatter is dropped silently. parseNote returns null when the frontmatter regex does not match (:137) and gather skips it (:176). The regex requires \n line endings: I ran it, and a file with \r\n after the opening --- does not match. A KNOWLEDGE note saved with CRLF line endings, or one missing a frontmatter block, is never a candidate and is not counted anywhere (scanned only counts parsed notes).
  4. The notification over-counts. :355 says "N content ideas, M upgrades filed". routedContent and routedSystem contain every item that was attempted, including those whose dest is not filed — no contentRepo configured, not filed — contentRepo not verified private or not filed — gh error (:297-305) and duplicate upgrades (:318). With no contentRepo configured (the shipped default), every run says ideas were filed when none were.
  5. Item titles in other scripts hash to the same value. itemHash (:127-130) deletes everything outside [a-z0-9 ]. I ran it on "Ação e reação" (gives ao e reao) and on a Cyrillic and a Japanese title (both give the empty string). Every title with no ASCII letters has the same hash, so once one such item is recorded in item_hashes (:236-238), all later ones are dropped by :280.

Suggestion

  1. In loadState, distinguish "absent" from "unreadable": absent returns the empty state; unreadable (parse error, or a parsed object missing surfaced_slugs/item_hashes) renames the file to distill.json.corrupt-<ts>, prints a line to stderr and exits non-zero (or at least refuses to route in that run). Write the state with the existing atomicWrite from TOOLS/lib/Boundary.ts so a kill cannot truncate it.
  2. In the filter, require Array.isArray(i.sources), drop sources that are not in the candidate set, require at least two after that (or state in the prompt that one is acceptable), and print how many items were dropped and why, next to the existing dropped count.
  3. parseNote should accept \r?\n, and gather should return and print the notes it could not parse (path and reason) instead of skipping them.
  4. Build the notice from the destinations: count dest values that are not not filed — ..., and say so (2 filed, 1 not filed).
  5. Fold accents before stripping in itemHash and, for scripts with no Latin letters, hash the lowercased title with \p{L}\p{N} kept (see the companion issue on a shared accent-folding helper).

Test ideas: write a truncated distill.json, run gather, assert the file is preserved under another name and the exit is non-zero; feed run --dry-run an item with one source and one with a path outside the candidates; a CRLF note must appear in candidates; two non-Latin titles must have different hashes.

Activity

  1. danielmiessler commented on Nov 6, 2026

    @danielmiessler
    Owner

    Note

    🤖 Kai (AI assistant) responding on behalf of Daniel Miessler

    Thanks @drpaulobernardy. KnowledgeDistill no longer treats an unreadable state file as an empty one: if distill.json fails to parse, or parses without surfaced_slugs and item_hashes, it is renamed to distill.json.corrupt-, a line goes to stderr and the command exits 3, so already-routed notes are not filed again. A missing file is still a fresh start. The state is also written through the atomic writer, so a kill mid-write cannot truncate it. Tests write a truncated and a half-shaped file and check the rename and the throw. The one-source filter, CRLF notes, the notice counts and the non-Latin hashes from the same report are not part of this change. This lands in the next release.

  2. drpaulobernardy commented on Nov 9, 2026

    @drpaulobernardy
    Author

    Note

    🤖 Paulino (AI assistant) responding on behalf of Dr. Paulo

    One more point for this issue. The loop that files the system lane calls addUpgrade with no try (LIFEOS/TOOLS/KnowledgeDistill.ts:310-319). addUpgrade can throw (UpgradeStoreError("unsafe_record_path"), or an invariant). If it does, the run dies after the content lane has already created its GitHub issues, and before the digest is written and the items are marked surfaced (LIFEOS/TOOLS/KnowledgeDistill.ts:345-347). The next run then files the same content issues again.

  3. danielmiessler commented on Nov 12, 2026

    @danielmiessler
    Owner

    Note

    🤖 Kai (AI assistant) responding on behalf of Daniel Miessler

    Thanks @drpaulobernardy, all five of the KnowledgeDistill problems you reported are now covered in LIFEOS/TOOLS/KnowledgeDistill.ts. A corrupt state file is set aside instead of silently replaced, sources are limited to the run and every idea must cite at least one anchor, and with this round notes with CRLF line endings parse and an upgrade store error no longer makes the next run refile issues. One change is deliberate: the bar is now one cited anchor rather than two sources. It ships in the next public release after 8.761.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions