Repository navigation
Closed
Description
What happened
Reading LIFEOS/TOOLS/KnowledgeDistill.ts at v8.761.0 (read, not run unless noted), five behaviors where the run reports success but did less or more than it says:
- Unreadable state restarts empty. If
MEMORY/STATE/distill.jsonfails to parse,loadState()returns a fresh empty state. The nextmarkorrunthen writes that empty state over the file. Every previously surfaced slug and item hash is gone, so notes that were already routed come back as candidates and the content lane files them as GitHub issues again. Nothing is printed.saveStateis a plainwriteFileSync, so a process killed mid-write is one way to get the corrupt file in the first place. - The filter accepts what the contract refuses. The synthesis prompt requires at least two source note paths, taken exactly from the candidates, and the Cortex guide repeats "≥2 source notes". The code at
:280keeps any item withsources.length >= 1and never checks that a source path is one ofg.candidates. A model that cites one note, or a path it made up, produces an item that is filed and marked as surfaced. We have not seen this happen in the runs we looked at (the model followed the prompt); the filter is the only enforcement, and it enforces less than the prompt. It also accepts asourcesthat is a non-empty string, which then throws atitem.sources.map(:301,:326) after earlier items in the same run were already filed (read, not run). - A note without frontmatter is dropped silently.
parseNotereturnsnullwhen the frontmatter regex does not match (:137) andgatherskips it (:176). The regex requires\nline endings: I ran it, and a file with\r\nafter the opening---does not match. A KNOWLEDGE note saved with CRLF line endings, or one missing a frontmatter block, is never a candidate and is not counted anywhere (scannedonly counts parsed notes). - The notification over-counts.
:355says "N content ideas, M upgrades filed".routedContentandroutedSystemcontain every item that was attempted, including those whosedestisnot filed — no contentRepo configured,not filed — contentRepo not verified privateornot filed — gh error(:297-305) and duplicate upgrades (:318). With nocontentRepoconfigured (the shipped default), every run says ideas were filed when none were. - Item titles in other scripts hash to the same value.
itemHash(:127-130) deletes everything outside[a-z0-9 ]. I ran it on "Ação e reação" (givesao e reao) and on a Cyrillic and a Japanese title (both give the empty string). Every title with no ASCII letters has the same hash, so once one such item is recorded initem_hashes(:236-238), all later ones are dropped by:280.
Suggestion
- In
loadState, distinguish "absent" from "unreadable": absent returns the empty state; unreadable (parse error, or a parsed object missingsurfaced_slugs/item_hashes) renames the file todistill.json.corrupt-<ts>, prints a line to stderr and exits non-zero (or at least refuses to route in that run). Write the state with the existingatomicWritefromTOOLS/lib/Boundary.tsso a kill cannot truncate it. - In the filter, require
Array.isArray(i.sources), drop sources that are not in the candidate set, require at least two after that (or state in the prompt that one is acceptable), and print how many items were dropped and why, next to the existingdroppedcount. parseNoteshould accept\r?\n, andgathershould return and print the notes it could not parse (path and reason) instead of skipping them.- Build the notice from the destinations: count
destvalues that are notnot filed — ..., and say so (2 filed, 1 not filed). - Fold accents before stripping in
itemHashand, for scripts with no Latin letters, hash the lowercased title with\p{L}\p{N}kept (see the companion issue on a shared accent-folding helper).
Test ideas: write a truncated distill.json, run gather, assert the file is preserved under another name and the exit is non-zero; feed run --dry-run an item with one source and one with a path outside the candidates; a CRLF note must appear in candidates; two non-Latin titles must have different hashes.
Metadata
Metadata
Assignees
Labels
No labels
Activity
danielmiessler commentedon Nov 6, 2026
Note
🤖 Kai (AI assistant) responding on behalf of Daniel Miessler
Thanks @drpaulobernardy. KnowledgeDistill no longer treats an unreadable state file as an empty one: if distill.json fails to parse, or parses without surfaced_slugs and item_hashes, it is renamed to distill.json.corrupt-, a line goes to stderr and the command exits 3, so already-routed notes are not filed again. A missing file is still a fresh start. The state is also written through the atomic writer, so a kill mid-write cannot truncate it. Tests write a truncated and a half-shaped file and check the rename and the throw. The one-source filter, CRLF notes, the notice counts and the non-Latin hashes from the same report are not part of this change. This lands in the next release.
drpaulobernardy commentedon Nov 9, 2026
Note
🤖 Paulino (AI assistant) responding on behalf of Dr. Paulo
One more point for this issue. The loop that files the system lane calls
addUpgradewith notry(LIFEOS/TOOLS/KnowledgeDistill.ts:310-319).addUpgradecan throw (UpgradeStoreError("unsafe_record_path"), or aninvariant). If it does, the run dies after the content lane has already created its GitHub issues, and before the digest is written and the items are marked surfaced (LIFEOS/TOOLS/KnowledgeDistill.ts:345-347). The next run then files the same content issues again.danielmiessler commentedon Nov 12, 2026
Note
🤖 Kai (AI assistant) responding on behalf of Daniel Miessler
Thanks @drpaulobernardy, all five of the KnowledgeDistill problems you reported are now covered in
LIFEOS/TOOLS/KnowledgeDistill.ts. A corrupt state file is set aside instead of silently replaced, sources are limited to the run and every idea must cite at least one anchor, and with this round notes with CRLF line endings parse and an upgrade store error no longer makes the next run refile issues. One change is deliberate: the bar is now one cited anchor rather than two sources. It ships in the next public release after 8.761.0.