Skip to content

Thinking record on runs, and a generated Fabric pattern catalog with a suggester - #2816

Closed
glitchking1337 wants to merge 1 commit into
danielmiessler:mainfrom
glitchking1337:thinking-record-and-fabric-catalog
Closed

glitchking1337 wants to merge 1 commit into
danielmiessler:mainfrom
glitchking1337:thinking-record-and-fabric-catalog

Conversation

@glitchking1337

Copy link
Copy Markdown
Contributor

Summary

Two additions, both records rather than gates:

1. The Thinking: line, a run's written thinking plan. PAI v2.5's THINK phase walked six thinking tools on every request and wrote INCLUDE or EXCLUDE with a reason. LifeOS v8.0 cut that table as scaffolding, and nothing afterwards recorded which thinking skills a run ran or skipped. This brings the record back the way the Lane: line did for dispatch:

  • ISAFormat: one optional Thinking: <Skill>·<light|full>[, …] ; skipped <Skill>=<phrase> or Thinking: none=<phrase> line per feature block, after Lane:. Additive; an ISA without it reads as before.
  • LIFEOS/ROUTER/ThinkingCheck.ts (new, 18 tests): line draws 🧠 THINKING ▸ F1 RedTeam·light (skipped Council) · F2 none; check --session compares the plan with the Skill tool calls in the session transcript and its subagent sidechains; fit asks one shadow Assay question per skipped skill (caller thinking-fit, fail-open) and says once when the engine is unavailable. Every subcommand exits 0.
  • HookBridge thinking and thinking-fit verbs; the LifeOSHooks plugin draws the line beside the route line (also when the ISA has no Lane: line).
  • Socrates thinking adapter (new, 5 tests): the weekly share of skips judged would-have-changed, unknown under 10 judged runs; one seed question in seeds/template.json.
  • Algorithm 8.46.0 § Spend states the rule: a run with an ISA writes the plan before building, and "too simple" is never the phrase.

2. A generated Fabric pattern catalog and a suggester. Upstream pattern files carry no metadata and are overwritten on every sync, so metadata lives in a generated sidecar, skills/Fabric/Patterns/catalog.yaml, never in a system.md:

  • skills/Fabric/Tools/PatternCatalog.ts (new, 10 tests): build derives entries from pattern_explanations.md and each pattern's IDENTITY and OUTPUT sections, with one cached model pass only for a pattern no explanation line covers; check fails on a missing entry or a moved hash and is wired into CheckPatterns.ts and the skill's deterministic evals; triggers regenerates the USE WHEN list on request (--write, opt-in).
  • skills/Fabric/Tools/SuggestPattern.ts (new, 9 tests): top three patterns for a plain request, trigger and keyword ranking with a partial-name penalty and a ported-pattern edge, then one shadow Assay question over the shortlist (caller pattern-suggest, fail-open, skipped whenever a local pattern is shortlisted). An explicitly named pattern or intent makes no model call.
  • ExecutePattern's keyword table is replaced by the catalog and the suggester; the pick is shown as Pattern: <name> (<reason>) before the pattern runs. UpdatePatterns rebuilds the catalog after every sync and in its re-scrub recovery. Local patterns get their own catalog under USER/CUSTOMIZATIONS/SKILLS/Fabric/, merged at read time, never written into the shipped file.

Notes for the maintainer

  • The catalog here was built against this branch's own Patterns/ with PatternCatalog.ts build --no-model; after any pattern change, run the same build or CheckPatterns.ts goes red (by design: the signal to rebuild).
  • The two Assay callers (thinking-fit, pattern-suggest) are not added to LifeOS/install/USER/CONFIG/assay-callers.json here because this PR touches system trees only; add them to the template on port, or they run unregistered (shadow, fail-open).
  • Data quirk found while building: fabric's pattern_explanations.md describes create_quiz as a reading plan, so that catalog purpose follows the wrong line; its triggers still route "create a quiz".
  • Happy to split this into two PRs (thinking record; Fabric catalog) if that ports more easily.

Tests

42 of 42 across LIFEOS/ROUTER/ThinkingCheck.test.ts, LIFEOS/SOCRATES/adapters/thinking.test.ts, skills/Fabric/Tools/PatternCatalog.test.ts, skills/Fabric/Tools/SuggestPattern.test.ts, run inside a fresh clone of main with this branch applied; PatternCatalog.ts check --complete and CheckPatterns.ts exit 0 there.

…a suggester

Per-feature Thinking: line in the ISA (ISAFormat), LIFEOS/ROUTER/ThinkingCheck.ts (line, check, fit), HookBridge thinking verbs, LifeOSHooks draws the line beside the route line, Socrates thinking adapter and seed question, Algorithm 8.46.0 § Spend rule. skills/Fabric: PatternCatalog.ts generates Patterns/catalog.yaml (purpose, triggers, input, output, ported intent, hash), SuggestPattern.ts ranks the top three for a plain request, CheckPatterns runs the catalog check, ExecutePattern and UpdatePatterns read the catalog, FabricSystem docs. Tests: 42 across four files. The catalog in this commit was built against this branch's own Patterns/ with --no-model.
@danielmiessler

Copy link
Copy Markdown
Owner

Note

🤖 Kai (AI assistant) responding on behalf of Daniel Miessler

Thanks for the care in this one. The tests, the fail-open callers and the split offer all came through.

I'm going to decline both halves, and here's why.

The thinking record: on 2026-11-10 we deliberately took out the machinery that planned, drew and checked which thinking skills a run used. That included the per-prompt judges, the route checks and the thinking heads. A per-run record plus a judge for every skip was exactly the kind of scaffolding we decided a capable session model doesn't need. What replaced it is one line in Routing preferences: before real work, look at every skill you have, thinking skills included, and use what helps without being asked. We measure whether that's happening from the transcripts directly, not from a line the run writes about itself. Also, Lane: is no longer drawn or checked by anything since that cut. It's a plain note now, so a sibling line built on it would bring back what we removed. The Algorithm files have also moved into the Constitution since this cut, so the v8.46.0 file wouldn't have a home.

The Fabric catalog and suggester: same reasoning. A generated catalog, a ranker and a shadow model call to pick among about 270 patterns is a skill-router. The model already reads pattern_explanations.md, where each pattern has a one-line description.

One real find in here: the create_quiz and create_reading_plan descriptions in pattern_explanations.md are swapped. That file comes from upstream Fabric and gets overwritten on every sync, so the fix belongs in the fabric repo. If you open it there, it'll flow down on the next pattern sync.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants