Merge pull request #13 from nickroci/feat/generalize-invariant-repair

feat(curator): generalise integrity-repair escalation to all invariant kinds

Author: nickroci

daemon/agent_mem_daemon/librarian_prompt.py

@@ -835,9 +835,18 @@ def _slugify(s: str) -> str:
 them to you. These are NOT discretionary — propose an action to repair \
 each one. They take priority over salience-driven proposals.
 
-Each task names a ``file`` (relative to ``knowledge/``), a broken ``target`` \
-(the wikilink that does not resolve), and a ``context`` snippet showing \
-where it appears. For EACH broken-wikilink task, do this:
+Each task names a ``kind`` (the invariant type), a ``file`` (relative to \
+``knowledge/`` — for an over-cap dir this is the directory), a ``target`` \
+(the offending value, interpreted per kind), and a ``context`` snippet. \
+ALWAYS set ``salience_signal: null`` on a repair proposal — these are \
+integrity fixes, not salience judgments, and the Scholar will \
+verify-and-execute them rather than judge them for novelty. In \
+``reasoning``, quote the task's ``file`` and ``target`` and state which fix \
+you chose and why. Dispatch on ``kind``:
+
+──────────────────────────────────────────────────────────────────
+kind: broken_wikilink  (``target`` = the wikilink that does not resolve)
+──────────────────────────────────────────────────────────────────
 
   1. **Research the intended target.** The broken target usually got the \
 PATH wrong, not the concept. Run ``mcp__agent_mem_library__bm25_search`` \
@@ -863,15 +872,61 @@ def _slugify(s: str) -> str:
 the link or replace it with plain descriptive text), preserving the rest \
 of the prose. Explain in ``reasoning`` why no target should exist.
 
-  3. In ``reasoning``, quote the task's ``file`` and ``target`` and state \
-which of the three fixes you chose and why (cite the entry you found, or \
-state that no entry exists). Set ``salience_signal: null`` for repair \
-proposals — they are integrity fixes, not salience judgments.
-
 Do the link research with the SAME parallel-search discipline as for \
 dedup. A repair proposal that guesses the path without searching will \
 likely be vetoed.
 
+──────────────────────────────────────────────────────────────────
+kind: overcap_dir  (``file``/``target`` = the over-capacity directory)
+──────────────────────────────────────────────────────────────────
+
+The directory holds more than 5 entry .md files and must be rebalanced. \
+The ``context`` lists every entry currently in it.
+
+  1. **Read the entries** (or their frontmatter ``title``/``keywords`` from \
+the snapshot) to find the natural thematic groupings. Use the existing \
+sub-structure of sibling folders as a guide for sensible subfolder names.
+
+  2. **Propose exactly ONE rebalancing action:**
+     - **The entries split into 2+ coherent themes** → ``split_folder`` on \
+``folder_path`` = the over-cap dir, with ``into`` mapping each new \
+subfolder NAME → the list of entry paths (relative to ``knowledge/``) that \
+move there. Leave no destination subfolder over 5; entries you don't list \
+stay put (so the remainder must also be ≤5). The Scholar's \
+``move_entries`` call rewrites all inbound wikilinks atomically.
+     - **A few entries clearly belong in an EXISTING sibling folder** → one \
+``move_entry`` per such entry (``from_path`` → ``to_path``) until the dir \
+is back at or below 5. Prefer this when only one or two entries are \
+outliers and the rest are cohesive.
+
+  3. Choose subfolder names that read well as a path \
+(``global/python/testing/`` not ``global/python/misc-2/``). Do NOT propose \
+a split that just shards the dir into ``part-1``/``part-2`` — the grouping \
+must be meaningful.
+
+──────────────────────────────────────────────────────────────────
+kind: bad_frontmatter  (``file``/``target`` = the entry with bad frontmatter)
+──────────────────────────────────────────────────────────────────
+
+The entry's YAML frontmatter is missing, unparseable, or short the \
+required fields (``context`` names the exact defect). The body content is \
+fine — only the frontmatter block needs repair.
+
+  1. **Read ``file``** to recover its current frontmatter (whatever is \
+salvageable) and its full body.
+
+  2. **Propose an ``update_entry``** on ``file`` whose ``new_body`` is the \
+entry's UNCHANGED body preceded by a corrected frontmatter block that \
+re-serialises valid YAML with EVERY required field present: ``id`` \
+(kebab-case matching the filename), ``type``, ``scope`` (consistent with \
+the path — ``global`` under ``global/``, ``project:<slug>`` under \
+``projects/<slug>/``), ``status``, ``confidence``, ``applies-when``, \
+``keywords``, ``title``, ``created``, ``updated``, ``fired``, \
+``fired-helpful``, ``sources``. Preserve any existing valid values; fill \
+missing fields from the body's content and sensible defaults (e.g. \
+``status: provisional``, ``confidence: 0.6``, ``fired: 0``, \
+``fired-helpful: 0``). Do NOT rewrite the body prose.
+
 ═══════════════════════════════════════════════════════════════════
 HIERARCHY INVARIANTS (the Scholar will veto violations)
 ═══════════════════════════════════════════════════════════════════

daemon/agent_mem_daemon/repair_queue.py

@@ -1,13 +1,22 @@
 """Integrity-repair task queue — escalation bridge into the curator.
 
-When the deterministic post-write pass (``scholar_prompt.repair_broken_wikilinks``)
-finds an invariant violation it CANNOT fix on its own — today, a broken
-wikilink with no unique on-disk target — it records a :class:`RepairTask`
-here. The next Librarian run drains the pending tasks, renders them into
-its prompt, and proposes an EXISTING curator action (rewrite the link,
-write the missing target, or remove it with a reason). The Scholar
-reviews/executes that proposal as a normal proposal. No new agent, no new
-role — the same Librarian→Scholar pipeline, fed a different kind of input.
+When a post-write check finds an invariant violation that cannot be fixed
+deterministically, it records a :class:`RepairTask` here. The next Librarian
+run drains the pending tasks, renders them into its prompt, and proposes an
+EXISTING curator action; the Scholar reviews/executes it as a normal
+proposal. No new agent, no new role — the same Librarian→Scholar pipeline,
+fed a different kind of input. Three invariant kinds escalate through this
+one mechanism (see the ``KIND_*`` constants), dispatched by ``kind``:
+
+  - ``broken_wikilink`` — surfaced by ``scholar_prompt.repair_broken_wikilinks``
+    when a link has no unique on-disk target → Librarian rewrites / writes
+    the missing target / removes the link.
+  - ``overcap_dir``     — surfaced by ``check_invariants`` when a flat dir
+    exceeds the entry cap → Librarian proposes a ``split_folder`` (or
+    ``move_entry``) to rebalance.
+  - ``bad_frontmatter`` — surfaced by ``check_invariants`` when an entry's
+    frontmatter is missing/unparseable/incomplete → Librarian proposes an
+    ``update_entry`` that re-serialises valid frontmatter.
 
 ────────────────────────────────────────────────────────────────────
 THE ONLY GUARD: in-flight, keyed on the issue fingerprint
@@ -39,12 +48,14 @@
 re-escalates it. (For that to hold, the escalation path must NOT neutralise
 the link it escalates — see ``scholar_prompt._repair_body_links``: an
 escalated link is left broken on disk so it stays detectable until the
-Scholar actually fixes it.)
-
-Extensibility: :class:`RepairTask` carries a ``kind`` discriminator so
-other invariant types (over-cap directories, malformed frontmatter) can
-plug into the same queue + guard later. Only ``broken_wikilink`` is wired
-today.
+Scholar actually fixes it. The over-cap and bad-frontmatter checks are
+already non-destructive — they only read — so re-detection is automatic.)
+
+Generality: :class:`RepairTask` carries a ``kind`` discriminator and the
+queue treats every kind identically. Adding a fourth invariant type is a
+matter of (a) a new ``KIND_*`` literal, (b) a detection point that enqueues
+it, and (c) a prompt section telling the Librarian how to repair it — no
+change to the queue itself.
 """
 
 from __future__ import annotations
@@ -57,18 +68,31 @@
 # issue" so repeated detections collapse onto one in-flight attempt.
 Fingerprint = Tuple[str, str, str]
 
-# v1 wires only this kind. New invariant types add their own literal and
-# reuse the same enqueue/drain/clear machinery.
+# Invariant-type discriminators. Every kind reuses the SAME
+# enqueue/drain/clear machinery + in-flight guard — the queue itself stays
+# kind-agnostic; only the detection point (Scholar) and the rendering point
+# (Librarian prompt) branch on the value.
+#
+#   - ``broken_wikilink`` — a wikilink with no unique on-disk target.
+#   - ``overcap_dir``     — a flat directory over ``MAX_FLAT_DIR_ENTRIES``.
+#   - ``bad_frontmatter`` — an entry whose frontmatter is missing,
+#                           unparseable, or short the required fields.
 KIND_BROKEN_WIKILINK = "broken_wikilink"
+KIND_OVERCAP_DIR = "overcap_dir"
+KIND_BAD_FRONTMATTER = "bad_frontmatter"
 
 
 @dataclass(frozen=True)
 class RepairTask:
     """One integrity-repair task escalated to the Librarian.
 
-    ``kind``    discriminator for the invariant type (``broken_wikilink``).
-    ``file``    knowledge-root-relative path of the file holding the issue.
-    ``target``  the offending value — for a wikilink, its broken target.
+    ``kind``    discriminator for the invariant type (one of ``KIND_*``).
+    ``file``    knowledge-root-relative path of the file (or directory, for
+                ``overcap_dir``) holding the issue.
+    ``target``  the offending value, interpreted per ``kind``: a wikilink's
+                broken target; the directory path for an over-cap dir; the
+                entry path for bad frontmatter. Always non-empty so the
+                fingerprint is a stable identity.
     ``context`` a short human-readable snippet so the Librarian can see
                 where/how the issue appears without re-reading the file.
     """

daemon/agent_mem_daemon/scholar.py

@@ -244,16 +244,55 @@ def _repair_wikilinks_safe(record: runs.InvocationRecord) -> None:
         log.exception("scholar.review: wikilink repair raised")
 
 
+def _escalate_invariant_violation(violation: scholar_prompt.InvariantViolation) -> None:
+    """Enqueue a repair task for one escalating invariant violation.
+
+    The SAME mechanism as ``_escalate_unresolved_wikilink``: build the
+    violation's :class:`repair_queue.RepairTask` and ``enqueue`` it. The
+    in-flight guard inside the queue collapses repeated detections of the
+    same fingerprint onto one attempt; there is no max-attempts cap, so a
+    still-unfixed violation re-escalates on the next pass once its marker is
+    released. Display-only violations (``repair_kind is None``) yield no
+    task and are skipped."""
+    task = violation.to_repair_task()
+    if task is None:
+        return
+    enqueued = repair_queue.get_queue().enqueue(task)
+    if enqueued:
+        log.info(
+            "scholar.review: escalated %s to Librarian (file=%s target=%s)",
+            task.kind,
+            task.file,
+            task.target,
+        )
+    else:
+        log.debug(
+            "scholar.review: %s already in-flight; skipping duplicate escalation "
+            "(file=%s target=%s)",
+            task.kind,
+            task.file,
+            task.target,
+        )
+
+
 def _check_invariants_safe(record: runs.InvocationRecord) -> None:
-    """Deterministic post-write invariants check. Safety net for anything
-    that survived the Scholar's judgement, the reconciler, and the
-    wikilink repair pass."""
+    """Deterministic post-write invariants check + escalation. Safety net
+    for anything that survived the Scholar's judgement, the reconciler, and
+    the wikilink repair pass.
+
+    Over-cap directories and bad/unparseable frontmatter are not
+    deterministically fixable, so each such violation is ESCALATED into the
+    Librarian→Scholar pipeline via the same repair queue + in-flight guard
+    that broken wikilinks use (``_escalate_invariant_violation``). The
+    violation is left in place on disk, so the next pass re-detects and
+    re-escalates it until the Scholar actually fixes it."""
     try:
-        violations = scholar_prompt.check_invariants(knowledge_dir())
+        violations = scholar_prompt.check_invariants_detailed(knowledge_dir())
         if violations:
             record.decisions["invariant_violations"] = len(violations)
             for v in violations:
-                log.warning("scholar.review: post-write violation: %s", v)
+                log.warning("scholar.review: post-write violation: %s", v.message)
+                _escalate_invariant_violation(v)
         else:
             log.debug("scholar.review: invariants clean")
     except Exception: