Skip to content

Commit 36eafa4

Browse files
Your Nameclaude
andcommitted
feat(core,server): truth-kernel hardening Waves 0-4 -- live verification, evidence semantics, assistant retrieval, coverage/noise honesty
Full delivery of docs/plans/2026-08-20-truth-kernel-hardening-execution-plan.md, a verification + execution plan against a large incoming audit claiming CALM's read/edit path could silently act on stale index coordinates. All 12 audit findings were independently confirmed against live code (one via direct SQLite query, five via live tool-call reproduction) before any fix landed. Wave 0 (stop the cheap, high-confidence leaks): path containment for source_range (reuses resolve_repo_path's PATH_ESCAPES_PROJECT_ROOT guard); fixed suggested_next args that don't validate against their own target tool's schema, plus a new invariant test that would have caught both; corrected the kind="text" tool description to stop claiming body coverage it didn't have yet; fixed edit_context's read_only_hint/idempotent_hint annotations (it mints a new ReviewAuthority/ChangeIntent on every call -- a real write, not a cacheable read); defensive clamps in source()/understand()/symbols_batch() against a stale line_start past current EOF panicking. Wave 1 (Live Truth Kernel, the core architectural fix): live-verification folded directly into resolve_symbol itself rather than a separate opt-in function agents could forget to call. New SymbolResolution::ReadFailed variant; hash-compares the live file against the indexed hash on every resolution, falling back to a fresh reparse matched by (name, kind, class_context) -- not bare name alone -- only on a mismatch. Threaded through all 9 existing call sites plus understand()/symbols_batch() (previously bypassed resolve_symbol_candidates entirely via independently-duplicated stale-slice logic). 5 new adversarial tests: stale/moved/duplicate-class/ genuinely-ambiguous/deleted-since-indexed. Wave 2 (Evidence/confidence semantics): EvidenceSnapshot::compute gained a mtime-only live-disk spot-check that downgrades Current to Degraded on any drift (fail-closed on a missing file or NULL mtime); the reconciliation fence in watch_supervisor now persists plain Current + a Current->Reconciled promotion instead of unconditional compute_after_reconciliation, so drift caught during the reconciliation's own reindex window blocks the Reconciled claim; canonical EdgeConfidence::is_verified/is_probable/is_lexical_lead predicates replace ad-hoc rank() comparisons at the two call sites judged correctness fixes (path()'s certain flag, the bridge-gate), with the coreness/ hub-detection bucket question deliberately deferred pending a false-hub-rate measurement pass (tightening it would loosen the edit gate). Wave 3 (Assistant-grade retrieval): search's default kind flipped symbol->hybrid (confirmed zero-regression via search_hybrid's existing graceful degrade to today's symbol-search output when no embedder is configured); a new fts_chunks FTS5 table gives kind="text" real function-body coverage, merged with the existing fts_exact hits via rrf_merge_n; understand() gained a top-1/top-2 margin check surfacing resolution_ confidence + alternatives instead of silently committing to a near-tied top hit; qualified_name threaded through resolve_symbol as an additional narrowing filter that still flows through the same live-verification step (the corrected design -- short-circuiting straight to a DB row by qualified_name would have reintroduced the exact staleness risk Wave 1 closed), restoring SearchResultItem.qualified_name (gated on kind.is_some() so synthetic file/gap-chunk identities are never surfaced as resolvable). Wave 4 (Coverage/noise honesty): fixed a genuine correctness bug found during this wave's own research, not in the original audit -- reindex_changed_ cancellable (the everyday incremental-reindex path) was deleting a file's indexed symbols/call_sites whenever it transiently failed to read (permission hiccup, briefly oversized, non-UTF-8 mid-write), indistinguishable from a genuinely deleted file even though the file still existed and would read fine next pass; landed first as its own isolated step. read_source_capped's return type changed Option<String> -> Result<String, String> so its 3 callers can record *why* a file was skipped via a new file_index.skip_reason column, surfaced through a new indexing_status.skipped_files field (capped list + true count) rather than leaving it write-only in the DB. search(kind="grep", include_tests=false) was a complete silent no-op (is_test hardcoded false for every grep result) -- enclosing_symbol now selects the real, already-computed symbols.is_test column instead. Verified: cargo test --workspace green across every wave's own closing pass (final state: calm-server 421/421, calm-core 1266/1266, plus schema-migration/ watcher-integration/doctest suites, 0 failures workspace-wide). cargo fmt --check clean throughout (via format_files, never raw rustfmt). cargo clippy --workspace --all-targets -- -D warnings clean. 12 toolsnaps regenerated across the session for every tool whose output schema changed. diff_impact confirms the full Wave 0-4 change surface matches expectations at each stage. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 parent 0d8d1f2 commit 36eafa4

31 files changed

Lines changed: 3243 additions & 198 deletions

crates/calm-core/src/authority/snapshot.rs

Lines changed: 160 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -146,10 +146,26 @@ impl EvidenceSnapshot {
146146
/// reconciliation first -- `freshness_class` reflects whatever
147147
/// `index_input_drift` reports right now (`Current` or `Degraded`,
148148
/// never `Reconciled`).
149+
/// Computes a snapshot from `conn`'s current state without forcing
150+
/// reconciliation first -- `freshness_class` reflects whatever
151+
/// `index_input_drift` reports right now (`Current` or `Degraded`,
152+
/// never `Reconciled`). A `Current` result additionally gets a cheap
153+
/// live-disk spot-check (2.1, Wave 2 -- `live_mtime_drift`) that can
154+
/// still downgrade it to `Degraded`: `index_input_drift` only tracks
155+
/// config/context fingerprints, not "has a source file changed on disk
156+
/// since the last successful index" -- that's a separate lag window
157+
/// (the watcher's debounce/reconciliation interval), closed here rather
158+
/// than left to `source_catalog_digest` (which only ever reads DB rows,
159+
/// never live bytes). The check is skipped when drift is already
160+
/// `Degraded` -- no need to pay for it when the answer can't change.
149161
pub fn compute(conn: &Connection, project_root: &Path) -> rusqlite::Result<Self> {
150162
let catalog = InputCatalog::for_project(project_root);
151163
let drift = index_input_drift(conn, &catalog)?;
152-
Self::build(conn, project_root, drift_to_freshness(drift))
164+
let mut freshness_class = drift_to_freshness(drift);
165+
if freshness_class == FreshnessClass::Current && live_mtime_drift(conn, project_root)? {
166+
freshness_class = FreshnessClass::Degraded;
167+
}
168+
Self::build(conn, project_root, freshness_class)
153169
}
154170

155171
/// Same as [`compute`](Self::compute), but for a caller that just ran a
@@ -169,10 +185,15 @@ impl EvidenceSnapshot {
169185
/// content's `snapshot_id` -- if a stronger freshness class was ever
170186
/// recorded for identical content (e.g. a past full reconciliation via
171187
/// [`compute_after_reconciliation`](Self::compute_after_reconciliation)),
172-
/// that's honored here too, not only at persist-time. Safe against
173-
/// TOCTOU by construction: `snapshot_id` is content-addressed, so any
174-
/// disk change since the recorded snapshot changes the id and the
175-
/// lookup simply misses -- there is no timestamp/TTL window to race.
188+
/// that's honored here too, not only at persist-time. `snapshot_id` is
189+
/// content-addressed over what the DB currently believes, so a disk
190+
/// change that has already been reindexed changes the id and this
191+
/// lookup simply misses -- no TTL window to race there. A disk change
192+
/// NOT yet reflected in any DB row is a different lag window, closed
193+
/// separately by `compute`'s own `live_mtime_drift` check (2.1, Wave 2):
194+
/// that degrades `freshness_class` to `Degraded` without needing a new
195+
/// `snapshot_id`, since the DB-visible content genuinely hasn't changed
196+
/// yet.
176197
pub fn compute_with_recorded_freshness(
177198
conn: &Connection,
178199
project_root: &Path,
@@ -326,6 +347,54 @@ fn source_catalog_digest(conn: &Connection) -> rusqlite::Result<String> {
326347
Ok(evidence_digest(material.as_bytes()))
327348
}
328349

350+
/// Live-disk companion to `source_catalog_digest` (2.1, Wave 2): that digest
351+
/// only ever reads DB rows, so a file edited on disk after its last
352+
/// successful index -- but before the watcher's debounce/reconciliation
353+
/// catches up -- is invisible to it (`snapshot_id` stays keyed on the stale
354+
/// DB hash). This closes that specific lag window cheaply by comparing each
355+
/// `file_index` row's live mtime (via `mtime_secs`, the exact function the
356+
/// indexer itself stamps rows with -- reused, not reimplemented, so both
357+
/// sides of the comparison come from one conversion) to its stored `mtime`
358+
/// column. Deliberately mtime-only, not a content rehash: `compute` runs on
359+
/// every gated edit (`edit_lines_impl_gated`), not just `edit_context`, so
360+
/// re-reading every indexed file's bytes here would double that cost across
361+
/// the whole catalog on every edit for a signal `source_catalog_digest`
362+
/// already gets once real reindexing happens.
363+
///
364+
/// **Documented residual, not silently claimed as caught:** a live mtime
365+
/// that matches the stored one on a file whose content genuinely differs
366+
/// (a same-timestamp overwrite) is not detected by this signal alone --
367+
/// closing that would need a full content rehash, deliberately not done
368+
/// here (see 2.1's design decision,
369+
/// docs/plans/2026-08-20-truth-kernel-hardening-execution-plan.md).
370+
///
371+
/// Fail-closed like `index_input_drift`'s own `Unknown` posture: a file
372+
/// missing from disk and a `NULL` stored `mtime` (a pre-migration row, or
373+
/// one indexed before this column existed) both count as drift rather than
374+
/// being silently skipped. Short-circuits on the first mismatch -- this is
375+
/// a boolean gate, not a digest, so there is no reason to keep scanning
376+
/// once drift is already proven.
377+
fn live_mtime_drift(conn: &Connection, project_root: &Path) -> rusqlite::Result<bool> {
378+
let mut stmt = conn.prepare("SELECT path, mtime FROM file_index")?;
379+
let mut rows = stmt.query([])?;
380+
while let Some(row) = rows.next()? {
381+
let path: String = row.get(0)?;
382+
let stored_mtime: Option<f64> = row.get(1)?;
383+
let Some(stored_mtime) = stored_mtime else {
384+
return Ok(true);
385+
};
386+
let full_path = project_root.join(&path);
387+
if !full_path.exists() {
388+
return Ok(true);
389+
}
390+
let live_mtime = crate::indexer::pipeline::mtime_secs(&full_path);
391+
if live_mtime != stored_mtime {
392+
return Ok(true);
393+
}
394+
}
395+
Ok(false)
396+
}
397+
329398
/// `evidence_digest` over the sorted `provider\0cache_key\0upgraded\0
330399
/// ruled_out\0inserted\0match_rate` rows of `scip_overlay_state` -- the
331400
/// same DB-resident table `scip::state` uses in place of the old
@@ -823,4 +892,90 @@ mod tests {
823892
assert_ne!(after_change.snapshot_id, stale_reconciled.snapshot_id);
824893
assert_ne!(after_change.freshness_class, FreshnessClass::Reconciled);
825894
}
895+
896+
#[test]
897+
fn live_disk_mtime_drift_downgrades_current_to_degraded() {
898+
use crate::indexer::refresh::{InputCatalog, persist_index_input_snapshot};
899+
let root = tmp_project();
900+
let file_path = root.path().join("a.rs");
901+
std::fs::write(&file_path, "fn a() {}").unwrap();
902+
let indexed_mtime = crate::indexer::pipeline::mtime_secs(&file_path);
903+
904+
let conn = Connection::open_in_memory().unwrap();
905+
init_db(&conn).unwrap();
906+
conn.execute(
907+
"INSERT INTO file_index (path, hash, last_indexed, mtime) \
908+
VALUES ('a.rs', 'h1', 0, ?1)",
909+
params![indexed_mtime],
910+
)
911+
.unwrap();
912+
persist_index_input_snapshot(&conn, &InputCatalog::for_project(root.path())).unwrap();
913+
914+
let before = EvidenceSnapshot::compute(&conn, root.path()).unwrap();
915+
assert_eq!(
916+
before.freshness_class,
917+
FreshnessClass::Current,
918+
"no drift yet -- stored mtime matches the file as indexed"
919+
);
920+
921+
// Mutate the file on disk WITHOUT reindexing (no file_index update) --
922+
// the exact "watcher hasn't caught up yet" lag window 2.1 exists to
923+
// catch. A short sleep guards against filesystems with coarse mtime
924+
// granularity reporting an identical timestamp for both writes.
925+
std::thread::sleep(std::time::Duration::from_millis(20));
926+
std::fs::write(
927+
&file_path,
928+
"fn a() { /* changed on disk, not reindexed */ }",
929+
)
930+
.unwrap();
931+
932+
let after = EvidenceSnapshot::compute(&conn, root.path()).unwrap();
933+
assert_eq!(
934+
after.freshness_class,
935+
FreshnessClass::Degraded,
936+
"live mtime no longer matches file_index.mtime -- must not still claim Current"
937+
);
938+
}
939+
940+
#[test]
941+
fn live_disk_mtime_drift_missing_file_fails_closed() {
942+
use crate::indexer::refresh::{InputCatalog, persist_index_input_snapshot};
943+
let root = tmp_project();
944+
// file_index claims a row for a file that was never actually written
945+
// to this project_root -- e.g. deleted since indexing.
946+
let conn = Connection::open_in_memory().unwrap();
947+
init_db(&conn).unwrap();
948+
conn.execute(
949+
"INSERT INTO file_index (path, hash, last_indexed, mtime) \
950+
VALUES ('missing.rs', 'h1', 0, 123.0)",
951+
[],
952+
)
953+
.unwrap();
954+
persist_index_input_snapshot(&conn, &InputCatalog::for_project(root.path())).unwrap();
955+
956+
let snap = EvidenceSnapshot::compute(&conn, root.path()).unwrap();
957+
assert_eq!(
958+
snap.freshness_class,
959+
FreshnessClass::Degraded,
960+
"a file_index row with no file on disk must fail closed, not be skipped"
961+
);
962+
}
963+
964+
#[test]
965+
fn live_disk_mtime_drift_null_mtime_fails_closed() {
966+
use crate::indexer::refresh::{InputCatalog, persist_index_input_snapshot};
967+
let root = tmp_project();
968+
std::fs::write(root.path().join("a.rs"), "fn a() {}").unwrap();
969+
// conn_with_file_index leaves `mtime` NULL -- a pre-migration row, or
970+
// one indexed before this column existed.
971+
let conn = conn_with_file_index(&[("a.rs", "h1")]);
972+
persist_index_input_snapshot(&conn, &InputCatalog::for_project(root.path())).unwrap();
973+
974+
let snap = EvidenceSnapshot::compute(&conn, root.path()).unwrap();
975+
assert_eq!(
976+
snap.freshness_class,
977+
FreshnessClass::Degraded,
978+
"a NULL stored mtime must fail closed, not be silently treated as no drift"
979+
);
980+
}
826981
}

crates/calm-core/src/db/schema.rs

Lines changed: 74 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,13 @@ CREATE TABLE IF NOT EXISTS symbols (
170170
docstring TEXT NOT NULL DEFAULT '',
171171
name_tokens TEXT NOT NULL DEFAULT '',
172172
caller_count INTEGER NOT NULL DEFAULT 0,
173+
-- 2.3 (Wave 2, truth-kernel hardening): is_verified()-only bucket
174+
-- (Formal/Resolved) alongside caller_count's broader definition
175+
-- (everything except Ambiguous) -- mirrors coreness/possible_coreness's
176+
-- existing dual-column pattern below. No consumer reads this yet;
177+
-- landed so a future gate can migrate to the stricter bucket
178+
-- deliberately, not by silently changing caller_count's meaning.
179+
verified_caller_count INTEGER NOT NULL DEFAULT 0,
173180
is_hub INTEGER NOT NULL DEFAULT 0,
174181
coreness INTEGER,
175182
possible_coreness INTEGER,
@@ -221,7 +228,8 @@ CREATE TABLE IF NOT EXISTS file_index (
221228
language TEXT,
222229
symbol_count INTEGER NOT NULL DEFAULT 0,
223230
last_indexed REAL NOT NULL,
224-
mtime REAL
231+
mtime REAL,
232+
skip_reason TEXT
225233
);
226234
227235
CREATE TABLE IF NOT EXISTS symbol_metrics_history (
@@ -966,6 +974,36 @@ CREATE TRIGGER IF NOT EXISTS symbols_au
966974
END;
967975
";
968976

977+
/// 3.2 (Wave 3): `code_chunks`' own FTS5 shadow table, `fts_exact`'s sibling
978+
/// but keyed on `chunk_text` (function-body text) instead of
979+
/// name/docstring/signature -- the real fix behind Wave 0's 0.3 stopgap.
980+
/// Unlike `symbols` (see `symbols_au` above), `code_chunks` rows are never
981+
/// UPDATEd in place -- every write path (`driver.rs::remove_file_rows`,
982+
/// `reindex_all_cancellable_with_phase`) DELETEs then re-INSERTs
983+
/// (`edges.rs::insert_code_chunks_batch`), confirmed by direct read before
984+
/// choosing this trigger set -- so only AFTER INSERT/DELETE are needed, no
985+
/// `code_chunks_au`.
986+
const FTS_CHUNKS_SQL: &str = "
987+
CREATE VIRTUAL TABLE IF NOT EXISTS fts_chunks USING fts5(
988+
chunk_text,
989+
content='code_chunks',
990+
content_rowid='id',
991+
tokenize='unicode61'
992+
);
993+
";
994+
995+
const CODE_CHUNKS_TRIGGERS_SQL: &str = "
996+
CREATE TRIGGER IF NOT EXISTS code_chunks_ai AFTER INSERT ON code_chunks BEGIN
997+
INSERT INTO fts_chunks(rowid, chunk_text)
998+
VALUES (new.id, new.chunk_text);
999+
END;
1000+
1001+
CREATE TRIGGER IF NOT EXISTS code_chunks_ad AFTER DELETE ON code_chunks BEGIN
1002+
INSERT INTO fts_chunks(fts_chunks, rowid, chunk_text)
1003+
VALUES ('delete', old.id, old.chunk_text);
1004+
END;
1005+
";
1006+
9691007
pub fn init_db(conn: &Connection) -> rusqlite::Result<()> {
9701008
conn.execute_batch("PRAGMA journal_mode=WAL;")?;
9711009
conn.execute_batch(SCHEMA_SQL)?;
@@ -1111,6 +1149,12 @@ fn run_migrations(conn: &Connection) -> rusqlite::Result<()> {
11111149
"INTEGER NOT NULL DEFAULT 0",
11121150
)?;
11131151
migrate_add_column(conn, "symbols", "coreness", "INTEGER")?;
1152+
migrate_add_column(
1153+
conn,
1154+
"symbols",
1155+
"verified_caller_count",
1156+
"INTEGER NOT NULL DEFAULT 0",
1157+
)?;
11141158
migrate_add_column(conn, "symbols", "possible_coreness", "INTEGER")?;
11151159
migrate_add_column(conn, "symbols", "class_context", "TEXT")?;
11161160
migrate_add_column(conn, "symbols", "is_test", "INTEGER NOT NULL DEFAULT 0")?;
@@ -1121,6 +1165,7 @@ fn run_migrations(conn: &Connection) -> rusqlite::Result<()> {
11211165
"INTEGER NOT NULL DEFAULT 1",
11221166
)?;
11231167
migrate_add_column(conn, "file_index", "mtime", "REAL")?;
1168+
migrate_add_column(conn, "file_index", "skip_reason", "TEXT")?;
11241169
// call_sites columns added after the table first shipped.
11251170
migrate_add_column(
11261171
conn,
@@ -1296,6 +1341,7 @@ fn run_migrations(conn: &Connection) -> rusqlite::Result<()> {
12961341
"INTEGER NOT NULL DEFAULT 0",
12971342
)?;
12981343
migrate_fts_add_signature(conn)?;
1344+
migrate_fts_chunks(conn)?;
12991345
migrate_add_scip_overlay_state(conn)?;
13001346
dedup_edges_and_add_unique_indexes(conn)?;
13011347
migrate_call_site_identity_v2(conn)?;
@@ -1569,6 +1615,33 @@ fn migrate_fts_add_signature(conn: &Connection) -> rusqlite::Result<()> {
15691615
Ok(())
15701616
}
15711617

1618+
/// 3.2 (Wave 3): one-time creation + backfill of `fts_chunks` for existing
1619+
/// databases whose `code_chunks` table already has rows from before this
1620+
/// migration existed -- `CREATE VIRTUAL TABLE IF NOT EXISTS` alone would
1621+
/// create an empty shadow table with no way to know about those
1622+
/// already-there rows (the AFTER INSERT trigger only fires for FUTURE
1623+
/// writes). Detected via `sqlite_master`, not a column-shape check (there's
1624+
/// no prior `fts_chunks` to compare a column against, unlike
1625+
/// `migrate_fts_add_signature`'s ALTER-style case) -- if it's already
1626+
/// there, both the table and any backfill already happened in a past run.
1627+
/// A fresh DB reaches this with `code_chunks` still empty, so the rebuild
1628+
/// below is a harmless no-op there.
1629+
fn migrate_fts_chunks(conn: &Connection) -> rusqlite::Result<()> {
1630+
let exists: bool = conn.query_row(
1631+
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'fts_chunks'",
1632+
[],
1633+
|r| r.get::<_, i64>(0),
1634+
)? > 0;
1635+
if exists {
1636+
return Ok(());
1637+
}
1638+
conn.execute_batch(FTS_CHUNKS_SQL)?;
1639+
conn.execute_batch(CODE_CHUNKS_TRIGGERS_SQL)?;
1640+
conn.execute_batch("INSERT INTO fts_chunks(fts_chunks) VALUES ('rebuild');")?;
1641+
tracing::info!("Migration: created fts_chunks and backfilled from existing code_chunks rows");
1642+
Ok(())
1643+
}
1644+
15721645
const PROJECT_MEMORY_FTS_SQL: &str = "
15731646
CREATE VIRTUAL TABLE IF NOT EXISTS project_memory_fts USING fts5(
15741647
topic,

0 commit comments

Comments
 (0)