Claude/calm product truth kirl5l - #51
Merged
Merged
Conversation
…dd CI gate README/AGENTS.md/llms.txt/marketplace.json/plugin.json/workflow.rs still said "30 tools" (llms.txt: "22") even after the 4 new WS-1 txn admin tools shipped in v0.5.0 and docs/status.generated.md had already been regenerated to 34 -- gen-status.sh --check only ever protected the generated file itself, not these hand-typed surfaces. AGENTS.md's `edit` preset row was also missing those same 4 tools, not just the count. Fix the drift once, then add scripts/check-doc-truth.sh (wired into ci.yml's status-drift job and release.yml's qualify-release gate) so it can't silently reappear. See docs/plans/2026-08-03-product-truth-closure-plan.md for the full investigation and the follow-up PRs (guarantee-level taxonomy, benchmark claim registry) left as design-only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BLfCwfh8CDwVQDmso9FmMN
…s/what-external-users-get.md Broader sweep beyond the tool-count fix turned up two more real drift instances: - docs/what-external-users-get.md still said "30 MCP tools"/"13 toolsets"/"all 30 tools" and was missing the txn toolset + its 4 tools entirely -- add to check-doc-truth.sh's file list too. - txn.rs's module comment, its edit_transaction_status tool/param descriptions, edit.rs's tx_id field doc, and toolset.rs's preset comment all still described the WS-1 transaction journal as "shadow-mode" / "does not yet change real write-path behavior" -- false since today's v0.5.0 (txn::begin is now fail-closed; a critical-risk edit without an approver is blocked). Rewrote these to state precisely what's enforced (txn::begin) vs. deliberately still best-effort by design (later FileCommitted->IndexCommitted->Done transitions, since rolling back post-write is riskier than tolerating a detectable drift) -- not a blanket "it's all enforced now" overcorrection. Regenerated the 3 affected __toolsnaps__ snapshots (UPDATE_TOOLSNAPS=1) and docs/status.generated.md accordingly; both drift gates pass. Evaluated (not implemented): promoting the `txn` toolset into SAFETY_FLOOR_TOOLSETS now that WS-1 left shadow mode. Declined -- the floor's own test/doc comment justifies membership by "hosts a gate a write path depends on being reachable"; txn's 4 tools are read/diagnostic only and don't gate anything, so excluding them via set_toolset can't create a bypass. Reasoning now lives in txn.rs's module comment for the next reader. Also noted (pre-existing, unrelated to this change, not fixed here): edit_lines_aborts_when_txn_begin_fails and format_files_skips_one_file_when_txn_begin_fails_without_aborting_the_batch fail in this sandbox because it runs as root -- their chmod-read-only fault injection is a no-op for uid 0, so txn::begin doesn't actually fail here. Confirmed via git stash against clean HEAD; almost certainly passes in real (non-root) CI. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BLfCwfh8CDwVQDmso9FmMN
…stry (PR2+PR3) PR2: docs/guarantee-levels.toml catalogs behavior guarantees (enforced/ advisory/best_effort/optional/provider_dependent/unsupported) with cited evidence, rendered into docs/status.generated.md's new "Guarantee levels" section by gen-status.sh. No new CI step needed -- it's covered by the existing gen-status.sh --check gate, since the rendering is a pure function of the same generator. PR3: benchmarks/claims.registry.jsonl gives every measured benchmark claim a structured, machine-readable record (commit/corpus/harness pins, canonical result, supersede chain) instead of only prose in each README. Backfilled 3 supersede chains: B10->B11 (methodology audit), Dart zero-edges->C3 call-edge extraction, and B13 Phase1+2->corrected (oracle + SCIP-race fixes, plus a real CALM Self::method() parser bug found and fixed along the way). The two older chains predate this checkout's git history (starts 2026-07-28) and are marked with an explicit evidence_gap rather than a fabricated commit SHA -- the B13 chain is fully commit-verified (aba60aa..52d1abe/c5823a8/9c0b0fe, all reachable in this history). scripts/check-claims-registry.sh validates structural integrity (unique ids, bidirectional supersedes/superseded_by, raw_output_path exists) -- deliberately does not attempt to verify commit SHAs via git, since commit fields are intentionally free text (real entries here already read "X plus uncommitted fixes, later committed as Y") and a regex- based check would either choke on that prose or false-negative on a corpus commit belonging to an entirely different repo. Wired into both ci.yml's status-drift job and release.yml's qualify-release gate, alongside check-doc-truth.sh. See docs/plans/2026-08-03-product-truth-closure-plan.md for the full design and what's still intentionally out of scope (WS-4/5/6/9). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BLfCwfh8CDwVQDmso9FmMN
…l + plugin.json version drift check WS-6 first slice (docs/plans/2026-08-03-ws6-verification-pipeline-execution-plan.md): `[verification] rust_check_on_write` (default false, zero behavior change unless opted in) routes a `.rs` write through the durable transaction's `VERIFY_PENDING` state -- a legal `allowed_next` target since WS-1 that nothing had ever actually produced -- instead of straight to `Done`. New `verify_change(tx_id)` tool runs `cargo check` scoped to the nearest ancestor Cargo package and advances the transaction to `Done`/`Failed`; a failing check does NOT revert the file already written to disk, same non-blocking-after-write philosophy txn.rs's module comment already documents. Deliberately narrow: only Rust, only a Fast+Semantic tier (cargo check), no Security/Deep/Runtime tier, no other language -- see the plan doc's anti-goals. New crates/calm-core/src/verify.rs: is_verifiable_rust_file, find_nearest_cargo_toml (same ancestor-walk strategy as format::detect_rust_edition), run_cargo_check. Also (docs/plans/2026-08-03-product-truth-closure-plan.md follow-up): plugins/calm/.claude-plugin/plugin.json's version had never been bumped since the file was created, unlike Cargo.toml/npm packages which get stamped from the release tag at build/publish time -- the Claude Code plugin has no such build/publish step in this repo, so nothing ever reminded anyone to bump it by hand. Bumped 0.3.3 -> 0.5.0 and added a check (scripts/check-doc-truth.sh) that fails if it ever drifts from Cargo.toml again. Tool count 34 -> 35 (verify_change) -- regenerated status.generated.md and bumped all 7 hand-authored references check-doc-truth.sh already tracks. cargo fmt/clippy -D warnings clean; 305 calm-server + 956 calm-core tests pass (the 2 pre-existing txn_begin_fails tests are still skipped in this sandbox -- confirmed pre-existing and root-environment-specific, unrelated to this change). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BLfCwfh8CDwVQDmso9FmMN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.