docs: pre-launch honesty + accuracy pass

Aligns the docs with what the code actually does and what the data
actually shows — the audit's most damaging findings were claims a
skeptic could disprove in seconds.

  - Drop the synthetic "67% rediscovery" masthead badge; the demo NOTE
    now labels it illustrative n=3 demo data, not a validated rate (HN-1).
  - Correct the pydantic-ai example counts everywhere: 9 scope groupings
    and 6 topics, matching the committed claims.jsonl / book.md (the data
    was right; the prose was stale) (DATA-1/2/3).
  - Fidelity claim now states what the gate certifies (the citation is
    real) and what it does not (that the statement is fully entailed —
    that's human/PR review) (HN-2). "Compile" is described honestly as an
    LLM-extraction step wrapped in deterministic verify/dedup/score (HN-3).
    "vs RAG" no longer implies semantic retrieval — retrieval is lexical
    today (HN-4).
  - Document the real `local` provider + `base_url`, the [openai]/[serve]
    extras, and the "secrets never travel" claim now precisely covers
    message content + tool-call args (DOC-4, SEC-1, install notes).
  - docs/scrub.md: meta scrubbing + broadened patterns + an honest
    "what is NOT redacted" note (raw AWS secret without its env-var name).
  - docs/mcp.md: stale `active.jsonl` path and nonexistent
    `--verbose`/`--debug` flags removed (DOC-2/3). README book sample now
    matches what render_book actually emits (DOC-5). Genericized a leaked
    local username; softened the premature "available on PyPI" wording.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: srijansk

CHANGELOG.md

@@ -4,6 +4,20 @@
 
 First working vertical slice — capture → compile → serve → actuate → measure, built test-first.
 
+### Pre-launch hardening
+
+A full adversarial audit before going public surfaced a cluster of fixes, all landed here:
+
+- **Incremental compile cost.** `lore watch` now caches extraction per session (sessions are immutable, so the id is a safe key) — each interval only LLM-extracts newly-ingested sessions instead of re-running the whole corpus against your key. `--rebuild` ignores the cache. The watch loop is now incremental in *cost*, not just idempotent in *output*.
+- **First-run works out of the box.** `anthropic` (the default provider) is now a base dependency, and SDK imports are guarded with an actionable message instead of a raw `ModuleNotFoundError`. `pipx install crewlore` → `lore compile` just works with a key.
+- **Local models are real.** `model.provider: local` + `model.base_url` routes to any OpenAI-compatible endpoint (Ollama, LM Studio, vLLM). Previously "local" was advertised but unimplemented and produced a dead-end error.
+- **`lore serve --mcp`** — the documented MCP flag now exists (it was missing; the copy-paste `mcp.json` failed to launch).
+- **Tool-call arguments are scrubbed.** Secrets passed as tool-call args live in event `meta`, which previously bypassed the scrubber; `meta` is now walked recursively. Coverage broadened (AWS `ASIA`/secret-key assignments, all `xox?-` Slack tokens, quoted multi-word secrets). See [`docs/scrub.md`](docs/scrub.md).
+- **Usage stats moved to a gitignored sidecar** so `lore query` no longer rewrites the git-tracked `claims.jsonl` on every call — `git log .lore/` stays clean.
+- **Robustness:** timestamps are always timezone-aware (a timestampless transcript no longer crashes the actuation loop); a single failing session no longer aborts the whole compile pass.
+- **`lore --version`** flag; one canonical-form definition shared between the fidelity gate and the reported fidelity number; honesty passes on the README (the synthetic rediscovery metric is labelled demo data, the fidelity claim states what the gate does and does not certify, retrieval is described as lexical).
+- Corrected the pydantic-ai example counts (9 scope groupings / 6 topics, matching the committed data).
+
 ### Added
 
 - **Schemas** — content-addressed `Claim` IDs (idempotent compile, clean merges); `NSFEvent`, `Anchor`, `Provenance`, `Conflict`, `UsageStats`. Actuation fields (`action`, `status`, `usage`) and a conflict-grouping `topic` baked in.
@@ -40,7 +54,7 @@ Compilation is now automatic by default — no human has to remember to run it:
 - **`lore watch`** runs that pass on an interval (`--once` for cron/CI); `lore compile` is the manual escape hatch.
 - **Signal gate** widened to capture procedures/conventions/team-norms, not only friction (was silently dropping "how we do X" / "the rule is Y" sessions).
 
-Live end-to-end on public-safe data (Haiku): 6 transcripts ingested, 2 secrets redacted, 7 compiled claims (decisions/gotchas/procedures), a rendered team-knowledge book, and a 67% preventable-rediscovery rate (2/3 held-out sessions).
+Live end-to-end on public-safe data (Haiku): 6 transcripts ingested, 2 secrets redacted, 7 compiled claims (decisions/gotchas/procedures), a rendered team-knowledge book, and 2 of 3 held-out sessions re-deriving known knowledge (illustrative demo data, n=3).
 
 ### Fidelity-gate contract made explicit (`_canonical_form`)
 
@@ -58,14 +72,14 @@ The fidelity gate's tolerance shape — what counts as a "verbatim" anchor — i
 `docs/examples/pydantic-ai/` now reflects three captured sessions (G1 #5679, G3 #5358, D1 #5536):
 
 - **18 active claims** (7 gotchas, 7 decisions, 3 procedures, 1 style)
-- **8 distinct scope groupings** spanning UI adapters, decorator introspection, durable-execution threat modeling, toolsets, tests, and version policy
+- **9 distinct scope groupings** spanning UI adapters, decorator introspection, durable-execution threat modeling, toolsets, tests, and version policy
 - **100% per-session canonical fidelity** — every anchor verified against its session's content under the explicit contract
 - **0 conflicts** (sessions disjoint in scope)
 - Provenance documents five real-data bugs the capture process found and fixed before publication.
 
 ### Renamed from `agent-lore` to `crewlore`
 
-The original working name `agent-lore` collided on PyPI and on GitHub (taken by an unrelated 2018 project). Renamed to `crewlore` — two familiar words, instant comprehension, available on PyPI + GitHub. The importable package, CLI command, and `.lore/` directory layout stay the same; only the distribution and GitHub URL change.
+The original working name `agent-lore` collided on PyPI and on GitHub (taken by an unrelated 2018 project). Renamed to `crewlore` — two familiar words, instant comprehension, with the name free on PyPI and the repo live on GitHub. The importable package, CLI command, and `.lore/` directory layout stay the same; only the distribution and GitHub URL change. (Not yet published to PyPI; install from git until the first release is cut.)
 
 ### README hero GIF rework
 

README.md

@@ -4,7 +4,6 @@
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![fidelity 100%](https://img.shields.io/badge/fidelity-100%25-success.svg)](docs/examples/pydantic-ai/)
 [![claims compiled 18](https://img.shields.io/badge/claims_compiled-18-informational.svg)](docs/examples/pydantic-ai/)
-[![rediscovery prevented 67%](https://img.shields.io/badge/rediscovery_prevented-67%25-blueviolet.svg)](docs/examples/pydantic-ai/)
 
 > **Your coding agents keep relearning what your team already figured out.**
 > `crewlore` compiles agent sessions into a citable, plaintext team-knowledge layer that lives in your git repo. Local-first.
@@ -56,14 +55,14 @@ The demo runs the full loop on bundled public-safe sessions and prints what it f
 > [!NOTE]
 > **Fidelity — 100%.** Every claim's citation resolves verbatim back to its source.
 > **Conflicts surfaced — 1.** A real disagreement kept with both provenances, not silently merged.
-> **Preventable rediscovery — 67%.** Two of three held-out follow-up sessions re-derived knowledge the layer already had.
+> **Preventable rediscovery — 2 of 3.** Two of the three held-out follow-up sessions re-derived knowledge the layer already had. (Illustrative demo data — n=3, not a benchmark.)
 
 ## See it run on a real codebase: pydantic-ai (17.3k ⭐)
 
 [`docs/examples/pydantic-ai/`](docs/examples/pydantic-ai/) is a committed snapshot of `crewlore` compiled on the public [`pydantic/pydantic-ai`](https://github.com/pydantic/pydantic-ai) repo — 3 Claude Code sessions on real issues, no synthetic data.
 
-- **18 claims** compiled across 8 scope groupings (UI adapters, decorator introspection, durable-execution threat modeling, toolsets, tests, version policy)
-- **100% fidelity** under the explicit [canonical-form contract](docs/anchors.md) — every anchor canonically resolves to a substring of its source session
+- **18 claims** compiled across 9 scope groupings (UI adapters, decorator introspection, durable-execution threat modeling, toolsets, tests, version policy)
+- **100% fidelity** under the explicit [canonical-form contract](docs/anchors.md) — every anchor's quote canonically resolves to a substring of its source session. (Fidelity certifies the *citation* is real, not that the model's *statement* is fully entailed by it — that's what human/PR review of the book is for.)
 - **0 conflicts** because the three sessions covered disjoint scopes — the conflict detector wasn't given anything to flag
 - **Receipts:** the rendered [`book.md`](docs/examples/pydantic-ai/book.md), the raw [`claims.jsonl`](docs/examples/pydantic-ai/claims.jsonl), and full [`provenance.md`](docs/examples/pydantic-ai/provenance.md) (session ids, commit hashes, compile cost, scrub redactions, five real-data bugs the capture surfaced and we fixed before publishing)
 
@@ -82,15 +81,18 @@ Raw, messy sessions go in. Out comes a structured, citable **compiled claim** 
 A human can verify it (the anchor points back to the exact session line); an agent can trust it (the citation is real, not hallucinated). Claims roll up into a knowledge book at `.lore/knowledge/README.md`, grouped by area and committed to your repo alongside your code:
 
 ```markdown
-## billing/webhook
-- [gotcha]   Billing webhook handler lacks idempotency check; dedupe on the Stripe key.
-- [decision] Use Stripe idempotency key for webhook dedup, not global locks.
+# Team knowledge (compiled by crewlore)
+
+## services/billing
+
+- **[gotcha]** Billing webhook handler lacks an idempotency check; dedupe on the Stripe key.
+  - *Do:* Dedupe on the Stripe idempotency key before processing.
+  - _anchor_ `ses_1#1`: "the handler has no idempotency check, so when Stripe retries a webhook the charge is processed again."
 
 ## deployment
-- [procedure] Always run migrations before deploy to prevent missing columns.
 
-## .
-- [procedure] Write the failing test first; PRs without one are rejected in review.
+- **[procedure]** Run migrations before deploy to prevent missing columns.
+  - *Do:* Run `make migrate` before every deploy.
 ```
 
 ## How it works
@@ -119,14 +121,14 @@ flowchart LR
 - **Serve** — writes a human- and agent-readable knowledge book to `.lore/knowledge/`, and exposes a query tool (including an optional MCP server) so any agent can pull the relevant slice on demand.
 - **Actuation loop** — every retrieval is recorded, and that usage drives a lifecycle: unused claims decay and archive, contradicted claims are retired, useful claims are reinforced. The store stays small and fresh instead of growing into a pile nobody reads.
 
-The intelligence is in **compile**; ingest and serve are deliberately thin, so supporting another coding agent is a small adapter, not a rewrite.
+The intelligence is in **compile**; ingest and serve are deliberately thin, so supporting another coding agent is a small adapter, not a rewrite. To be precise about the word "compile": extraction is an LLM step (the only non-deterministic part), wrapped in deterministic stages — verbatim-anchor verification, content-addressed dedup, conflict recording, and authority scoring. "Compile" means the repeatable session → claims transform, not that an LLM is absent.
 
 ## How it differs
 
 - **vs. hosted memory (Letta, mem0)** — their store lives in someone else's cloud and you can't `git log` it; `crewlore`'s lives in your repo as plaintext.
 - **vs. per-IDE memory (Cursor rules, Claude memory, Continue, Cody)** — tied to one developer, one IDE; `crewlore` is a *team* artifact, committed and reviewed like code.
 - **vs. hand-curated `CLAUDE.md` / `.cursorrules`** — humans write those by hand and they go stale; `crewlore` compiles + reinforces from real sessions and retires what stops being used.
-- **vs. RAG over a vector DB** — RAG retrieves *documents*; `crewlore` compiles atomic, citable *claims* with verbatim anchors, so a human or agent can verify the source line in two seconds, and so the same claim survives across phrasing variance.
+- **vs. RAG over a vector DB** — RAG retrieves *document chunks*; `crewlore` compiles atomic, citable *claims* with verbatim anchors, so a human or agent can verify the cited source in seconds. (Retrieval today is deterministic lexical overlap, not embeddings — simpler and dependency-free; semantic ranking is on the roadmap.)
 
 ## Why this exists
 
@@ -140,9 +142,9 @@ Knowledge discovered inside an agent session is private by default and lost by d
 
 ## Your data stays yours
 
-- **Local-first.** Capture, compile, and serve all run on infrastructure you control. Point the compiler at your own model provider or a local model — nothing routes through any `crewlore`-operated service, because there is none.
+- **Local-first.** Capture, compile, and serve all run on infrastructure you control. Point the compiler at your own model provider or a local OpenAI-compatible model (Ollama, LM Studio, vLLM) via `provider: local` — nothing routes through any `crewlore`-operated service, because there is none.
 - **Plaintext, in your repo.** The knowledge layer is human-readable Markdown and JSONL under `.lore/`, versioned by git. `git log .lore/` is your audit trail.
-- **Secrets never travel.** Scrubbing happens at ingest, before storage or any model call. Raw session captures are git-ignored by default.
+- **Secrets never travel.** Scrubbing — of both message content and tool-call arguments — happens at ingest, before storage or any model call. It's a high-precision pattern set (a floor, not a DLP guarantee; see [`docs/scrub.md`](docs/scrub.md)), and raw session captures are git-ignored by default regardless.
 
 ## CLI
 
@@ -161,16 +163,18 @@ Knowledge discovered inside an agent session is private by default and lost by d
 
 ```yaml
 model:
-  provider: anthropic          # anthropic | openai | (local)
+  provider: anthropic          # anthropic | openai | local
   name: claude-sonnet-4-6
+  # For provider: local — point at any OpenAI-compatible endpoint you run:
+  # base_url: http://localhost:11434/v1   # e.g. Ollama, LM Studio, vLLM
 capture:
   transcripts: ~/.claude/projects
 compile:
   cadence: auto                # `lore watch` interval below
   watch_interval_seconds: 300
 ```
 
-Bring your own key (`ANTHROPIC_API_KEY` / `OPENAI_API_KEY`); `crewlore` never ships keys anywhere.
+Bring your own key (`ANTHROPIC_API_KEY` / `OPENAI_API_KEY`); `crewlore` never ships keys anywhere. The default Anthropic provider works out of the box. For OpenAI or a local OpenAI-compatible model, add the SDK: `pipx inject crewlore openai` (or `pip install 'crewlore[openai]'`). With `provider: local` nothing leaves your machine at all — the compile call hits your own endpoint.
 
 ## Roadmap & limitations
 

docs/evaluating-on-your-codebase.md

@@ -114,5 +114,5 @@ After 4–6 sessions and one compile:
 For sensitive codebases (regulated industries, IP-heavy work):
 
 1. Review the `claims.jsonl` diff in every PR that compiles new sessions, exactly like reviewing code. The plaintext format makes this practical.
-2. Use a local model (configured in `.lore/config.yaml`) instead of a hosted API if your compliance posture forbids sending code context to a third party. Quality tracks the model; Sonnet-class is the current sweet spot.
+2. Use a local model instead of a hosted API if your compliance posture forbids sending code context to a third party: set `model.provider: local` and `model.base_url` (any OpenAI-compatible endpoint — Ollama, LM Studio, vLLM) in `.lore/config.yaml`. Quality tracks the model; Sonnet-class is the current sweet spot.
 3. The on-disk format is plaintext + git; no proprietary store; no phone-home. The audit trail is `git log .lore/`.

docs/examples/pydantic-ai/README.md

@@ -9,7 +9,7 @@ Real-data evidence of `crewlore` working on a public codebase. Every claim in `b
 | **Target repo** | [`pydantic/pydantic-ai`](https://github.com/pydantic/pydantic-ai) (17.3k ⭐, MIT) |
 | **Sessions captured** | 3 Claude Code sessions on real open issues ([#5679](https://github.com/pydantic/pydantic-ai/issues/5679), [#5358](https://github.com/pydantic/pydantic-ai/issues/5358), [#5536](https://github.com/pydantic/pydantic-ai/issues/5536)) |
 | **Compiled claims** | **18** (7 gotchas · 7 decisions · 3 procedures · 1 style) |
-| **Distinct scopes** | **8 groupings** spanning UI adapters, decorator introspection, durable execution, toolsets, tests, and the version policy |
+| **Distinct scopes** | **9 groupings** spanning UI adapters, decorator introspection, durable execution, toolsets, tests, and the version policy |
 | **Fidelity** | **100%** — every anchor's quote canonically resolves against its session's content. See [`docs/anchors.md`](../../anchors.md) for the canonical-form spec. |
 | **Conflicts** | 0 (the three sessions cover different scopes; no cross-session disagreement to record) |
 | **Compile cost** | ~$0.60 with Sonnet 4.6 (deterministic at temperature=0) |

docs/examples/pydantic-ai/provenance.md

@@ -23,7 +23,7 @@ All three sessions ran in Claude Code, in the cloned target repo directory.
 | Session id (Claude Code) | `a8fa3beb-ec13-4ba2-8f68-9c745eecfaa2` |
 | Captured | 2026-05-28 |
 | Wall-clock duration | ~35 minutes |
-| NSF events extracted | 179 (1 user_message, 38 agent_message, 67 tool_call, 67 tool_result) |
+| NSF events extracted | 173 (1 user_message, 38 agent_message, 67 tool_call, 67 tool_result) |
 | Signal-gate reasons fired | `correction · error · gotcha · convention` |
 | Claims contributed | 8 |
 
@@ -58,13 +58,13 @@ All three sessions ran in Claude Code, in the cloned target repo directory.
 | Model | `claude-sonnet-4-6` via Anthropic API |
 | Temperature | `0` (deterministic structured-output extraction) |
 | Max tokens | `8192` |
-| Compile command | equivalent to `lore compile --repo ~/demos/pydantic-ai --transcripts ~/.claude/projects/-Users-srijansaket-demos-pydantic-ai` |
+| Compile command | equivalent to `lore compile --repo ~/demos/pydantic-ai --transcripts ~/.claude/projects/-Users-you-demos-pydantic-ai` |
 | Compile cost | ~$0.60 (combined input + output across three sessions) |
 | Scrub redactions | 0 (no secrets in any of the three sessions) |
 
 ## Result stats
 
-- **18 active claims** across **8 scope groupings** and **5 distinct topics**.
+- **18 active claims** across **9 scope groupings** and **6 distinct topics**.
 - **100% fidelity** under the canonical-form contract defined in [`docs/anchors.md`](../../anchors.md). Every anchor's quote canonically resolves to a substring of its source session's content. The canonical form tolerates whitespace, case, and Markdown decoration; it rejects content drift, fabrication, paraphrase, and out-of-order stitching.
 - **0 conflicts** — the three sessions cover disjoint scopes (UI adapter round-tripping, decorator introspection, and durable-execution threat modeling), so no `(scope, kind, topic)` group has claims from ≥2 sessions disagreeing.
 

docs/mcp.md

@@ -68,9 +68,9 @@ The instrumentation closes the loop: the more the layer is *used*, the more its
 
 **`lore: command not found` after install** — `pipx ensurepath` and restart your shell, or use the absolute path to the binary in `mcp.json`'s `command` field.
 
-**The tool list doesn't show `lore_query`** — your client may be caching the previous tool list. Restart the client. If still not listed, run `lore serve --mcp` manually in a terminal and verify it stays running (i.e. it's waiting on stdin); if it exits immediately, run with `--verbose` to see the error.
+**The tool list doesn't show `lore_query`** — your client may be caching the previous tool list. Restart the client. If still not listed, run `lore serve --mcp` manually in a terminal and verify it stays running (i.e. it's waiting on stdin); if it exits immediately, the error prints to stderr.
 
-**Returns empty claims** — the repo's `.lore/claims/active.jsonl` is empty or your query doesn't overlap with any compiled claim's scope/statement/topic vocabulary. Run `lore status` to see how many active claims you have, and `lore query "<your task>" --debug` to see ranking scores.
+**Returns empty claims** — the repo's `.lore/claims/claims.jsonl` is empty or your query doesn't overlap with any compiled claim's scope/statement/topic vocabulary. Run `lore status` to see how many active claims you have, and `lore query "<your task>"` to check what the same ranking returns from the CLI.
 
 ## Privacy posture