Skip to content

Fix decision/error extraction gaps found by the external benchmark - #52

Merged
Shweta-Mishra-ai merged 1 commit into
mainfrom
claude/tokenizer-benchmark-research-p24fpu
Aug 13, 2026
Merged

Fix decision/error extraction gaps found by the external benchmark#52
Shweta-Mishra-ai merged 1 commit into
mainfrom
claude/tokenizer-benchmark-research-p24fpu

Conversation

@Shweta-Mishra-ai

Copy link
Copy Markdown
Owner

What this PR does

An independent 100-session, 8-method benchmark (Shweta-Mishra-ai/tokenmizer-research) scored this extractor's two weakest categories — decisions (50% F1) and errors (36% F1) — well behind two comparison methods (65% and 66%). This PR fixes the specific gaps that benchmark identified, relocated to patterns.py since an earlier attempt at this fix (superseded, never merged).

  • Adds _is_question_context, wired into all four decision passes: a question weighing options (e.g. "Should we go with Postgres or Redis?") was previously extracted as a decision made.
  • Broadens the decision trigger-verb vocabulary ("leaning toward", "recommend(ed/s)") and adds a handful of verified-safe technology names to the closed whitelist (sqlc, dbt, nats, pnpm, uv, ruff, kong, airflow).
  • Expands the error-symptom vocabulary (nil pointer dereference, gc pressure, poison message, consumer lag, schema drift, partition skew, goroutine leak, connection churn, thundering herd) and widens the trailing-context capture after a symptom.
  • Rejected and reverted two more aggressive fixes tried during this work: a whitelist-free CamelCase/kebab-case/ALLCAPS tech-name fallback (dropped decision precision 90%→81% on the eval corpus), and adding vite to the whitelist on its own (dropped precision to 88% via a false positive on "Completed: project scaffold with Vite"). Both are documented as negative results in the CHANGELOG.
  • Version bumped 0.5.3 → 0.5.4 with a full CHANGELOG entry.

Type

  • Bug fix
  • Extraction improvement (graph_memory/)
  • New provider
  • Performance
  • Documentation

Tests

  • pytest tests/ -v passes — 644/647 (3 known-unrelated Gemini/cffi native-library failures, pre-existing and unrelated to this change)
  • ruff check tokenmizer/ clean
  • Memory accuracy test added/updated — python -m benchmarks.eval confirms no regression: decisions steady at 90% precision / 95% recall, errors 93%/96% (both matching or exceeding the pre-fix baseline). The product's own 14-session corpus doesn't contain the specific gaps this fix targets, so it verifies no regression rather than the improvement itself — that signal comes from re-running the external n=100 benchmark, where TokenMizer's macro F1 rose 57%→60% and decisions/errors F1 rose 50%→59% / 36%→44%.

Checklist

  • No raw dicts crossing layer boundaries (use DTOs)
  • No os.getenv() outside config/settings.py
  • External imports are lazy (inside functions, with try/except ImportError)

Generated by Claude Code

Re-applies the fixes to patterns.py/hybrid_extractor.py (relocated
here by an unrelated PR since these were last measured): a question
false-positive guard for all four decision passes, broadened decision
trigger-verb vocabulary, a handful of safe tech-name whitelist
additions, and expanded error-symptom vocabulary with wider trailing
context. Two more aggressive fixes (a generic CamelCase/kebab-case
tech-name fallback, and adding "vite" to the whitelist) were tried and
reverted after they measurably dropped decision precision on the eval
corpus; documented as negative results in the CHANGELOG.

Full test suite: 644/647 passing (3 known-unrelated Gemini/cffi
failures). Own eval harness unchanged at 90%/95% decisions and 93%/96%
errors — that corpus doesn't contain the specific gaps this fix
targets, so it verifies no regression rather than the improvement
itself.
@Shweta-Mishra-ai
Shweta-Mishra-ai merged commit cd2cc90 into main Aug 13, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants