Fix decision/error extraction gaps found by the external benchmark - #52
Merged
Shweta-Mishra-ai merged 1 commit intoAug 13, 2026
Merged
Conversation
Re-applies the fixes to patterns.py/hybrid_extractor.py (relocated here by an unrelated PR since these were last measured): a question false-positive guard for all four decision passes, broadened decision trigger-verb vocabulary, a handful of safe tech-name whitelist additions, and expanded error-symptom vocabulary with wider trailing context. Two more aggressive fixes (a generic CamelCase/kebab-case tech-name fallback, and adding "vite" to the whitelist) were tried and reverted after they measurably dropped decision precision on the eval corpus; documented as negative results in the CHANGELOG. Full test suite: 644/647 passing (3 known-unrelated Gemini/cffi failures). Own eval harness unchanged at 90%/95% decisions and 93%/96% errors — that corpus doesn't contain the specific gaps this fix targets, so it verifies no regression rather than the improvement itself.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
An independent 100-session, 8-method benchmark (
Shweta-Mishra-ai/tokenmizer-research) scored this extractor's two weakest categories — decisions (50% F1) and errors (36% F1) — well behind two comparison methods (65% and 66%). This PR fixes the specific gaps that benchmark identified, relocated topatterns.pysince an earlier attempt at this fix (superseded, never merged)._is_question_context, wired into all four decision passes: a question weighing options (e.g. "Should we go with Postgres or Redis?") was previously extracted as a decision made.sqlc,dbt,nats,pnpm,uv,ruff,kong,airflow).nil pointer dereference,gc pressure,poison message,consumer lag,schema drift,partition skew,goroutine leak,connection churn,thundering herd) and widens the trailing-context capture after a symptom.viteto the whitelist on its own (dropped precision to 88% via a false positive on "Completed: project scaffold with Vite"). Both are documented as negative results in the CHANGELOG.Type
Tests
pytest tests/ -vpasses — 644/647 (3 known-unrelated Gemini/cffi native-library failures, pre-existing and unrelated to this change)ruff check tokenmizer/cleanpython -m benchmarks.evalconfirms no regression: decisions steady at 90% precision / 95% recall, errors 93%/96% (both matching or exceeding the pre-fix baseline). The product's own 14-session corpus doesn't contain the specific gaps this fix targets, so it verifies no regression rather than the improvement itself — that signal comes from re-running the external n=100 benchmark, where TokenMizer's macro F1 rose 57%→60% and decisions/errors F1 rose 50%→59% / 36%→44%.Checklist
os.getenv()outsideconfig/settings.pyGenerated by Claude Code