Raise the benchmark guard threshold to 25%: it's a tripwire, not an instrument - #187
Merged
Merged
Conversation
…nstrument The 10% threshold sat below the noise floor of GitHub-hosted runners. Even with the interleaved fastest-branch-vs-slowest-baseline gate, whole runs of identical code were observed to differ by 10-30%: on a recent PR the same commit went fail/fail/pass across three attempts, each time flagging different benchmarks, including code paths the diff never touched (one untouched write-trap benchmark swung +31.7% within a single run set). Gating below the noise floor doesn't catch regressions, it breeds re-run-until-green rituals that erode trust in the check. At 25% the guard is honest about what hosted runners can deliver: a tripwire for gross accidental regressions (an accidental copy in a hot path, an algorithmic slip), not a precision instrument. Changes too small to trip it should be measured deliberately instead, with repeated local runs of the bench suite on an idle machine - which is also more sensitive than CI ever was at 10%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fDq6fyKg6QscqyyN6CcwC
Merged
berendkleinhaneveld
added a commit
that referenced
this pull request
Jul 10, 2026
First stable release. Since v0.19.0, observ has had a broad performance overhaul, gained full type-hint coverage and a new trigger_ref API, and picked up a documentation site. New features - trigger_ref(): force-notify the watchers of a proxy, mirroring Vue's triggerRef (#188, closes #123). - Fully typed: modern type hints throughout, a py.typed marker, and ty type-checking enforced in CI (#185, #184, closes #114). Performance - Reworked deep-watch traversal: plain leaf values are filtered out of the traverse stack at push time and raw targets are traversed directly. Traversing large flat/shallow structures is up to ~85% faster (#190). - Lower per-operation trap overhead and proxies constructed with positional flags: reactive reads/writes and proxy creation are ~35-50% faster (#192, #193). - proxy_db lifetimes are managed by reference counting instead of a gc hook, eliminating reference cycles and making cleanup deterministic (#177). - Scheduler and dependency-bookkeeping micro-optimizations: in-place bisect insertion (#191), single-read/write flush counting (#196), a set difference in cleanup_deps (#194), and reading the arg count off the code object in weak() (#195). Correctness - Guarantees, with tests, that observ creates no reference cycles (#189). Documentation - New MkDocs site published to GitHub Pages (#176), with an Internals section (#179) and a rewritten README (#180). Tooling - Modernized CI workflows (#182) and a more robust benchmark guard (#181, #187).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the discussion on #186 (closed): instead of adding retry machinery, reposition the guard honestly.
Why
The 10% threshold sat below the noise floor of GitHub-hosted runners. Even with the interleaved fastest-branch-vs-slowest-baseline gate, whole runs of identical code were observed to differ by 10–30%:
A gate below the noise floor doesn't catch regressions — it breeds re-run-until-green rituals that erode trust in the check (and waste the CI minutes it was meant to justify).
What this changes
--threshold 0.10→0.25in the workflow, and the matching default incompare_runs.py.🤖 Generated with Claude Code
https://claude.ai/code/session_016fDq6fyKg6QscqyyN6CcwC
Generated by Claude Code