fix: read the criterion output a cached baseline directory breaks apart - #193
Merged
Conversation
criterion-benchmarks has been failing with `no criterion results parsed, nothing to compare`. A cache hit brings benchmarks/target back with target/criterion in a half state: the per-benchmark base directory is there but the sample.json inside it is not. Criterion takes the directory alone as a baseline to compare against, cannot read it, and prints the error to stdout rather than stderr. It lands in the middle of the line the bencher formatter was writing, so all thirteen results arrive as a header and a measurement on separate lines and the single-line regex matches none of them. Both benchmark workflows now clear target/criterion before the run. The baseline for the comparison comes from the benchmark-data branch, so criterion's own on-disk one is only ever leftovers from an unrelated PR. The parser reads the header and the measurement separately, so anything criterion prints between the two no longer takes the whole comparison with it. A name with no measurement behind it is dropped instead of picking up the next benchmark's figure, which keeps a benchmark that produced nothing reported as missing. The refresh workflow and the local runner each kept their own copy of the old regex and both now call the one in compare_criterion.py, with the refresh failing rather than storing an empty baseline for every later comparison to skip over. The parser has tests, and CI runs them.
…ing it A run that produced results for some benchmarks and not others compared the ones it had and exited 0, so the check went green while the comment it posted said "treat this as a failed comparison rather than a pass". The comparison already worked out which benchmarks went missing; it just had nowhere to put the answer. build_report now returns that list and the run exits 4, which both workflows read as a failure that still has a table worth showing, unlike the exit 1 that means no comparison happened at all. The missing-benchmark warning also sat behind the regression warning in an if/elif chain, so a run that both regressed and lost benchmarks only ever mentioned the regression. They are independent now, and missing takes precedence over the advisory regression exit. Tests cover build_report and the exit codes the workflows branch on. The bencher fixtures carried criterion's progress notes on stdout, where it writes them with eprintln! and tee never sees them.
Contributor
Criterion Benchmark ResultsBaseline is the per-benchmark median of the last 5 stored runs, so one unusually fast or slow runner cannot skew the comparison. The range column is the spread across those runs.
Runs in the baseline
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
criterion-benchmarksfails whenever the job gets a cache hit. A restoredtarget/criterionhas the per-benchmarkbasedirectory but not thesample.jsonin it, and criterion prints the resulting read failure to stdout, in the middle of the line the bencher formatter is writing. The workflows now cleartarget/criterionbefore the run, and the parser reads a split line as well as a whole one.A second problem surfaced alongside it: a run that lost some benchmarks compared the rest and exited 0, so the check went green while the comment it posted said the result should not be trusted. That now exits 4 and fails the step, keeping the table that names what went missing.
Closes #192.
Note for whoever merges
feat/vector-indexes: 5533bae on that branch adds the samerm -rfinline on thecargo benchline. Keep one, not both.Checklist
cargo fmt --checkandcargo clippy -- -D warningspass locallyCHANGELOG.mdupdated if this is a user-visible change(MIT License and Apache License, Version 2.0)
DynamoDB compatibility note
Not applicable.