ci(bench): add benchmarkoor replay workflow#24172
Merged
Merged
Conversation
mediocregopher
approved these changes
May 13, 2026
Run prerun/setup with immediate engine persistence, verify the post-prerun head is durable before promoting the schelk baseline, and default benchmarkoor replay runs to 6 cores and 32G memory.
Use static-file headers plus the MDBX Finish stage checkpoint to verify the post-prerun baseline after graceful shutdown. Add the unwind reset strategy as the default path and keep schelk promotion available as an alternate strategy.
Continue running selected benchmarkoor fixtures after per-test replay failures. Record invalid newPayload failures in results and summaries without treating them as benchmark samples.
Track selected, completed, and recorded replay failures for each benchmarkoor run so suite executions show that later tests continue after per-fixture failures.
Start benchmarkoor replay immediately after launching the Reth systemd scope instead of polling eth_blockNumber, eth_syncing, or the recovered head. Replay failures are already recorded per fixture.
Create the benchmarkoor JWT secret before launching Reth and pass the same file to Reth and benchmarkoor-replay. Default the workflow reset strategy to schelk while keeping unwind selectable.
Poll the authenticated Engine API TCP port with a short interval after launching Reth so benchmarkoor-replay does not race node startup. This avoids the old JSON-RPC readiness and sync polling.
Rjected
force-pushed
the
dan/create-benchmarkoor-replay-workflow
branch
from
May 14, 2026 02:25
0f62936 to
2e87f85
Compare
Member
|
I am meaning to clean the branch trigger later or remove the whole thing altogether. |
Thegreatsura
pushed a commit
to Thegreatsura/tempo
that referenced
this pull request
May 15, 2026
Automated nightly update of reth dependencies from `paradigmxyz/reth` main branch. ## Upstream reth changes [`9b6a79f...4206935`](paradigmxyz/reth@9b6a79f...4206935) 🔗 Amp thread: https://ampcode.com/threads/T-019e2762-5e1e-7627-8558-d18aa6373f05 **Engine** - Reserve prewarm proof target sets for performance ([#24198](paradigmxyz/reth#24198)) **CLI** - Add `download --force` snapshot overwrite support ([#24229](paradigmxyz/reth#24229)) **Examples** - Add custom state root example ([#24135](paradigmxyz/reth#24135)) **Migration** - Properly handle dummy blocks in `migrate-v2` ([#24189](paradigmxyz/reth#24189)) **Bench** - Add benchmarkoor replay workflow ([#24172](paradigmxyz/reth#24172)) - Add git-sha, git-ref, platform, and scenario metadata to txgen bench; use `DEREK_BENCH_ACK_TOKEN` for org membership check ([#24190](paradigmxyz/reth#24190), [#24191](paradigmxyz/reth#24191)) ## Migrations 🔗 Amp thread: https://ampcode.com/threads/T-019e2762-730d-7364-8c00-0ff1da43811b - Bumped all `reth-*` git dependencies in `Cargo.toml` from rev `9b6a79f` to rev `4206935` (Reth SDK upstream version bump); no API or code changes included in this diff. [GitHub Workflow](https://github.com/tempoxyz/tempo/actions/runs/25872584315)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new
bench-benchmarkoorworkflow that replays a selected benchmarkoor test set against a feature branch or PR and compares it withmain.The workflow imports the perfnet MinIO snapshot, checkpoints with schelk after gas bump and funding, and then runs the measured replay through
benchmarkoor-replay.