Skip to content

ci(rebase): Stage nightly Agave rebases with per-channel landing - #1523

Open
0xEdgar wants to merge 17 commits into
masterfrom
feat/mev-12307-nightly-rebase
Open

ci(rebase): Stage nightly Agave rebases with per-channel landing#1523
0xEdgar wants to merge 17 commits into
masterfrom
feat/mev-12307-nightly-rebase

Conversation

@0xEdgar

@0xEdgar 0xEdgar commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Why

The nightly rebase job (rebase.yaml) has been dead since at least 2026-08-25. Its personal access token expired, every run fails at checkout, and nothing alerted. Master and the release branches have drifted from Agave and are being rebased by hand.

What this does

weekdays 19:00 UTC, one job per channel in rebase-channels.json

  agave/<upstream>                   origin/<channel>
        │                                  │
        └──────────┐            ┌──────────┘
                   ▼            ▼
             git rebase carry commits onto agave tip
                   │
        ┌──────────┴──────────┐
        │ conflict            │ clean
        ▼                     ▼
  git rebase --abort     push ci/rebase/<channel>  (reused if same tree)
  open/refresh issue          │
  "Nightly rebase             ├─────────────────────────┐
   conflict: <channel>"       │ landing: draft          │ landing: auto
  assign owner                ▼                         ▼
        │              open/refresh draft PR     poll buildkite/jito-solana
        │              · carry commits                  │
        │              · range-diff             ┌───────┴────────┐
        │              · force-push command     │ green          │ red / 4h
        │                     │                 ▼                ▼
        │                     ▼          git push --force-   report ci_failure
        │              human reviews,    with-lease=<old      staging left
        │              runs the command  tip> <channel>       in place
        │                     │                 │
        │                     ▼          ┌──────┴──────┐
        │              PR marked merged  │ ok          │ lease failed
        │                                ▼             ▼
        │                             landed        stale (channel
        │                                           moved; retry
        │                                           next run)
        └────────────────────┬───────────────────────┘
                             ▼
                one Slack line per channel, every run

How it authenticates

A GitHub App token is minted per run with actions/create-github-app-token (GitHub-owned). It expires in an hour and is scoped to this repo. Commits are GPG-signed with a bot key imported inline. There are no third-party actions and GITHUB_TOKEN stays read-only.

Why master auto-lands but release branches don't

Master has high upstream churn (about 13 Agave commits a day, about 3 a week conflicting with our patch) and nothing runs from it directly. Release branches move slowly and feed mainnet tags, so a human should look before they change. Flipping a channel between modes is a one-word edit to the JSON.

Testing

.github/scripts/nightly-rebase-smoke.sh runs the script against local throwaway repos with a fake gh, covering draft staging, staging reuse, auto landing with signature check, fresh, conflict, and the stale-lease case.

Also run end to end on a fork with real Agave branches: master and v4.2 hit genuine conflicts and filed issues, v4.3 rebased cleanly, opened a draft PR, then landed via the auto path on a second run, and reported fresh on a third.

Before enabling

  1. An org admin creates a jito-foundation-owned GitHub App (Contents, Issues, Pull requests: read and write) and installs it on this repo.
  2. Add that App as an always bypass actor on the master repo ruleset and on the org rulesets "Restrict force push to master" and "Block Main branch deletion and require PRs to merge". Leave the Feature branches ruleset alone so the bot cannot rewrite v*.*.
  3. Secrets: REBASE_APP_ID, REBASE_APP_PRIVATE_KEY, GPG_PRIVATE_KEY, GPG_PASSPHRASE, SLACK_WEBHOOK_URL. The GPG key must be registered on the bot's GitHub account so landed commits verify.

Expect the first run to file a conflict on master: the Aug 31 Jito Patch conflicts with current Agave master in core/src/tpu.rs, gossip/src/protocol.rs, and the deleted ci/channel-info.sh.

User guide: dev/REBASE.md.

@0xEdgar

0xEdgar commented Jul 18, 2026

Copy link
Copy Markdown
Contributor Author

Testing moved to 0xEdgar#15.

@0xEdgar 0xEdgar closed this Jul 18, 2026
@0xEdgar 0xEdgar reopened this Jul 18, 2026
@0xEdgar
0xEdgar marked this pull request as ready for review July 20, 2026 13:07
@0xEdgar
0xEdgar requested a review from buffalu July 20, 2026 13:07

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5ce0ad254b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

uses: actions/download-artifact@v4
with:
pattern: nightly-rebase-*
merge-multiple: true

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve each channel result in the summary

When more than one matrix channel runs, each job uploads the same filename, nightly-rebase-result.json, and this merge-multiple: true download extracts all artifacts into one directory. The actions/upload-artifact v4 migration docs note that same-name files in merged artifacts use last-writer-wins behavior, so the summary loop will only see whichever channel JSON survived and can omit the other channels' statuses/conflicts from Slack. Use per-artifact directories or channel-specific result filenames before merging.

Useful? React with 👍 / 👎.

only reroute if relayer connected (#123)
feat: add client tls config (#121)
remove extra val (#129)
fix clippy (#130)
copy all binaries to docker-output (#131)
Ledger tool halts at slot passed to create-snapshot (#118)
update program submodule (#133)
quick fix for tips and clearing old bundles (#135)
update submodule to new program (#136)
Improve stake-meta-generator usability (#134)
pinning submodule head (#140)
Use BundleAccountLocker when handling tip txs (#147)
Add metrics for relayer + block engine proxy (#149)
Build claim-mev in docker (#141)
Rework bundle receiving and add metrics (#152) (#154)
update submodule + dev files (#158)
Deterministically find tip amounts, add meta to stake info, and cleanup pubkey/strings in MEV tips (#159)
update jito-programs submodule (#160)
Separate MEV tip related workflow (#161)
Add block builder fee protos (#162)
fix jito programs (#163)
update submodule so autosnapshot exits out of ledger tool early (#164)
Pipe through block builder fee (#167)
pull in new snapshot code (#171)
block builder bug (#172)

Pull in new slack autosnapshot submodule (#174)

sort stake meta json and use int math (#176)

add accountsdb conn submod (#169)

Update tip distribution parameters (#177)

new submodules (#180)

Add buildkite link for jito CI (#183)

Fixed broken links to repositories (#184)

Changed from ssh to https transfer for clone

Seg/update submods (#187)

fix tests (#190)

rm geyser submod (#192)

rm dangling geyser references (#193)

fix syntax err (#195)

use deterministic req ids in batch calls (#199)

update jito-programs

revert cargo

update Cargo lock

update with path fix

fix cargo

update autosnapshot with block lookback (#201)

[JIT-460] When claiming mev tips, skip accounts that won't have min rent exempt amount after claiming (#203)

Add logging for sol balance desired (#205)

* add logging

* add logging

* update msg

* tweak vars

update submodule (#204)

use efficient data structures when calling batch_simulate_bundles (#206)

[JIT-504] Add low balance check in uploading merkle roots (#209)

add config to simulate on top of working bank (#211)

rm frozen bank check

simulate_bundle rpc bugfixes (#214)

rm frozen bank check in simulate_bundle rpc method

[JIT-519] Store ClaimStatus address in merkle-root-json (#210)

* add files

* switch to include bump

update submodule (#217)

add amount filter (#218)

update autosnapshot (#222)

Print TX error in Bundles (#223)

add new args to support single relayer and block-engine endpoints (#224)

point to new jito-programs submod and invoke updated init tda instruction (#228)

fix clippy errors (#230)

fix validator start scripts (#232)

Point README to gitbook (#237)

use packaged cargo bin to build (#239)

Add validator identity pubkey to StakeMeta (#226)

The vote account associated with a validator is not a permanent link, so log the validator identity as well.

bugfix: conditionally compile with debug flags (#240)

Seg/tip distributor master (#242)

* validate tree nodes

* fix unit tests

* pr feedback

* bump jito-programs submod

Simplify bootstrapping (#241)

* startup without precompile

* update spacing

* use release mode

* spacing

fix validation

rm validation skip

Account for block builder fee when generating excess tip balance (#247)

Improve docker caching

delay constructing claim mev txs (#253)

fix stake meta tests from bb fee (#254)

fix tests

Buffer bundles that exceed cost model (#225)

* buffer bundles that exceed cost model

clear qos failed bundles buffer if not leader soon (#260)

update Cargo.lock to correct solana versions in jito-programs submodule (#265)

fix simulate_bundle client and better error handling (#267)

update submod (#272)

Preallocate Bundle Cost (#238)

fix Dockerfile (#278)

Fix Tests (#279)

Fix Tests (#281)

* fix tests

update jito-programs submod (#282)

add reclaim rent workflow (#283)

update jito-programs submod

fix clippy errs

rm wrong assertion and swap out file write fn call (#292)

Remove security.md (#293)

demote frequent relayer_stage-stream_error to warn (#275)

account for case where TDA exists but not allocated (#295)

implement better retries for tip-distributor workflows (#297)

limit number of concurrent rpc calls (#298)

Discard Empty Packet Batches (#299)

Identity Hotswap (#290)

small fixes (#305)

Set backend config from admin rpc (#304)

Admin Shred Receiver Change (#306)

Seg/rm bundle UUID (#309)

Fix github workflow to recursively clone (#327)

Add recursive checkout for downstream-project-spl.yaml (#341)

Use cluster info functions for tpu (#345)

Use git rev-parse for git sha

Remove blacklisted tx from message_hash_to_transaction (#374)

Updates bootstrap and start scripts needed for local dev. (#384)

Remove Deprecated Cli Args (#387)

Master Rebase

improve simulate_bundle errors and response (#404)

derive Clone on accountoverrides (#416)

Add upsert to AccountOverrides (#419)

update jito-programs (#430)

[JIT-1661] Faster Autosnapshot (#436)

Reverts simulate_transaction result calls to upstream (#446)

Don't unlock accounts in TransactionBatches used during simulation (#449)

first pass at wiring up jito-plugin (#428)

[JIT-1713] Fix bundle's blockspace preallocation (#489)

[JIT-1708] Fix TOC TOU condition for relayer and block engine config (#491)

[JIT-1710] - Optimize Bundle Consumer Checks (#490)

Add Blockhash Metrics to Bundle Committer (#500)

add priority fee ix to mev-claim (#520)

Update Autosnapshot (#548)

Run MEV claims + reclaiming rent-exempt amounts in parallel. (#582)

Update CI (#584)
- Add recursive submodule checkouts.
- Re-add solana-secondary step

Add more release fixes (#585)

Fix more release urls (#588)

[JIT-1812] Fix blocking mutexs (#495)

 [JIT-1711] Compare the unprocessed transaction storage BundleStorage against a constant instead of VecDeque::capacity() (#587)

Automatically rebase Jito-Solana on a periodic basis. Send message on slack during any failures or success.

Fix periodic rebase #594

Fixes the following bugs in the periodic rebase:
Sends multiple messages on failure instead of one
Cancels entire job if one branch fails

Ignore buildkite curl errors for rebasing and try to keep curling until job times out (#597)

Sleep longer waiting for buildkite to start (#598)

correctly initialize account overrides (#595)

Fix: Ensure set contact info to UDP port instead of QUIC (#603)

Add fast replay branch to daily rebase (#607)

take a snapshot of all bundle accounts before sim (#13) (#615)

update jito-programs submodule

Add 2.0 to daily rebase (#626)

Export agave binaries during docker build (#627)

Buffer bundles that exceed processing time and make the allowed processing time longer (#611)

Publish releases to S3 and GCS (#633)

Rebase from different repos (#637)

Point SECURITY.md to immunefi (#671)

Loosen requirements on tip accounts touchable in BankingStage (#683)

Separate out broadcast + retransmit shredstream (#703)

Add packet flag for staked node (#705)

Add auto-rebase to v2.1 (#739)

Fix release github (#745)

Move block_cost_limit tracking to BankingStage in preparation for SIMD-0207 (#753)

Add precompile checks in BundleStage (#787)

Add auto-rebase to v2.2 (#818)

Add better error handling around missing transaction signatures for bundle id generation (#860)

Remove unwrap from authentication (#861)

Revert Jito-Solana WorkingBankEntry changes (#873)

Add libclang to Dockerfile (#885)

Remove the tip distributor code (#888)

Rebase: Update anchor to not use deprecated crates

Add TLS webpki roots back in (#933)

Remove trusted relayer packets (#952)

Fix shred retransmit (#954)

Add daily v3.0 rebase (#972)

[Master] Automatically use optimal Block Engine region (#974)

Disable autoconfig (#995)

Make shredstream optional (#997)

Fix flaky test (#1027)
Remove bundle reservation (#1025)

Ensure execution-based failures are dropped (#1072)

Update programs to newest tip distribution + tip payment code (#1063)

Remove commented out code (#1074)

Remove unused bundle committer (#1091)

Fix bundle account locker TOCTOU (time of check time of use) bug (#1093)

fix unstaked xdp retransmit to block engine (#1090)

Bam Patch

This commit contains the contents migrated from the bam repository here: https://github.com/jito-labs/bam-client

Add BAM Connectivity Management Integration Tests (#1127)

[Master] Shoot down block engine connection if BAM enabled (#1139)

Fix priority graph TOCTOU (#1128)

Dynamic Client Id (#1138)

Set ClientId based on if it is actively connected to the BAM Node; with Jito-Solana being the default until a bam_connection exists.

Simple ping pong response implementation (#1145)

Problem

Implement pong response from the validator

Summary of Changes

This PR implements a simple ping-pong response mechanism for the BAM (Block Auction Market) connection system. The changes add handling for incoming ping messages by responding with pong messages, along with corresponding metrics tracking.

Changes:

Added ping message handling in the outbound message processing loop
Introduced three new metrics to track ping-related activity (ping_sent, outbound_ping_sent, outbound_ping_fail)
Updated the bam-protos subproject commit reference

Fix dockerfile warnings (#1174)

[master] Reduce BAM Scheduler allocs, remove lock contention in consume_work (#1161)

chore: fix cargo audit (#1184)

Propagate RUST_LOG to bam-local-cluster & use a valid merkle root upload authority (#1196)

Disconnect BAM when url is cleared (#1199)

feat: add support for multiple shred receivers (#1200)

Add documentation (#1214)

Fix insert_new_batch slab-capacity check (#1215)

Add a tag to scheduler metrics to distinguish between schedulers (#1198)

Clean up bam connection handling (#1213)

Reduce verbosity of slot logging  (#1234)

Stat improvement (#1225)

GRPC latency measurement (#1192)

Co-authored-by: haoranjito <haoran@jito.wtf>

Log new bam url (#1239)

Use better error codes for BAM connection (#1164)

fix ci secondary build  (#1240)

fix fetch stage manager (#1172)

Fix BAM fee-payer validation regression for bundled transactions. (#1236)

Proxy stage metric fix (#1244)

Small cleanups (#1268)

Use size hint (#1272)

skip secondary build for CI (#1273)

Bundle Lifecycle Management Cleanup (#1098)

Changes
- Remove unused solana-bundle-sdk
- Simplify insert_bundle callsite by removing method and chaining the recv_timeout with a try_recv
- Simplify BundleAccountLocker/BundleStorage management when processing bundles
- Simplify bundle result aggregation
- Cache static_instruction_limit
- Remove unused data structures

Some cleanup in FetchManager (#1281)

Bugfix: Fix tpu socket bug considering the offsetting math has changed in v4.X (#1261)

fix: suppress WARN log when --relayer-url is not configured (#1232)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Small cleanups (#1285)

Fix BAM enabled flag in update_tpu_config

Set ExecutionFlags values to revert_on_error

Disable incremental re-check inside BAM (#1334)

Multicast: send to DZ (#1355)

* Add MulticastShredCheckService

- Monitors /proc/net/route every 60s for multicast routes
- Mainnet: 233.84.178.1:7733, Testnet: 233.84.178.10:7733
- Automatically adds multicast address to shred receivers when route appears
- Automatically removes multicast address from shred receivers when route disappears
- Enabled on MainnetBeta and Testnet (controlled by disable_multicast_shred_check flag)
- Logs state changes once, no repeated messages while state is stable

* add

---------

Co-authored-by: Juan Olveira <juan@malbeclabs.com>

Bugfix - External Scheduler Account Checks (#1356)

* pipe blacklisted account into the external consume worker

* extract blacklist check logic to contains_blacklisted_account for use in both internal/external schduler

* add happy path test

* use arc for cloning the blisted accounts and minor cleaning

* format and sort

* update for cargo audit

* add quick little unit test

Update team references in .mergify.yml (#1363)

[Master] Support remote hosts connecting to local cluster (#1366)

add

[master] Defer BAM post-auth tasks until stream acceptance (#1368)

* add

* fmt

ci: Disable Dependabot version update PRs (#1393)

remove dependabot

feat: Add backport automation with GPG signing (#1386)

Fix: Adding SIGTERM handling for local cluster manager (#1396)

Adding SIGTERM handling for local cluster

Dynamically pinned PoH core in local cluster (#1397)

add

[improvement] adding --limit-ledger-size to cluster manager to limit RocksDB size i… (#1407)

Adding --limit-ledger-size to cluster manager to limit RocksDB size in local cluster mode

Fix race condition in admin RPC BAM URL updater registration (#1418)

* Register BAM identity updater before spawning manager

* fix test

[master] turbine: use a dedicated 0.0.0.0-bound socket for ShredReceiverAddresses (#1420)

add

Update shred receiver documentation (#1422)

* add

* fix

refactor (#1410)

[Master] Poh Core ordering (#1427)

add

[Master] Shred rework (#1431)

* Forward BAM shreds near leader windows

* dz fix

* shred cleanups

---------

Co-authored-by: haoranjito <haoran@jito.wtf>

[Master] More efficient retransmit for BAM (#1434)

gate BAM receivers to leader window start

Add program cache test (#1439)

add

feat: Add simulateTransaction extra fields to simulateBundle (#1443)

* feat: Add simulateTransaction extra fields to simulateBundle

* add replacementBlockhash

[master] Shred multicast update order (#1447)

feat(turbine): prioritize multicast receiver address in shred broadcast order

Move multicast_receiver_address to the front of external_receiver_addrs
so multicast shreds are sent before shred_receiver_addresses entries.

Co-authored-by: Juan Olveira <juan@malbeclabs.com>

- Increase BAM shred retransmit lookahead (#1455)
- fix(bundle): Don't unwrap priority fee cache for BundleStage (#1460)
- Fix client id to JitoLabs (#1461)
- [Master] Fix prio graph (#1464)
- [master] Allow local cluster startup (#1466)
- fix(ci): adding v4.1 to autorebase CI (#1458)
- Enable TLS Connections to BAM Node (#1467)
- [master] Add Multicast root receiver (#1469)
- ci(backport): Add v4.1 target branch (#1482)
- Ignore quinn-proto RustSec advisory (#1484)
- [codex] Run cargo audit at the end of CI (#1483)
- Clean up priority handling (#1479)
- Polling cleanup (#1480)
- Zercopy bam (#1481)
- BAM URL switch cleanups (#1487)
- Target slot adjustment (#1491)
- Add 4.2 to gh workflows (#1495)
- Use native CA roots (#1493)
- fix(bam-local-cluster): Enable dev utilities
- test(rpc): Use unique validator ports
- Refactor build pipeline to drive upstream tooling under BuildKit (#1441)
- Centralize Tonic Endpoint Creation (#1499)
- Local cluster no XDP (#1507)
- Configurable faucet mint sol for local cluster (#1508)
- fix(build): Clean up tagging mechanics (#1509)
- Cleanup BAM Connection (#1504)
- Simplify URL parsing (#1514)
- fix(bam): Fix flakey CI tests by signaling parsing thread shutdown (#1520)
- Disable strict_nonce_size_check in replay because its a leader-only check (#1525)
- Activate BAM local-cluster slot-time features (#1529)
- Fix bundle entry byte accounting (#1536)
- Reject IPv6 shred receiver addresses (#1535)
- Simplify bundle recording after SIMD-0083 (#1539)
- ci: Add v4.3 backport target (#1547)
- Avoid duplicate BAM transaction parsing (#1551)
- Resolve bundle ALTs against the root bank (#1542)
- Resanitize bundled transactions against execution bank (#1541)
- Fix Block Engine autoconfig candidate failover (#1553)
- Prevent BAM and external scheduler conflicts (#1546)
- Fix BAM Block Engine cutover (#1537)
- Deploy tip programs in local-cluster genesis and drop obsolete SIMD-406 test (#1588)
- Fail closed atomic transactions on sad handover (#1545)
- ci: exempt v4.3 from mergify EoL auto-close (#1593)
- Prevent stale tip-crank configuration across BAM/Block Engine transitions (#1561)
- feat: support tx v1 in bam receive and buffer (#1552)
- Fix compile fallout from upstream API drift
- Remove dead Jito artifacts (#1572)
- Prune unused Jito dependencies (#1573)
- Correct Jito live metrics (#1579)
- Share bundle consumer test fixtures (#1580)
- Consolidate and isolate bundle RPC integration tests (#1584)
- Isolate bank bundle simulation and tests (#1583)
- Minimize Jito SPL fetch overrides (#1576)
- Restore the program cache extract API (#1574)
- Restore banking stage compatibility APIs (#1578)
- Adapt restored extract and banking APIs to agave-master base
- Share Jito validator CLI arguments (#1575)
- Restore upstream broadcast packet selection (#1577)
- Share proxy auth refresh state (#1582)
- Isolate bundle RPC implementation and test (#1585)
- ci: Build and publish release artifacts on tag push (#1602)
- rpc: add filtered fan-out for bank notifications (#1554)
@0xEdgar
0xEdgar force-pushed the master branch 2 times, most recently from 683fd4e to 42be233 Compare September 1, 2026 16:52
Replace the dead force-push workflow (its PAT expired; every run since
2026-08-25 failed at checkout) with a config-driven rebase.

- master lands automatically after Buildkite passes, pushed with
  --force-with-lease so a merge during CI aborts the landing
- v4.2 and v4.3 refresh a bot-owned draft PR carrying the upstream delta
  and a range-diff of the carry commits for a human to land
- conflicts open or refresh an issue assigned to the channel owner
- short-lived GitHub App token per run; no third-party actions
- one Slack summary per run
@0xEdgar 0xEdgar changed the title ci(rebase): Stage nightly Agave draft PRs ci(rebase): Stage nightly Agave rebases with per-channel landing Sep 2, 2026
@0xEdgar
0xEdgar force-pushed the feat/mev-12307-nightly-rebase branch from 5ce0ad2 to dbb208c Compare September 2, 2026 15:52
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 2, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-09T22:03:09.699107Z 8a8aa3a New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dbb208ce3b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +352 to +355
if [[ -n "${staging_sha}" ]] &&
git diff --quiet "${staging_sha}" "${candidate_sha}" &&
git merge-base --is-ancestor "agave/${UPSTREAM_CHANNEL}" "${staging_sha}"; then
candidate_sha="${staging_sha}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Tie staging reuse to the current channel history

When the channel history changes without changing its final tree—for example, carry commit messages are amended or two new carry commits cancel each other's content—both checks still pass and candidate_sha is replaced with yesterday's staging SHA. In auto mode, the lease then confirms the new channel tip and force-pushes the older staging history over it; in draft mode, the suggested landing does the same. Record and compare the channel SHA or carry series from which the staging branch was produced rather than treating tree equality as sufficient.

Useful? React with 👍 / 👎.

: "${GITHUB_SERVER_URL:?GITHUB_SERVER_URL is required}"
: "${UPSTREAM_REPO:=https://github.com/anza-xyz/agave.git}"
: "${CI_CONTEXT:=buildkite/jito-solana}"
: "${CI_TIMEOUT_MINUTES:=240}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Refresh the App token during long CI waits

When Buildkite takes more than roughly an hour, this four-hour polling window outlives the App installation token minted once near the start of the job. GitHub documents that installation access tokens expire after one hour, so later status polls and the final authenticated push cannot complete even if CI succeeds. Mint and install a fresh token before landing, or constrain the wait to the token lifetime.

Useful? React with 👍 / 👎.

Comment on lines +369 to +371
git range-diff --no-color \
"$(git merge-base "origin/${CHANNEL}" "agave/${UPSTREAM_CHANNEL}")..origin/${CHANNEL}" \
"agave/${UPSTREAM_CHANNEL}..${candidate_sha}" >| "${range_diff_file}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Handle empty carry ranges before invoking range-diff

When a draft channel is simply behind upstream and has no Jito carry commits, both constructed ranges are empty; the same can happen if rebase skips every carry commit because upstream already contains the patches. git range-diff rejects an empty range with fatal: need two commit ranges, so the staging branch is pushed but the script exits before creating or refreshing its draft PR. Emit an explicit no-carry range-diff section instead of invoking this command when either range has no commits.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7e26a8996b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +8 to +9
# Secrets: REBASE_APP_ID, REBASE_APP_PRIVATE_KEY (GitHub App with contents,
# issues and pull-requests write), GPG_PRIVATE_KEY, GPG_PASSPHRASE (may be

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Grant the App commit-status read access

When the GitHub App is provisioned with only the documented contents, issues, and pull-request permissions, every non-fresh auto run aborts at the first status poll in wait_for_ci() rather than landing. GitHub's combined-status endpoint documentation requires the fine-grained Commit statuses repository permission with read access; add that permission to the App setup requirements.

Useful? React with 👍 / 👎.

Comment on lines +8 to +9
{
"channel": "v4.2",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restore v4.1 to the channel matrix

The replacement workflow silently stops scheduling nightly upstream rebases for v4.1: the deleted .github/workflows/rebase.yaml matrix included that branch, and the current .github/workflows/backport.yml still lists v4.1 as a backport target, indicating it remains maintained. Unless this branch is explicitly being retired, add a v4.1 channel entry so it does not stop receiving upstream updates.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b89dfca920

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +8 to +9
# Secrets: REBASE_APP_ID, REBASE_APP_PRIVATE_KEY (GitHub App with contents,
# issues and pull-requests write), GPG_PRIVATE_KEY, GPG_PASSPHRASE (may be

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Grant the App workflow-file write access

When a nightly rebase replays a carry commit that changes .github/workflows—including the commit introducing this workflow—the new rebased commit is pushed using the App token, and GitHub rejects workflow-file updates unless the App has the Workflows repository permission with write access. The documented App setup only grants Contents, Issues, and Pull requests, so the staging push can fail before Buildkite starts; add Workflows write to the required App permissions.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 520f6ff1e4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +341 to +344
if [[ -n "${staging_sha}" ]] &&
git diff --quiet "${staging_sha}" "${candidate_sha}" &&
git merge-base --is-ancestor "agave/${UPSTREAM_CHANNEL}" "${staging_sha}"; then
candidate_sha="${staging_sha}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Retrigger CI before reusing a statusless staging SHA

When the staging push fails to start Buildkite (for example, due to a lost webhook), rerunning the workflow takes this reuse path and performs no new push, so wait_for_ci polls the same statusless SHA for another four hours and times out again. This makes the documented ci_timeout recovery in dev/REBASE.md:106 ineffective and can leave an auto channel stuck until someone manually triggers Buildkite or deletes the staging ref; reuse the SHA only when a Buildkite status exists, or explicitly retrigger CI.

Useful? React with 👍 / 👎.

@0xEdgar
0xEdgar requested a review from ebin-mathews September 9, 2026 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants