ci: rebalance the e2e and shreds-e2e shard matrices (#4249)

## Summary

- The e2e matrix runs 5 round-robin shards instead of 4. Shards are
filled round-robin by test *count*, not by duration, so which shard gets
the slow tests is luck. The heaviest shard was carrying ~3735s of tests
against a 15-minute job timeout and was cancelled partway through, while
the lightest carried 2840s. A fifth shard brings the worst case back to
~3022s.
- shreds-e2e pins one heavy test to its own shard instead of three.
`TestE2E_MultiUserInstantAllocationAndWithdrawal` and
`TestE2E_DeviceScale` no longer exist in doublezero-shreds, so the pin
validation failed on every run and the matrix was never built — the
workflow has been red on main since 05fb0ab. Only
`TestE2E_FeedSubscriptionOracleExpiryTeardown` stays pinned, leaving 1
pinned + 2 round-robin shards.
- `CHECK_SHARDS` follows the matrix in both workflows, so the docs-only
skip path reports exactly the contexts the matrix creates.

## Required ruleset changes

`CHECK_SHARDS` must match the required status check contexts in the main
ruleset, so merging this needs two manual admin steps:

- **add** `e2e (shard 6)`
- **remove** `shard-e2e (shard 4)` and `shard-e2e (shard 5)` — until
these are removed they are required but never reported, which blocks
merges.

## Testing Verification

- Measured total test-seconds per shard from CI logs to find the cliff.
Shards at 3568s, 3735s and 3783s hit the 15-minute timeout; shards at
2840s, 3428s and 3507s passed. main itself was already running at
3012-3318s, leaving almost no headroom for any new test.
- Verified the 6-shard e2e layout on a branch: work per shard landed at
2214-3022s and every shard finished in 5.5-9.4 minutes, against a
15-minute timeout.
- Confirmed the shreds-e2e pin validation failure reproduces on main and
that `TestE2E_FeedSubscriptionOracleExpiryTeardown` is present in the
upstream discovery output.

Author: elitegreg

.github/workflows/e2e.yml

@@ -28,7 +28,7 @@ concurrency:
   cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 env:
-  SHARD_COUNT: "4"
+  SHARD_COUNT: "5"
   # Use registry-qualified image names so we can push/pull between jobs
   DZ_IMAGE_REPO: ghcr.io/malbeclabs/dz-e2e
   DZ_IMAGE_TAG: ${{ github.event.inputs.image_tag || github.sha }}
@@ -153,7 +153,7 @@ jobs:
           CHECK_NAME: e2e
           # SHARD_COUNT round-robin shards + 1 dedicated shard; must match the
           # required status check contexts in the main ruleset.
-          CHECK_SHARDS: "5"
+          CHECK_SHARDS: "6"
         with:
           script: |
             try {

.github/workflows/shreds-e2e.yml

@@ -55,9 +55,9 @@ jobs:
         uses: actions/github-script@v7
         env:
           CHECK_NAME: shard-e2e
-          # 3 pinned shards + 2 round-robin shards; must match the required
+          # 1 pinned shard + 2 round-robin shards; must match the required
           # status check contexts in the main ruleset.
-          CHECK_SHARDS: "5"
+          CHECK_SHARDS: "3"
         with:
           script: |
             try {
@@ -192,28 +192,26 @@ jobs:
           echo "Discovered $count tests"
           echo "$tests"
 
-          # Pinned heavy tests: each gets its own shard slot to keep the
-          # biggest rocks off the same runner. Update this list if a new test
-          # exceeds ~5min, or if a pinned test is renamed/removed upstream
+          # Pinned heavy test: it gets its own shard slot to keep the biggest
+          # rock off a shared runner. Add to this list if a new test exceeds
+          # ~5min, or update it if a pinned test is renamed/removed upstream
           # (the validation below will fail fast in that case).
-          pinned_1="TestE2E_MultiUserInstantAllocationAndWithdrawal"
-          pinned_2="TestE2E_DeviceScale"
           # ~9min, over half of it a single 40-day ledger warp (three validator
           # restart hops). On a round-robin shard it only passed when Go's
           # parallel scheduler happened to start it early.
-          pinned_3="TestE2E_FeedSubscriptionOracleExpiryTeardown"
+          pinned_1="TestE2E_FeedSubscriptionOracleExpiryTeardown"
 
           # Fail fast if a pinned test no longer exists in the shreds repo —
           # otherwise its shard would silently run zero tests.
-          for pin in "$pinned_1" "$pinned_2" "$pinned_3"; do
+          for pin in "$pinned_1"; do
             if ! echo "$tests" | grep -qxF "$pin"; then
               echo "::error::Pinned test '$pin' not found in doublezero-shreds. Update the pin list in this workflow."
               exit 1
             fi
           done
 
-          # Remaining tests round-robin across shards 4 and 5.
-          remaining=$(echo "$tests" | grep -vE "^(${pinned_1}|${pinned_2}|${pinned_3})$")
+          # Remaining tests round-robin across shards 2 and 3.
+          remaining=$(echo "$tests" | grep -vE "^(${pinned_1})$")
 
           ROUND_ROBIN_SHARDS=2
           declare -a shards
@@ -229,12 +227,10 @@ jobs:
             i=$((i + 1))
           done <<< "$remaining"
 
-          # Build JSON matrix: shards 1-3 are pinned, shards 4-5 are round-robin.
+          # Build JSON matrix: shard 1 is pinned, shards 2-3 are round-robin.
           matrix="[{\"shard\":1,\"run\":\"^(${pinned_1})$\"}"
-          matrix="${matrix},{\"shard\":2,\"run\":\"^(${pinned_2})$\"}"
-          matrix="${matrix},{\"shard\":3,\"run\":\"^(${pinned_3})$\"}"
           for ((i=0; i<ROUND_ROBIN_SHARDS; i++)); do
-            matrix="${matrix},{\"shard\":$((i + 4)),\"run\":\"^(${shards[i]})$\"}"
+            matrix="${matrix},{\"shard\":$((i + 2)),\"run\":\"^(${shards[i]})$\"}"
           done
           matrix="${matrix}]"
 

CHANGELOG.md

@@ -12,6 +12,8 @@ All notable changes to this project will be documented in this file.
 - SDK
   - The TypeScript and Python `GlobalState` deserializers expose `ip_verifier_authority_pk`, the RFC-27 trust root the Go SDK and the Rust state already carried, so those consumers can read which key signs IP ownership proofs. The field is appended, so an account written before the upgrade decodes it as the default pubkey rather than failing. (#4231)
 - CI
+  - The e2e matrix runs 5 round-robin shards instead of 4. Shards are filled by test count, not by duration, so the heaviest one was carrying ~3735s of tests against a 15-minute job timeout and was cancelled mid-run; the extra shard brings the worst case back to ~3022s. Adding `e2e (shard 6)` to the required status checks in the main ruleset is a separate, manual step.
+  - shreds-e2e pins one heavy test to its own shard instead of three. `TestE2E_MultiUserInstantAllocationAndWithdrawal` and `TestE2E_DeviceScale` no longer exist in doublezero-shreds, so the pin validation failed every run and the matrix was never built. Only `TestE2E_FeedSubscriptionOracleExpiryTeardown` stays pinned, leaving 1 pinned + 2 round-robin shards. Dropping `shard-e2e (shard 4)` and `shard-e2e (shard 5)` from the required status checks in the main ruleset is a separate, manual step — until it happens those contexts are required but never reported.
   - `.cursor/BUGBOT.md` and `.github/copilot-instructions.md` now tell Bugbot and Copilot to read the nearest sibling, flag a path that skips a zero or a duplicate, and assert a specific error and the exact log line at the expected index. Onchain checks apply only when the repository has onchain code. The eight path-scoped files under `.github/instructions/` are removed so Copilot reads only the repo-wide file. (#4247)
 - E2E/QA
   - Remove `TestQA_MulticastSettlement`. It funded a seat through `doublezero-solana shreds pay`, which is going away. The agent seat-pay RPC now returns Unimplemented if something still calls it. Unused settlement helpers go with the test. (#4248)