fix(ci): drop RE max-jobs 16 to 8 after server-side crash

The first -j16 action (gcc-stage1) coincided with a full CAS/RE outage:
RST_STREAM INTERNAL on the action plus mass FetchBlob DEADLINE_EXCEEDED
on pulls at 23:20 UTC. Likely server OOM under 2x16 compile jobs plus
casd cache pressure on the 128 GB builder. 2 builders x -j8 keeps an 8x
speedup over the old -j1 while staying under the RAM ceiling.

Assisted-by: Claude Fable 5 via GitHub Copilot

Author: castrojo

.github/actions/generate-bst-ci-config/action.yml

@@ -44,16 +44,18 @@ runs:
             cat >> buildstream-ci.conf <<'BSTCONFBUILD'
         build:
           retry-failed: True
-          # 16 = physical cores on the RE builder (7950X3D, 16c/32t, 128 GB).
-          # With builders: 2, two concurrent actions saturate all 32 threads.
           # max-jobs does NOT affect cache keys (buildelement.py: "normally
           # automatically resolved and does not affect the cache key"), so
-          # raising it preserves the warm CAS. The old max-jobs: 1 (GCC
-          # bootstrap OOM on gimple-match.cc) is obsolete: the -O1 patch
-          # fixed the ICE and gcc is already cached; -j1 made every big
-          # element (llvm, WebKit x2) a 10-30h single-core compile that
-          # mathematically busted the 480m budget.
-          max-jobs: 16
+          # tuning it preserves the warm CAS. History: max-jobs: 1 made every
+          # big element (llvm, WebKit x2) a 10-30h single-core compile that
+          # mathematically busted the 480m budget (11-day outage). The first
+          # attempt at 16 coincided with a server-wide outage (RST_STREAM
+          # INTERNAL + FetchBlob DEADLINE_EXCEEDED on all pulls, 23:20 UTC
+          # 2026-07-09) — likely OOM: gimple-match.cc compiles eat 4-6 GB
+          # each, and 16 in parallel plus casd cache pressure exceeds the
+          # 128 GB builder. 2 builders x -j8 is a safe 8x speedup over -j1
+          # while staying well under the RAM ceiling.
+          max-jobs: 8
         BSTCONFBUILD
           else
             cat >> buildstream-ci.conf <<'BSTCONFBUILD'

docs/skills/ci.md

@@ -2463,3 +2463,28 @@ element-scoped fix (variables in a repo patch), never a global -j1.
 
 **Changing max-jobs mid-outage costs nothing:** already-built elements pull from the
 artifact cache by BST cache key; only not-yet-built elements get new RE action digests.
+
+### max-jobs: 16 crashed the RE server — 8 is the ceiling (2026-07-09)
+
+The first run at `max-jobs: 16` died at ~23:20 UTC: gcc-stage1.bst, 28 minutes into
+its remote action, got `INTERNAL: Stream removed (Received RST_STREAM with error
+code 2)`, and simultaneously every artifact pull started failing with `FetchBlob
+failed with status DEADLINE_EXCEEDED`. The whole CAS/RE endpoint at
+cache.projectbluefin.io:11002 stopped serving mid-run, then recovered on its own
+(TLS handshake succeeded again minutes after the run failed).
+
+**Diagnosis:** likely server-side OOM. gimple-match.cc and friends eat 4-6 GB per
+compiler process; 16 parallel jobs plus buildbox-casd's own cache pressure exceeds
+the 128 GB builder, the worker (or the whole host) OOMs, and every open gRPC stream
+gets RST_STREAM'd. From the client all you see is one build failure plus mass
+DEADLINE_EXCEEDED on unrelated pulls — the correlated timing is the tell.
+
+**Signature to recognize:** one RE action fails with `Stream removed / RST_STREAM
+error code 2` AND pulls fail en masse with `FetchBlob ... DEADLINE_EXCEEDED` at the
+same timestamp = server-side crash/restart, not a client or network problem. Probe
+recovery with `echo | openssl s_client -connect cache.projectbluefin.io:11002`
+(no SSH access to the server exists from CI or dev machines).
+
+**Setting:** `max-jobs: 8` with `builders: 2`. Still 8x the old -j1 throughput,
+peak compile RAM ~2x8x6 GB = ~96 GB worst case, under the ceiling. Do not raise
+back to 16 without server-side memory monitoring in place.