docs(skills): record CAS hardware, fetcher count, log API cap, build timing

- ci-reference: add AX102-U CAS server specs (1Gbit/s, NVMe Gen4) and
  fetcher count reasoning (32 per job justified by hardware)
- ci-reference: correct log API cap from ~23K to ~11K lines; note GHA UI
  as the only way to see live progress monitor output mid-build
- buildstream: document why warm-cache builds still take 90-120 min
  (pull volume, two parallel variants, sequential OCI assembly, cold elements)
- buildstream: document 32 fetchers as correct for AX102-U CAS
- workflow-map: update paths-ignore entries to include files/scripts/**

Assisted-by: Claude Sonnet 4.6 via GitHub Copilot

Author: castrojo

docs/skills/buildstream.md

@@ -133,3 +133,18 @@ BST option names only allow alphanumeric characters and underscores. A name like
 ### Weak-key caching can hide new packages behind a clean build (2026-06-07)
 
 Changing a `kind: stack` dependency does not always invalidate downstream `compose` outputs in non-strict mode. If a package is present in the graph but missing from the final image, inspect cache behavior before assuming the package element is wrong.
+
+### Warm-cache builds still take 90-120 min — this is normal (2026-06-23)
+
+Even with a fully warm remote CAS, a full build takes 90-120 min. Common misconception: "cache is hot = fast build." Actual breakdown:
+
+- **Pull volume:** ~1,400 elements × a few seconds each / 32 parallel fetchers = 15-30 min just for network pulls
+- **Two parallel jobs:** `default` and `nvidia` both run simultaneously, each hitting the same CAS endpoint, halving effective bandwidth per job
+- **OCI assembly is sequential:** After all elements pull/build, `oci/bluefin.bst` runs chunkify + image assembly — single-threaded, typically 20-40 min on its own
+- **Cold elements:** Any junction ref bump (Renovate PRs for distrobox, gnome-build-meta, etc.) invalidates those subtrees → full recompile from source adds 30-90 min
+
+Do not cancel a build under 120 min just because it "seems slow." Historical range for successful builds: 90-150 min.
+
+### 32 fetchers is the right setting for cache.projectbluefin.io (2026-06-23)
+
+`buildstream-ci.conf` uses `fetchers: 32` (BST default is 10). With default + nvidia running simultaneously = 64 concurrent gRPC streams. The CAS server is a Hetzner AX102-U (1 Gbit/s uplink, NVMe Gen4) and can serve 64 streams comfortably. The bottleneck is network bandwidth (~125 MB/s total), not server capacity. Do not reduce fetchers without evidence of server-side saturation.

docs/skills/ci-reference.md

@@ -83,6 +83,14 @@ Route through `ci.md` first, then come here only when the focused skills do not
 
 `cache.projectbluefin.io:11002` handles all five BST remote services: artifact cache, source cache, CAS storage, remote execution, and action cache. All use the same endpoint with mTLS auth.
 
+**CAS server hardware (Hetzner AX102-U):**
+- CPU: AMD Ryzen 9 7950X3D (16c/32t, 4.2 GHz base)
+- RAM: 128 GB DDR5
+- Storage: 2 × 1.92 TB NVMe SSD Gen4
+- Network: 1 Gbit/s dedicated uplink (unlimited traffic)
+
+**Fetcher count reasoning:** `buildstream-ci.conf` uses `fetchers: 32` per job. With default + nvidia running simultaneously = 64 concurrent gRPC streams. The AX102-U can comfortably serve 64 streams; the actual ceiling is the 1 Gbit/s uplink (~125 MB/s total). 32 fetchers per job is appropriate. Do not lower without evidence of CAS-side saturation.
+
 ### mTLS Authentication
 
 | Variable | Type | Content |
@@ -578,7 +586,7 @@ for job in d.get('jobs', []):
         print(f\"{job['id']} | {job['status']} | {mins}m | {job['name'][:60]}\")
 "
 
-# 2. Fetch the live log (note: truncated at ~23K lines for long builds)
+# 2. Fetch the live log (note: truncated at ~11K lines = first 2-5 min of dense BST output)
 gh api repos/projectbluefin/dakota/actions/jobs/<job-id>/logs > /tmp/bst-live.log
 
 # 3. Count cache hits vs elements being compiled
@@ -589,11 +597,19 @@ grep "Running commands" /tmp/bst-live.log | tail -20  # what's actively building
 grep "START.*Running commands" /tmp/bst-live.log | grep -oE "\[.*\]" | sort -u
 ```
 
-**Important:** The live log endpoint is a snapshot, not a stream. For builds
-running > ~90 minutes, the log may be stale by 60–90 minutes relative to current
-wall-clock time. If the last log timestamp is behind by > 1 hour, the build is
-still running but log data is not being returned. Use `gh api
-repos/.../actions/runs/<id>/jobs` to confirm `status: in_progress`.
+**Important:** The live log endpoint is a snapshot, not a stream. The API caps at
+**~11K lines** which covers only the first 2-5 minutes of dense BST output — for a
+2-hour build you see essentially nothing useful after the first few minutes. This
+is a GitHub API limitation, not a BST issue.
+
+**To see progress monitor output mid-build:** Open the GHA UI directly:
+`https://github.com/projectbluefin/dakota/actions/runs/<run-id>` — the
+`bst-progress.py` monitor emits lines every 30 seconds in the live log stream
+viewable in browser. The API endpoint does NOT stream these.
+
+For builds running > ~90 minutes, the API log timestamp will be far behind
+wall-clock. Use `gh api repos/.../actions/runs/<id>/jobs` to confirm
+`status: in_progress`.
 
 **Deciding whether to re-trigger:** A build making steady progress on
 gnome-build-meta `core-deps/` elements is normal cache-warming after a GNOME

docs/skills/workflow-map.md

@@ -78,8 +78,8 @@ push: testing/main (BST paths) / Tuesday 04:00 UTC / manual   ← PARALLEL, DECO
 
 | Workflow | Owns | Normal trigger |
 |---|---|---|
-| `.github/workflows/build.yml` | BST build into remote CAS | `push: main/next/testing` (paths-ignore: docs, workflows, md), `merge_group`, `workflow_dispatch`. `validate` job runs on `pull_request` only; `build` job skips `pull_request`. |
-| `.github/workflows/build-aarch64.yml` | aarch64 OCI build + GHCR push | `push: main/testing` (same paths-ignore as build.yml), `schedule: Tuesday 04:00 UTC`, `workflow_dispatch`. Fully decoupled — never in `needs:` of publish/promote/release. |
+| `.github/workflows/build.yml` | BST build into remote CAS | `push: main/next/testing` (paths-ignore: docs, workflows, md, `files/scripts/**`), `merge_group`, `workflow_dispatch`. `validate` job runs on `pull_request` only; `build` job skips `pull_request`. |
+| `.github/workflows/build-aarch64.yml` | aarch64 OCI build + GHCR push | `push: main/testing` (same paths-ignore as build.yml including `files/scripts/**`), `schedule: Tuesday 04:00 UTC`, `workflow_dispatch`. Fully decoupled — never in `needs:` of publish/promote/release. |
 | `.github/workflows/publish.yml` | export, sign, boot-check, promote tags | `workflow_run` from build |
 | `.github/workflows/publish-smoke.yml` | observational smoke only | `workflow_run` from publish |
 | `.github/workflows/e2e.yml` | PR-facing testsuite check | `pull_request` |