Merge remote-tracking branch 'origin/main' into claude/aocc-infra

Author: mattgodbolt

.claude/commands/update-beman-tags.md

@@ -0,0 +1,115 @@
+Discover newly-released git tags on `bemanproject/*` repos and propose
+infra + compiler-explorer changes to expose them.
+
+The infra repo (this one) hosts `bin/yaml/libraries.yaml`. The
+compiler-explorer repo lives in a sibling directory (look around
+`../compiler-explorer` or `../compiler-explorer-*`); its config file is
+`etc/config/c++.amazon.properties`. If you can't find a sibling clone,
+ask the user to point at the right path before continuing.
+
+## Where things live
+
+Both groups now live under shared `beman:` parents that declare common
+settings (`build_type`, `check_file`, `target_prefix`, `type`, and a
+templated `repo: bemanproject/{{ context[-1] | replace('beman_', '', 1) }}`):
+
+- **Tagged versions** — `libraries.c++.beman.beman_<lib>` (no `if:`,
+  default archive method).
+- **Trunk** — `libraries.c++.nightly.beman.beman_<lib>` (inherits
+  `if: nightly` and `method: nightlyclone` from the surrounding
+  `nightly:` group).
+
+Most leaves only declare `targets:`. Genuine overrides today:
+`beman_iterator_interface` (cmake / staticliblink / package_install)
+and `beman_scope` (`target_prefix: ''` because its only tag is
+`0.0.1`). Per-target `target_prefix: ''` is also used for individual
+non-`v`-prefixed tags within otherwise-`v`-prefixed libraries
+(e.g. `beman_any_view`'s `1.0.0`, several `beman_exemplar` versions).
+
+## What to do
+
+1. **Enumerate beman libraries.** The authoritative list is the keys
+   under `libraries.c++.nightly.beman` in `bin/yaml/libraries.yaml`
+   (it's a superset, including repos that have no tags yet). Don't
+   hard-code names.
+2. **Fetch tags via `gh api`** for each repo, e.g.
+   `gh api repos/bemanproject/<name>/tags --jq '.[].name'`. Skip repos
+   with no tags. Exclude pre-release / rc / alpha / beta tags unless
+   the user asks otherwise.
+3. **Compare against what's already configured.** Look at the keys and
+   targets under `libraries.c++.beman` to see which `(target_prefix +
+   target_name)` strings are already present, and compute the diff
+   against the GitHub tag list.
+4. **If there are no new tags, stop and report that to the user.** Do
+   not open empty PRs.
+5. **Add the new tagged versions** under `libraries.c++.beman`. Match
+   the existing style:
+   - If the lib doesn't have an entry yet, add a child whose only key
+     is `targets:`. Common settings are inherited from the parent.
+   - When all of a lib's tags are `v`-prefixed, list bare versions
+     under `targets:` and rely on the parent's `target_prefix: v`.
+   - For tags missing the `v`, add a per-target dict with
+     `target_prefix: ''`:
+     ```yaml
+     - name: 1.0.0
+       target_prefix: ''
+     ```
+   - When **none** of a lib's tags are `v`-prefixed, override at the
+     entry level with `target_prefix: ''` (see `beman_scope`).
+   - `beman_iterator_interface` is the only built lib; preserve its
+     `cmake` / `lib_type: static` / `staticliblink` / `package_install`
+     overrides if you're adding a new tag for it.
+   - Don't introduce `method: nightlyclone` here; the default `archive`
+     method is correct for tagged versions.
+6. **Validate locally** before committing:
+   ```bash
+   bin/ce_install list 'libraries/c++/beman/**'
+   bin/ce_install --dry-run --dest ~/opt/compiler-explorer install \
+       'libraries/c++/beman/beman_<lib> <ver>'
+   ```
+   For at least the new tags, run a real install into
+   `~/opt/compiler-explorer` and confirm the install path
+   (`libs/beman_<lib>/<target_prefix><ver>/`) contains the expected
+   `include/`. Note the install dir uses `target_prefix + target_name`,
+   so a tag `v0.1.0` installs at `libs/beman_<lib>/v0.1.0/` and a
+   bare `0.0.1` tag installs at `libs/beman_<lib>/0.0.1/`.
+7. **Update the compiler-explorer config.** Open
+   `etc/config/c++.amazon.properties` in the sibling clone and extend
+   each affected `libs.beman_<lib>.versions=...` list. Use the same
+   compact version key style the file already uses (e.g. `0.1.0` →
+   `010`, `2.1.1` → `211`). The `path` entry must match the install
+   directory created by `ce_install`, including the `v` prefix when
+   present (so `0.1.0` from a `v0.1.0` tag becomes
+   `.../beman_<lib>/v0.1.0/include`, while `0.0.1` from a bare
+   `0.0.1` tag becomes `.../beman_<lib>/0.0.1/include`).
+8. **Run `npm run test:props`** in the compiler-explorer repo to make
+   sure the property file is still valid.
+9. **Open two PRs**, one in each repo, on a fresh branch. Match the
+   style of the existing PRs (see compiler-explorer/infra#2111 and
+   compiler-explorer/compiler-explorer#8683):
+   - Infra branch first, then reference its PR number from the
+     compiler-explorer PR description.
+   - Don't bypass pre-commit hooks. Terraform-related hook failures on
+     a workstation without `terraform` installed are environmental, not
+     a real failure — but everything else must pass.
+10. **Report back** the full list of tags added per library and the two
+    PR URLs so the user can review.
+
+## Things to skip
+
+- Repos with zero tags (e.g. `beman_execution`, `beman_map`,
+  `beman_net`, `beman_span`, `beman_task` at time of writing). Their
+  `libraries.c++.nightly.beman.<name>` trunk entries stay; nothing
+  needs to be added under `libraries.c++.beman`.
+- Pre-release / rc / alpha / beta tags.
+- Re-running the discovery for repos that already have the latest tag
+  configured.
+
+## Notes
+
+- Pre-commit will run tests and lint. `make pre-commit` (infra) and
+  `npm run test:props` (compiler-explorer) must be clean.
+- Don't touch `c++.amazonwin.properties` unless the user asks — the
+  Windows path for these libs hasn't been wired up.
+- Don't touch the `libraries.c++.nightly.beman.*` entries; their
+  `main` trunk install is independent of tagged releases.

.github/workflows/bespoke-build.yaml

@@ -30,6 +30,7 @@ on:
         - iwyu
         - jakt
         - lfortran
+        - lua
         - mads
         - micropython
         - nasm
@@ -40,7 +41,6 @@ on:
         - pythran
         - rocm
         - ruby
-        - rust-cg-gcc
         - rust-linux
         - vala
       version:

.github/workflows/cefs-gc.yml

@@ -21,7 +21,7 @@ jobs:
     runs-on: ['admin']
     steps:
       - name: Clean workspace
-        run: find "$GITHUB_WORKSPACE" -mindepth 1 -delete
+        run: sudo find "$GITHUB_WORKSPACE" -mindepth 1 -delete
       - uses: actions/checkout@v5
       - name: Set up environment
         run: make ce

.github/workflows/lin-lib-build.yaml

@@ -11,6 +11,10 @@ on:
                 description: 'Compiler'
                 default: 'popular-compilers-only'
                 required: true
+            branch:
+                description: 'infra branch to clone (defaults to main)'
+                default: 'main'
+                required: false
     workflow_call:
         inputs:
             library:
@@ -23,6 +27,11 @@ on:
                 default: 'popular-compilers-only'
                 required: true
                 type: string
+            branch:
+                description: 'infra branch to clone (defaults to main)'
+                default: 'main'
+                required: false
+                type: string
 
 jobs:
     librarybuild:
@@ -56,10 +65,32 @@ jobs:
 
           echo "language=$LANGUAGE" >> $GITHUB_OUTPUT
           echo "library_name=$LIBRARY_NAME" >> $GITHUB_OUTPUT
+          # actions/upload-artifact rejects forward slashes in artifact names, so
+          # provide a sanitized variant for use in the failure-artifact step.
+          echo "library_safe=${LIBRARY_NAME//\//-}" >> $GITHUB_OUTPUT
           echo "Detected language: $LANGUAGE, library: $LIBRARY_NAME"
 
       - name: Download build script
-        run: curl -sL "https://raw.githubusercontent.com/compiler-explorer/infra/refs/heads/main/init/start-builder.sh" -o "/tmp/start-builder.sh"
+        run: curl -sL "https://raw.githubusercontent.com/compiler-explorer/infra/refs/heads/${{ inputs.branch || 'main' }}/init/start-builder.sh" -o "/tmp/start-builder.sh"
 
       - name: Build libraries
-        run: cd /tmp && chmod +x start-builder.sh && ./start-builder.sh "${{ secrets.CONAN_PASSWORD }}" "${{ steps.extract.outputs.language }}" "${{ steps.extract.outputs.library_name }}" "${{ inputs.compiler }}"
+        run: cd /tmp && chmod +x start-builder.sh && ./start-builder.sh "${{ secrets.CONAN_PASSWORD }}" "${{ steps.extract.outputs.language }}" "${{ steps.extract.outputs.library_name }}" "${{ inputs.compiler }}" "${{ inputs.branch || 'main' }}"
+
+      - name: Upload build failure artifacts
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: build-failure-${{ inputs.compiler }}-${{ steps.extract.outputs.library_safe }}-${{ github.run_id }}
+          path: |
+            /tmp/*.cpp
+            /tmp/*.ii
+            /tmp/*.c
+            /tmp/*.s
+            /tmp/*.sh
+            /tmp/ce-cefs-temp/staging/**/cemakelog*.txt
+            /tmp/ce-cefs-temp/staging/**/cecmakelog.txt
+            /tmp/ce-cefs-temp/staging/**/ceconfiglog.txt
+            /tmp/ce-cefs-temp/staging/**/cemakeclean.txt
+            /tmp/ce-cefs-temp/staging/**/ceinstall_*.txt
+          if-no-files-found: warn
+          retention-days: 14

.github/workflows/package-fxc.yaml

@@ -0,0 +1,16 @@
+name: Package FXC
+run-name: Package FXC compilers from the Windows SDK
+
+on:
+  workflow_dispatch:
+
+jobs:
+  package-fxc:
+    runs-on: windows-latest
+    steps:
+      - uses: actions/checkout@v5
+      - shell: pwsh
+        run: .\msvc-install\fxc-install.ps1
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

.github/workflows/start_staging.yml

@@ -18,8 +18,6 @@ jobs:
 
     steps:
     - name: Update infra
-      run: cd /home/ubuntu/infra && git pull
-    - name: Set build number
-      run: ce builds set_current ${{ github.event.inputs.buildnumber }} --branch ${{ github.event.inputs.branch }} --confirm
-    - name: Start environment
-      run: ce environment start
+      run: cd /home/ubuntu/infra && git pull && make ce
+    - name: Blue-green deploy to staging
+      run: ce --env staging blue-green deploy --branch ${{ github.event.inputs.branch }} --skip-confirmation --no-notify ${{ github.event.inputs.buildnumber }}

.gitignore

@@ -18,3 +18,7 @@ node_modules
 /uv.lock
 /lambda/.build_site_packages
 *.egg-info
+
+# Claude Code runtime/per-user state (committed config lives in .claude/agents, .claude/commands, etc.)
+.claude/scheduled_tasks.lock
+.claude/settings.local.json

.pre-commit-config.yaml

@@ -13,14 +13,14 @@ repos:
       - id: check-toml
       - id: detect-private-key
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.12.9
+    rev: v0.15.15
     hooks:
       - id: ruff
         args: [ --fix, --preview ]
       - id: ruff-format
         args: [ --preview ]
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v1.17.1
+    rev: v2.1.0
     hooks:
       - id: mypy
         additional_dependencies:
@@ -35,8 +35,10 @@ repos:
     rev: 3.0.0
     hooks:
       - id: shellcheck
+        # severity is CLI-only (not a valid .shellcheckrc directive); gate on errors.
+        args: ["--severity=error"]
   - repo: https://github.com/antonbabenko/pre-commit-terraform
-    rev: v1.100.0
+    rev: v1.106.0
     hooks:
       - id: terraform_fmt
       - id: terraform_validate

.shellcheckrc

@@ -1,3 +1,2 @@
 disable=SC2164  # cd and popd
 disable=SC2043  # loop will only execute once
-severity=error

CLAUDE.md

@@ -23,6 +23,10 @@ When writing, especially PRs and commit messages:
 - Build events lambda package: `make events-lambda-package`
 - **NEVER USE THE SYSTEM PYTHON** - always use `uv` to invoke python or pytest or to run experiments with python syntax
 
+## Documentation
+
+This repository has extensive documentation in the `docs/` directory. Before making changes to a subsystem, read the relevant documentation first. Existing CLI commands, library configuration, and infrastructure patterns are already documented -- don't reinvent or guess at behavior that's written down.
+
 ## Important Workflow Requirements
 
 - ALWAYS run pre-commit hooks before committing: `make pre-commit`
@@ -207,6 +211,16 @@ The CLI system (`bin/ce`) uses Click framework with a modular command structure:
        """Subcommand description."""
    ```
 
+## Admin Instance
+
+The admin instance runs fish as its default shell, so `ce` is not in its path. Use the full path when running commands remotely:
+
+```bash
+bin/ce admin exec /home/ubuntu/infra/bin/ce conan restart
+```
+
+After restarting the conan proxy, wait ~10 seconds before making requests to it -- it returns 502 while starting up.
+
 ## GitHub Workflow Integration
 
 The `ce workflows` command group provides functionality to trigger GitHub Actions workflows:
@@ -602,7 +616,7 @@ Library YAML settings, build types, library types (`cshared`, `shared`, `static`
 
 The `ce_install build-status` command group manages build failure records on the Conan proxy server. Failed builds are tracked so they are not re-attempted; these commands allow clearing that status.
 
-- **`ce_install build-status list-failed`** - List failed builds (requires at least one of `--library` or `--compiler-version`)
+- **`ce_install build-status list-failed`** - List failed builds (requires at least one of `--library` or `--compiler-version`; optional `--version` to filter by library version)
 - **`ce_install build-status clear-for-library LIBRARY [--version VERSION]`** - Clear failures for a library
 - **`ce_install build-status clear-for-compiler COMPILER_VERSION`** - Clear failures for a compiler (e.g. `g141`, `clang1400`)