macOS: APFS clonefile fast path + concurrency cap + zero-byte fix for Bazel input materialization

.bazelrc

@@ -34,6 +34,19 @@ common --@rules_python//python/config_settings:py_linux_libc=musl
 # Don't leak PATH and LD_LIBRARY_PATH into the build.
 build --incompatible_strict_action_env
 
+# hermetic-llvm <-> rules_rs interop. These are harmless when LRE-CC is the
+# active CC toolchain (Nix path) and required when hermetic-llvm provides it
+# (non-Nix path):
+#   - Rust passes -lgcc_s when linking, so libgcc_s must be stubbed.
+#   - cc-rs doesn't honor $AR/$ARFLAGS, so disable llvm-libtool-darwin as the
+#     default macOS archiver.
+# Musl + PIE harmonization (Rust forces -no-pie; hermetic-llvm defaults to
+# -static-pie) is a follow-up: it requires adding @llvm//constraints/pie:off
+# to the lre-rs musl platforms, which are @generated and need a generator-side
+# change.
+build --@llvm//config:experimental_stub_libgcc_s=True
+build --@rules_cc//cc/toolchains/args/archiver_flags:use_libtool_on_macos=False
+
 # Don't use legacy repository rules.
 build --incompatible_disable_native_repo_rules
 

.bazelversion

@@ -1 +1 @@
-8.2.1
+9.0.2

.claude/skills/nativelink-bazel-verification/SKILL.md

@@ -0,0 +1,70 @@
+---
+name: nativelink-bazel-verification
+description: Use when verifying NativeLink changes with Bazel, selecting focused test targets, debugging rustfmt/clippy aspect failures, or preparing a confidence report for Rust/Bazel changes.
+---
+
+# NativeLink Bazel Verification
+
+Use this skill when a NativeLink change needs a focused Bazel build/test plan, when CI reports a Bazel failure, or when you need to decide how much verification is enough.
+
+## Start With Scope
+
+1. Inspect the touched files and nearby `BUILD.bazel` files.
+2. Map changed Rust files to their crate package:
+   - `nativelink-store/` for storage backends, CAS/AC, compression, Redis/S3/GCS/filesystem/memory stores.
+   - `nativelink-scheduler/` for action scheduling and worker matching.
+   - `nativelink-worker/` for local worker execution and action lifecycle.
+   - `nativelink-service/` for REAPI, bytestream, BEP, and worker API services.
+   - `nativelink-config/` for JSON5 configuration structs and examples.
+   - `nativelink-util/` for shared helpers.
+   - `nativelink-proto/` for protobuf-generated surfaces.
+3. Prefer focused package tests first, then broaden only when the touched surface is shared.
+
+## Common Commands
+
+Use the repository's Bazel defaults unless the user asked for a specific Bazel version.
+
+```bash
+bazel build //:nativelink
+bazel build //...
+bazel test //...
+bazel test //nativelink-store:memory_store_test
+bazel run --config=rustfmt @rules_rust//:rustfmt
+```
+
+For this workspace, when the user requests Bazelisk 9.0.2:
+
+```bash
+USE_BAZEL_VERSION=9.0.2 bazelisk --output_user_root=/tmp/bazelisk-9.0.2 --batch build //:nativelink
+USE_BAZEL_VERSION=9.0.2 bazelisk --output_user_root=/tmp/bazelisk-9.0.2 --batch test //...
+```
+
+## Failure Handling
+
+- If `rustfmt` fails, run:
+
+```bash
+bazel run --config=rustfmt @rules_rust//:rustfmt
+```
+
+- If `clippy` fails, fix the code. Don't silence lints unless the suppression is local, justified, and consistent with existing code.
+- If a test fails under a broad target like `//...`, rerun the specific failing target with output enabled before editing.
+- If Bazel can't find a target, inspect `BUILD.bazel` rather than guessing labels.
+
+## Verification Ladder
+
+Use the smallest rung that proves the change:
+
+1. Single package test for narrow implementation changes.
+2. `bazel build //:nativelink` for server-path changes.
+3. Package-wide tests for crate-level behavior.
+4. `bazel test //...` for shared crates, proto/config surfaces, toolchain changes, or release confidence.
+
+## Final Report
+
+Always report:
+
+- The exact commands run.
+- Whether each command passed or failed.
+- Any known unverified area.
+- Any pre-existing dirty files that weren't part of the task.

.claude/skills/nativelink-config-protocol/SKILL.md

@@ -0,0 +1,68 @@
+---
+name: nativelink-config-protocol
+description: Use when changing NativeLink JSON5 configuration, deployment examples, protobuf/service protocol surfaces, or compatibility-sensitive config behavior.
+---
+
+# NativeLink Config And Protocol Changes
+
+Use this skill when a task touches `nativelink-config`, JSON5 examples, deployment config, protobuf definitions, generated protocol code, or service APIs.
+
+## Config Changes
+
+NativeLink config changes usually require more than editing one struct.
+
+Check:
+
+- `nativelink-config/` for config structs and serialization behavior.
+- `nativelink-config/examples/` for JSON5 examples.
+- `deployment-examples/`, `deploy/`, `kubernetes/`, and `templates/` for user-facing config samples.
+- Tests that load example config files.
+
+Preserve compatibility where practical:
+
+- Add default values when old config files should continue to load.
+- Avoid renaming public fields without aliases or migration notes.
+- Keep comments in JSON5 examples accurate and actionable.
+- Treat config validation errors as user-facing diagnostics.
+
+## Protocol Changes
+
+For service or proto work, inspect both the generated API surface and the implementation layer:
+
+- `nativelink-proto/` for protobuf definitions and generated interfaces.
+- `nativelink-service/` for REAPI, bytestream, BEP, worker API, and health service behavior.
+- `nativelink-util/` for shared digest/proto conversion helpers.
+
+Don't manually edit generated code unless the repository's generation workflow requires checked-in generated changes.
+
+## Testing
+
+Start focused:
+
+```bash
+bazel test //nativelink-config/...
+bazel test //nativelink-service/...
+```
+
+Then broaden when the change crosses service boundaries:
+
+```bash
+bazel build //:nativelink
+bazel test //...
+```
+
+For docs or examples:
+
+```bash
+bazel build docs
+bazel build nativelink-config:docs
+bazel test doctests
+```
+
+## Review Checklist
+
+- Existing config files still load.
+- New fields have defaults where needed.
+- Error messages identify the invalid field or service behavior.
+- Examples and templates match the implementation.
+- Protocol changes are reflected in service tests and consumers.

.claude/skills/nativelink-dependency-update/SKILL.md

@@ -0,0 +1,69 @@
+---
+name: nativelink-dependency-update
+description: Use when updating NativeLink Cargo dependencies, Bazel module dependencies, rules_rust pins, lock files, toolchains, or generated dependency metadata.
+---
+
+# NativeLink Dependency Updates
+
+Use this skill for dependency bumps, Rust toolchain changes, Bazel module changes, lock file updates, and dependency cleanup.
+
+## Files To Inspect
+
+NativeLink keeps Cargo and Bazel dependency state connected. Check the relevant set before editing:
+
+- `Cargo.toml`
+- crate-level `Cargo.toml` files
+- `Cargo.lock`
+- `MODULE.bazel`
+- `MODULE.bazel.lock`
+- `.bazelrc`
+- `flake.nix` and `flake.lock` when tooling comes from Nix
+
+`MODULE.bazel` uses `crate.from_cargo` with workspace crate manifests. Cargo changes can affect Bazel resolution.
+
+## Update Rules
+
+- Prefer the repository's established tooling over hand-editing generated lock content.
+- Use `cargo update -p <crate>` for targeted Cargo bumps when appropriate.
+- Use Bazel module tooling for Bazel dependency lock updates when required.
+- If changing workspace lints or flags, check whether `.bazelrc` must be regenerated. The file notes that lint config is kept in sync with the top-level Cargo config through `tools/generate-bazel-rc`.
+- Don't delete or regenerate broad lock files unless the task explicitly requires it.
+- Keep feature changes narrow. Feature unification can change behavior across crates.
+
+## Validation
+
+Use a dependency-specific verification ladder:
+
+```bash
+cargo check --all
+bazel build //:nativelink
+```
+
+For Rust dependency bumps:
+
+```bash
+cargo test --all --profile=smol
+bazel test //...
+```
+
+For Bazel, rules, or toolchain changes:
+
+```bash
+bazel build //...
+bazel test //...
+```
+
+When tool versions mismatch locally, enter the Nix development shell:
+
+```bash
+nix --extra-experimental-features nix-command --extra-experimental-features flakes develop
+```
+
+## Final Report
+
+Include:
+
+- Dependency names and old/new versions.
+- Lock files or generated metadata changed.
+- Commands run.
+- Any known resolver, feature, or platform risk.

.claude/skills/nativelink-lre-debug/SKILL.md

@@ -0,0 +1,70 @@
+---
+name: nativelink-lre-debug
+description: Use when debugging NativeLink local remote execution, remote cache/executor self-tests, BEP event ingestion, worker scheduling, CAS misses, or Bazel remote execution behavior.
+---
+
+# NativeLink Local Remote Execution Debugging
+
+Use this skill when NativeLink is acting as a remote cache, remote executor, local remote execution server, or BEP target and something is missing, slow, or failing.
+
+## Identify The Mode
+
+Start by recording the exact Bazel command and which NativeLink mode is involved:
+
+- Remote cache self-test: `--config=self_test`, usually `grpc://127.0.0.1:50051`.
+- Remote executor self-test: `--config=self_execute`, usually `grpc://127.0.0.1:50052`.
+- Local remote execution overlays under `local-remote-execution/`.
+- BEP ingestion or dashboard testing, which needs Build Event Protocol upload flags.
+
+For rich action-level BEP data, include:
+
+```bash
+--build_event_publish_all_actions
+```
+
+This matters for helping users find build failures and slow actions faster.
+
+## First Checks
+
+1. Confirm the NativeLink server process is running.
+2. Confirm the expected ports aren't occupied by stale processes.
+3. Capture NativeLink logs and the exact Bazel stdout/stderr.
+4. Check whether the failure is CAS/AC storage, scheduler matching, worker execution, protocol, or client config.
+
+Don't kill user services unless the user explicitly asks. Inspect ports and processes first.
+
+## Subsystem Map
+
+- CAS/AC or cache misses: start in `nativelink-store/`.
+- Worker execution failure: start in `nativelink-worker/`.
+- Queueing, platform properties, or worker compatibility: start in `nativelink-scheduler/`.
+- REAPI, bytestream, action cache, or BEP API behavior: start in `nativelink-service/`.
+- Client templates and flags: start in `templates/` and `local-remote-execution/`.
+
+## Useful Verification Commands
+
+Inspect labels before running if unsure.
+
+```bash
+bazel build //:nativelink
+bazel test //nativelink-store/...
+bazel test //nativelink-scheduler/...
+bazel test //nativelink-worker/...
+bazel test //nativelink-service/...
+```
+
+For full confidence:
+
+```bash
+bazel test //...
+```
+
+## Debug Output To Preserve
+
+When reporting a failure, include:
+
+- NativeLink server config path and relevant ports.
+- Bazel remote cache/executor URL.
+- Whether `--build_event_publish_all_actions` was set.
+- First server-side error and first client-side error.
+- The failing target label and action mnemonic, if available.

.claude/skills/nativelink-rust-change/SKILL.md

@@ -0,0 +1,65 @@
+---
+name: nativelink-rust-change
+description: Use when implementing or reviewing Rust changes in NativeLink crates, especially storage, scheduler, worker, service, config, and shared utility behavior.
+---
+
+# NativeLink Rust Change Workflow
+
+Use this skill for Rust implementation work in `nativelink/`.
+
+## Read The Local Contract First
+
+Before editing, inspect the trait, caller, and tests around the change. NativeLink has several shared contracts where small behavior changes can affect remote execution correctness.
+
+Crate map:
+
+- `nativelink-store/`: storage backends, CAS/AC, compression, Redis/S3/GCS/filesystem/memory stores.
+- `nativelink-scheduler/`: scheduling, worker matching, action queues, awaited action state.
+- `nativelink-worker/`: worker execution, input materialization, process execution, output upload.
+- `nativelink-service/`: REAPI, bytestream, BEP, worker API, health checks.
+- `nativelink-config/`: JSON5 config structs and validation.
+- `nativelink-util/`: digest, filesystem, proto, retry, metrics, TLS, and async helpers.
+- `nativelink-proto/`: protobuf-generated API surfaces.
+
+## Implementation Rules
+
+- Prefer existing traits and helper types over new abstractions.
+- Preserve error semantics. Map user/config/input mistakes to explicit error codes and reserve internal errors for unexpected failures.
+- Avoid blocking filesystem, process, or network work inside async code unless the surrounding code already does so intentionally.
+- Keep metrics labels bounded and consistent with existing metric names.
+- For storage and scheduler changes, think through cancellation, retries, duplicate work, and missing CAS entries.
+- For worker changes, preserve stdout/stderr, exit status, timeouts, and output upload behavior.
+- For service changes, preserve protocol compatibility and avoid changing response shape without tests.
+
+## Tests
+
+Add or update the closest existing tests first. Prefer focused tests that prove the contract:
+
+```bash
+bazel test //nativelink-store/...
+bazel test //nativelink-scheduler/...
+bazel test //nativelink-worker/...
+bazel test //nativelink-service/...
+```
+
+For narrow Cargo-side checks:
+
+```bash
+cargo test --all --profile=smol
+```
+
+For server-wide confidence:
+
+```bash
+bazel build //:nativelink
+bazel test //...
+```
+
+## Final Report
+
+Summarize:
+
+- The behavior changed.
+- Files touched.
+- Focused tests run.
+- Broader tests skipped or still needed.

.github/actions/end-nix/action.yaml

@@ -0,0 +1,23 @@
+---
+# This should be a post step, but can't do it in composite actions easily. See https://github.com/actions/runner/issues/1478
+name: Teardown Nix
+description: "Common teardown for all runs using Nix."
+inputs:
+  nix_name:
+    default: ""
+  nativelink_attic_token:
+    required: true
+runs:
+  using: "composite"
+  steps:
+      - name: Check attic push
+        if: ${{ inputs.nativelink_attic_token != '' }}
+        run: |
+          cat /tmp/attic-push.log
+        shell: bash
+
+      - name: Attic push
+        if: ${{ inputs.nix_name != '' && inputs.nativelink_attic_token != '' }}
+        run: |
+          /tmp/result/bin/attic push nativelink $(nix eval --raw ${{ inputs.nix_name }})
+        shell: bash