vendor: build-unit entries in spinel-ext.json (cmake|make) — spinelgems#14

A 'build' entry declares a constrained native build (tool cmake|make, dir,
args, targets, artifacts) run inside the consumer's vendor tree, with {dir}
link-flag expansion relative to it. Heavy-native gems (toy's vendored ggml)
vendor self-contained + relocatable; the per-consumer post-vendor absolute-
path rewrite hook and its ffi_manifest cflags canary retire. Override via
SPINEL_EXT_<PLACEHOLDER> skips the build (prebuilt escape hatch). Smoke-
tested: make + cmake paths, override path.

Also: reprobe-corpus.sh now honors SPINEL_NO_FREEZE=1 (skip self-freeze when
SPINEL_DIR is already a frozen copy) — the in-repo freeze it otherwise
creates is the stray-nested-dir class that corrupted the 9c0a5f0 freeze.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: OriPekelman

bin/reprobe-corpus.sh

@@ -29,14 +29,20 @@ elif command -v nproc >/dev/null 2>&1; then SHARDS="$(nproc)"
 else SHARDS=4; fi
 
 # Freeze the Spinel checkout (rev-stable for the whole run), like survey-run.sh.
+# SPINEL_NO_FREEZE=1 skips the self-freeze — set it when SPINEL_DIR already IS
+# a frozen copy (e.g. /srv/data/scratch/spinelgems-rp/spinel-frozen-<rev>).
+# The in-repo freeze this would otherwise create is exactly the stray nested
+# dir that corrupted the 9c0a5f0 freeze; harness-run.sh honors the same flag.
 SP_SRC="${SPINEL_DIR:-$HOME/spinel}"
 SP_REV="$(git -C "$SP_SRC" rev-parse --short HEAD 2>/dev/null || echo unknown)"
-SP_FROZEN="$HERE/spinel-frozen-$SP_REV"
-if [ ! -d "$SP_FROZEN" ]; then
-  echo "[reprobe] freezing $SP_SRC -> $SP_FROZEN"
-  cp -al "$SP_SRC" "$SP_FROZEN" 2>/dev/null || cp -r "$SP_SRC" "$SP_FROZEN"
+if [ -z "${SPINEL_NO_FREEZE:-}" ]; then
+  SP_FROZEN="$HERE/spinel-frozen-$SP_REV"
+  if [ ! -d "$SP_FROZEN" ]; then
+    echo "[reprobe] freezing $SP_SRC -> $SP_FROZEN"
+    cp -al "$SP_SRC" "$SP_FROZEN" 2>/dev/null || cp -r "$SP_SRC" "$SP_FROZEN"
+  fi
+  export SPINEL_DIR="$SP_FROZEN"
 fi
-export SPINEL_DIR="$SP_FROZEN"
 
 OUT="${2:-$HERE/survey-$SP_REV}"
 mkdir -p "$OUT"

docs/c-ext.md

@@ -218,3 +218,41 @@ works fine without; the hook is small when we want it.)
 - **Trust is unchanged.** This is placement/build wiring, not gating — a
   C-ext gem still probes `risky` and earns trust only through the `verified`
   rung (a behaviour smoke through the harness).
+
+## Build-units: heavy native deps (spinelgems#14)
+
+Per-`.c` `source` entries fit small shims (tep's `sphttp.c`). They can't
+express toy's case: `tinynn/*.c` shims *plus a vendored ggml CMake build*
+producing three static archives. That gap forced toy's consumers into a
+per-consumer post-vendor hook that rewrote vendored `-L` flags into
+**absolute paths into toy's checkout** — non-relocatable, unpublishable.
+
+A `build` entry declares the native build instead:
+
+```json
+{ "name": "ggml",
+  "build": {
+    "tool": "cmake",                       // or "make" — only these two
+    "dir": "vendor/ggml",                  // relative to gem root
+    "args": ["-DBUILD_SHARED_LIBS=OFF"],   // configure args (cmake) / make args
+    "targets": ["ggml", "ggml-cpu"],       // build targets (optional)
+    "artifacts": ["build/src/libggml.a"]   // verified to exist post-build
+  },
+  "placeholder": "@GGML_LINK@",
+  "link": ["-L{dir}/build/src", "-L{dir}/build/src/ggml-cpu"] }
+```
+
+At vendor time: the `dir` is **copied into the consumer's vendor tree**, the
+tool runs there, artifacts are verified, and `{dir}` in `link` expands to the
+*vendored* path (project-relative when `--into` is relative, the default) —
+self-contained and relocatable, the same end-state tep's small shims already
+had. A consumer override (`SPINEL_EXT_<PLACEHOLDER>` / `--ext`) supplies
+replacement flags and skips the build entirely (the prebuilt escape hatch).
+
+Why this doesn't break "Gemfiles are Gemfiles": CRuby gems already run
+arbitrary native builds at install time — `extconf.rb`, rake-compiler, and
+most on-point **nokogiri building its vendored libxml2 via mini_portile2**,
+plus rubygems' native Cargo builder. A *declared* cmake/make unit is the
+Spinel analogue of `extensions:` in a gemspec — strictly narrower than the
+extconf precedent (no free-form shell, declared artifacts, auditable), and
+it's what lets a heavy-native gem publish to RubyGems as a normal gem.

lib/bundler/spinel/vendorer.rb

@@ -128,6 +128,31 @@ def wire_extensions(src, dest, overrides, disable)
             next
           end
 
+          # Build-unit entry (spinelgems#14): a declared native build (cmake|make)
+          # producing archives *inside the consumer's vendor tree*, with `link`
+          # flags expanded relative to it ({dir} -> the vendored build dir). This
+          # is the heavy-native analogue of `source` per-.c entries — nokogiri's
+          # mini_portile2 precedent, Spinel-shaped. It replaces the per-consumer
+          # post-vendor absolute-path rewrite hooks (toy's prep/post_vendor_toy.rb)
+          # that made vendored trees non-relocatable and toy unpublishable.
+          # A consumer override (SPINEL_EXT_<PLACEHOLDER> / --ext) supplies the
+          # full replacement flags and skips the build (prebuilt escape hatch).
+          if e["build"]
+            if placeholder && (ov = overrides[placeholder] || ENV[ext_env_key(placeholder)])
+              substitute_placeholder(dest, placeholder, ov.to_s)
+              wired += 1
+              next
+            end
+            ven_dir = build_unit(src, dest, e) or next # build failed (warned)
+            if placeholder
+              parts = Array(e["link"]).map { |t| t.gsub("{dir}", ven_dir) }
+              parts.concat(Array(e["libs"]))
+              substitute_placeholder(dest, placeholder, parts.join(" ").strip)
+            end
+            wired += 1
+            next
+          end
+
           # Compile / place the .o (or take a prebuilt override path). Both forms
           # need this; post-#1011 const-fold form skips the substitution below.
           obj = nil
@@ -214,6 +239,72 @@ def compile_ext(src, dest, entry, extra_cflags = [])
         nil
       end
 
+      # Build-unit (spinelgems#14): copy the gem's declared build dir into the
+      # vendor tree, run the declared tool there, verify the declared artifacts.
+      # Returns the vendored dir path (project-relative when `into` was given
+      # relative — the usual case — so substituted -L flags stay relocatable
+      # with the consumer project) or nil on failure (warned, entry skipped).
+      #
+      # The tool surface is deliberately constrained to cmake|make with declared
+      # args/targets/artifacts — no free-form shell. extconf.rb is precedent for
+      # arbitrary install-time code in gems, but there's no need to copy that
+      # mistake into spinel-ext.json: a declarative unit stays auditable and the
+      # detector-inferable, consumer-side philosophy survives.
+      def build_unit(src, dest, entry)
+        b = entry["build"]
+        dir_rel = b["dir"].to_s
+        src_dir = File.join(src, dir_rel)
+        unless File.directory?(src_dir)
+          warn "[vendor] build dir not found for #{entry['name'] || entry['placeholder']}: #{dir_rel}"
+          return nil
+        end
+
+        ven_dir = File.join(dest, dir_rel)
+        FileUtils.mkdir_p(File.dirname(ven_dir))
+        FileUtils.rm_rf(ven_dir)
+        FileUtils.cp_r(src_dir, ven_dir)
+
+        jobs = begin
+          require "etc"
+          Etc.nprocessors.to_s
+        rescue StandardError
+          "4"
+        end
+        cmds =
+          case b["tool"].to_s
+          when "cmake"
+            build_dir = File.join(ven_dir, "build")
+            cfg = ["cmake", "-S", ven_dir, "-B", build_dir, *Array(b["args"]).map(&:to_s)]
+            bld = ["cmake", "--build", build_dir, "-j", jobs]
+            targets = Array(b["targets"]).map(&:to_s)
+            bld.push("--target", *targets) unless targets.empty?
+            [cfg, bld]
+          when "make"
+            [["make", "-C", ven_dir, "-j", jobs,
+              *Array(b["args"]).map(&:to_s), *Array(b["targets"]).map(&:to_s)]]
+          else
+            warn "[vendor] unknown build tool #{b['tool'].inspect} for #{entry['name']} (cmake|make)"
+            return nil
+          end
+
+        cmds.each do |cmd|
+          out, st = Open3.capture2e(*cmd)
+          unless st.success?
+            warn "[vendor] build failed (#{entry['name']}): #{cmd.take(2).join(' ')} ... : " \
+                 "#{out.lines.last(3).join.strip}"
+            return nil
+          end
+        end
+
+        missing = Array(b["artifacts"]).reject { |a| File.exist?(File.join(ven_dir, a.to_s)) }
+        unless missing.empty?
+          warn "[vendor] build for #{entry['name']} succeeded but artifacts missing: #{missing.join(', ')}"
+          return nil
+        end
+
+        ven_dir
+      end
+
       def substitute_placeholder(dest, placeholder, repl)
         Dir[File.join(dest, "**", "*.rb")].each do |f|
           body = File.read(f)