feat(rp-20): extend probe_gpu_quality with full GPU runtime DLL surface

zhmiao · Copilot · zhmiao · commit 07f3b3a77052 · 2026-05-27T17:43:01.000-07:00
Before this change, probe_gpu_quality.{sh,ps1} only probed for
libcudnn.so.9 / cudnn64_9.dll. A user with a partial CUDA install
(e.g. cuDNN present but cuFFT missing — common after 'pip install
sparrow-engine-gpu' because the wheel deliberately doesn't bundle
the NVIDIA runtime sidecars) passed the existing probe but failed
at first inference with a libdl / LoadLibraryW error pointing at
whichever sidecar was the actual culprit.

Add probes for the full DT_NEEDED set of ORT 1.25.1's CUDA provider,
plus engine-side nvJPEG.

Ground truth (verified 2026-05-27 via readelf -d on the published
onnxruntime-gpu 1.25.1 wheel's libonnxruntime_providers_cuda.so):

  Linux:                       Windows:
    libcudart.so.12              cudart64_12.dll       (CUDA runtime base)
    libcublas.so.12              cublas64_12.dll       (matmul)
    libcublasLt.so.12            cublasLt64_12.dll     (cuBLAS Lt)
    libcurand.so.10              curand64_10.dll       (RNG)
    libcufft.so.11               cufft64_11.dll        (FFT — audio)
    libcudnn.so.9                cudnn64_9.dll         (existing version-checked)

Plus engine-side (sparrow-engine-gpu/src/decode.rs dlopen'd at first
JpegDecoder::new — yolo.rs / tiled.rs / classifier.rs all raise
SparrowEngineError::NvjpegUnavailable on construction failure):

    libnvjpeg.so.12              nvjpeg64_12.dll       (image decode)

UX: 'collect ALL missing libs' rather than 'first-failing-wins' so the
user fixes them in one cycle rather than iterating one-DLL-per-rerun.

Verdict semantics: the existing 'cudnn_err' verdict name is preserved
(any new name would force a parallel change in sparrow-engine-install.{sh,ps1}'s
exit-code translation), but the docstring + reason string make clear it
is now the catch-all 'GPU runtime prereq missing' hard-fail bucket.
The reason string carries which specific library is the actual culprit
and the matching pip-sidecar / apt install hint pointing at
docs/user-manual.md §2.5 Option A / B.

Search dirs (Linux):
  1. $LD_LIBRARY_PATH (colon-split)
  2. ~/.sparrow-engine/cuda-sidecars/lib/python*/site-packages/nvidia/*/lib
     (matches the Python sidecar venv pattern from user-manual.md §2.5 Option B
     and the brew GPU wrapper's auto-discovery path).
  3. /usr/lib/python3/dist-packages/torch/lib   (Lambda Stack / PyTorch-bundled)
  4. /usr/lib/python3/dist-packages/tensorflow  (Lambda Stack / TF-bundled)
  5. /usr/local/cuda/lib64                       (NVIDIA CUDA toolkit)
  6. /usr/lib/x86_64-linux-gnu                   (Ubuntu apt nvidia-cudnn)
  7. /usr/lib64 + /usr/lib                       (RHEL / fallback)

Search dirs (Windows):
  1. %SystemRoot%\System32
  2. %CUDA_PATH%\bin
  3. %CUDA_PATH_V*_*%\bin    (side-by-side CUDA Toolkit installs)

Defensive: PS Search-RequiredDll guards $env:SystemRoot null (Linux pwsh
test surface) to avoid noisy Join-Path errors. Windows always has it set.

The existing cuDNN version-floor check + compute-cap (sm_80) check are
preserved unchanged behind the new gate (if all required libs are
present, the existing checks run as before).

Local validation: GREEN end-to-end on the dev-box (cuDNN 9.19, all 6
sidecars present); PowerShell parser GREEN (dotnet pwsh 7.6 ParseFile).

Refs: RP-20 (docs/ideas.md:249), OQ-2026-05-27-2 (related — CLI tarball lane).

Co-authored-by: Copilot &lt;223556219+Copilot@users.noreply.github.com&gt;
diff --git a/installer/probe_gpu_quality.ps1 b/installer/probe_gpu_quality.ps1
@@ -1,13 +1,22 @@
 ﻿# installer/probe_gpu_quality.ps1 — dot-sourceable layer-2 quality probe (Windows).
+#                                    GPU runtime sidecar DLL surface +
 #                                    cuDNN ≥9.10 floor + driver-version sanity.
 #
 # Purpose
 #   Layer-2 of the Sparrow Engine install-time selector on Windows. Runs ONLY after
-#   layer-1 (`probe.ps1`) has returned `gpu`. Reports two follow-up signals:
-#     1. Is cuDNN ≥9.10 reachable? Windows ships cuDNN as `cudnn64_9.dll`
+#   layer-1 (`probe.ps1`) has returned `gpu`. Reports three follow-up signals:
+#     1. Are all hard-required GPU runtime sidecar DLLs reachable? ORT 1.25.1's
+#        ORT CUDA provider links against six CUDA libs (verified via readelf
+#        against the published Linux wheel, 2026-05-27); Windows equivalents
+#        are `cudart64_12.dll`, `cublas64_12.dll`, `cublasLt64_12.dll`,
+#        `curand64_10.dll`, `cufft64_11.dll`, `cudnn64_9.dll`. The
+#        sparrow-engine GPU image-decode path additionally loads
+#        `nvjpeg64_12.dll` via dlopen — missing nvJPEG fails at first
+#        inference with `SparrowEngineError::NvjpegUnavailable`.
+#     2. Is cuDNN ≥9.10 reachable? Windows ships cuDNN as `cudnn64_9.dll`
 #        plus version-stamped sidecar files (e.g. `cudnn_9.10.2.21.dll`); the
 #        FileVersion property of `cudnn64_9.dll` is the most reliable signal.
-#     2. Is the GPU compute-capability ≥sm_80? FP16 production cells need
+#     3. Is the GPU compute-capability ≥sm_80? FP16 production cells need
 #        Ampere Tensor Cores; pre-Ampere falls back to FP32 at 2-3× latency.
 #
 # Usage
@@ -21,6 +30,12 @@
 #           'cudnn_err'  { exit 11 }
 #       }
 #
+#   The `cudnn_err` verdict is the hard-fail bucket for ANY missing GPU
+#   runtime prerequisite (cuDNN, cuBLAS, cuFFT, nvJPEG, etc.); the verdict
+#   name is preserved for back-compat with `sparrow-engine-install.ps1`'s
+#   switch-handling. The verdict REASON string carries which specific DLL is
+#   the actual culprit.
+#
 #   Direct invocation:
 #       powershell -NoProfile -ExecutionPolicy Bypass -File .\installer\probe_gpu_quality.ps1
 #
@@ -40,44 +55,104 @@
 #   docs/design/phase4.1-install-selector/final_design.md § 2.4
 #   docs/design/phase4.1-install-selector/round_02/scripts-architect_proposal.md § 1.2.1 + § 3.2
 #
+# DLL surface ground truth (2026-05-27, RP-20):
+#   ORT 1.25.1 CUDA provider DT_NEEDED CUDA libs:
+#     cublasLt64_12.dll, cublas64_12.dll, curand64_10.dll, cufft64_11.dll,
+#     cudart64_12.dll, cudnn64_9.dll
+#   Engine-side nvJPEG (dlopen at first JpegDecoder::new):
+#     nvjpeg64_12.dll
+#
 # cuDNN ≥9.10 floor citation (verified 2026-05-08):
 #   - sparrow-engine/scripts/ort-env.sh:167-168 — "cuDNN: we require 9.10+ for SpeciesNet
 #     on sm_89 (cuDNN 9.8 has a Conv engine bug with asymmetric padding —
 #     'No valid engine configs for ConvFwd_'). PyTorch/TF bundle 9.8."
 #   - docs/lessons.md:29
 #   - docs/tech_report/06_gotchas_and_constraints.md:17-25
 
+# ---------------------------------------------------------------------------
+# Helper: search for a CUDA runtime sidecar DLL by exact basename.
+#
+# Looks in the canonical Windows locations a GPU install drops DLLs:
+#   1. %SystemRoot%\System32 (cuDNN MSI installer drops here; CUDA Runtime
+#      apt-style installs land here on some setups).
+#   2. %CUDA_PATH%\bin (standalone CUDA Toolkit installer).
+#   3. %CUDA_PATH_V*_*%\bin (side-by-side Toolkit installs — NVIDIA writes
+#      version-suffixed env vars for each major.minor).
+#
+# Returns the first matching absolute path, or $null if not found.
+# ---------------------------------------------------------------------------
+function Search-RequiredDll {
+    [CmdletBinding()]
+    param([Parameter(Mandatory)][string]$DllName)
+    $candidates = @()
+    if ($env:SystemRoot) {
+        $candidates += (Join-Path $env:SystemRoot "System32\$DllName")
+    }
+    if ($env:CUDA_PATH) {
+        $candidates += (Join-Path $env:CUDA_PATH "bin\$DllName")
+    }
+    Get-ChildItem env: -ErrorAction SilentlyContinue |
+        Where-Object { $_.Name -match '^CUDA_PATH_V\d+_\d+$' } |
+        ForEach-Object { $candidates += (Join-Path $_.Value "bin\$DllName") }
+    foreach ($p in $candidates) {
+        if ($p -and (Test-Path -LiteralPath $p -PathType Leaf)) {
+            return $p
+        }
+    }
+    return $null
+}
+
 function probe_gpu_quality {
     [CmdletBinding()]
     param()
 
     $env:SPARROW_ENGINE_GPU_QUALITY = ''
     $env:SPARROW_ENGINE_GPU_QUALITY_REASON = ''
 
-    # 1. cuDNN check — locate cudnn64_9.dll in the canonical Windows locations
-    #    (System32 from the cuDNN MSI installer, %CUDA_PATH%\bin from the
-    #    standalone CUDA toolkit installer). Read FileVersion to compare
-    #    against the 9.10.0 floor.
-    $cudnnDll = $null
-    $candidates = @(
-        (Join-Path $env:SystemRoot 'System32\cudnn64_9.dll')
+    # 0. Hard-required GPU runtime sidecar DLLs (RP-20). Collect ALL missing
+    #    DLLs so the user fixes them in one cycle.
+    $requiredDlls = @(
+        @{ Name = 'cudart64_12.dll';   PipPkg = 'nvidia-cuda-runtime-cu12' },
+        @{ Name = 'cublas64_12.dll';   PipPkg = 'nvidia-cublas-cu12' },
+        @{ Name = 'cublasLt64_12.dll'; PipPkg = 'nvidia-cublas-cu12' },
+        @{ Name = 'curand64_10.dll';   PipPkg = 'nvidia-curand-cu12' },
+        @{ Name = 'cufft64_11.dll';    PipPkg = 'nvidia-cufft-cu12' },
+        @{ Name = 'nvjpeg64_12.dll';   PipPkg = 'nvidia-nvjpeg-cu12' }
     )
-    if ($env:CUDA_PATH) {
-        $candidates += (Join-Path $env:CUDA_PATH 'bin\cudnn64_9.dll')
+    $missing = @()
+    foreach ($req in $requiredDlls) {
+        $found = Search-RequiredDll -DllName $req.Name
+        if (-not $found) {
+            $missing += $req
+        }
     }
-    # Also search any CUDA_PATH_V*_* env vars (NVIDIA installs version-suffixed
-    # vars for side-by-side toolkit installs).
-    Get-ChildItem env: -ErrorAction SilentlyContinue |
-        Where-Object { $_.Name -match '^CUDA_PATH_V\d+_\d+$' } |
-        ForEach-Object { $candidates += (Join-Path $_.Value 'bin\cudnn64_9.dll') }
 
-    foreach ($p in $candidates) {
-        if ($p -and (Test-Path -LiteralPath $p -PathType Leaf)) {
-            $cudnnDll = $p
-            break
+    if ($missing.Count -gt 0) {
+        $lines = $missing | ForEach-Object {
+            "    - $($_.Name)  (pip pkg: $($_.PipPkg))"
         }
+        $hint = ($lines -join "`n")
+        $env:SPARROW_ENGINE_GPU_QUALITY = 'cudnn_err'
+        $env:SPARROW_ENGINE_GPU_QUALITY_REASON = @"
+GPU runtime sidecar DLL(s) missing — spe-gpu will fail at first inference with a LoadLibraryW error.
+$hint
+
+Install via ONE of (see docs/user-manual.md §2.5):
+  Option A — CUDA Toolkit installer: download CUDA 12.6+ from https://developer.nvidia.com/cuda-downloads and cuDNN 9.10+ from https://developer.nvidia.com/cudnn
+  Option B — Python sidecar wheels (no admin): uv pip install --target $env:USERPROFILE\.sparrow-engine\cuda-sidecars nvidia-cudnn-cu12 nvidia-cublas-cu12 nvidia-curand-cu12 nvidia-cufft-cu12 nvidia-nvjpeg-cu12 nvidia-cuda-runtime-cu12
+"@
+        $script:SparrowEngineGpuQuality = $env:SPARROW_ENGINE_GPU_QUALITY
+        $script:SparrowEngineGpuQualityReason = $env:SPARROW_ENGINE_GPU_QUALITY_REASON
+        Write-Output 'fail'
+        return
     }
 
+    # 1. cuDNN check — locate cudnn64_9.dll in the canonical Windows locations
+    #    (System32 from the cuDNN MSI installer, %CUDA_PATH%\bin from the
+    #    standalone CUDA toolkit installer). Read FileVersion to compare
+    #    against the 9.10.0 floor.
+    $cudnnDll = Search-RequiredDll -DllName 'cudnn64_9.dll'
+
     if (-not $cudnnDll) {
         $env:SPARROW_ENGINE_GPU_QUALITY = 'cudnn_err'
         $env:SPARROW_ENGINE_GPU_QUALITY_REASON = "cuDNN 9.x DLL (cudnn64_9.dll) not found in System32 or %CUDA_PATH%\bin; the GPU flavor will fail at first inference. Install cuDNN 9.10+ from https://developer.nvidia.com/cudnn"
diff --git a/installer/probe_gpu_quality.sh b/installer/probe_gpu_quality.sh
@@ -1,15 +1,25 @@
 #!/usr/bin/env bash
 # installer/probe_gpu_quality.sh — sourceable layer-2 quality probe
-#                                  (cuDNN ≥9.10 floor + driver-version sanity).
+#                                  (GPU runtime sidecar DLL surface +
+#                                  cuDNN ≥9.10 floor + driver-version sanity).
 #
 # Purpose
-#   Layer-2 of the Sparrow Engine install-time selector. Runs ONLY after layer-1
-#   (`probe.sh`) has returned `gpu`. The basic CUDA probe answers "is CUDA
-#   reachable?" — this layer answers two follow-up questions:
-#     1. Is cuDNN ≥9.10 reachable? (canonical project floor — cuDNN 9.8 has
+#   Layer-2 of the Sparrow Engine install-time selector. Runs ONLY after
+#   layer-1 (`probe.sh`) has returned `gpu`. The basic CUDA probe answers
+#   "is CUDA reachable?" — this layer answers three follow-up questions:
+#     1. Are all hard-required GPU runtime sidecar libraries reachable?
+#        ORT 1.25.1's `libonnxruntime_providers_cuda.so` DT_NEEDED list
+#        (verified via `readelf -d` on the published wheel, 2026-05-27)
+#        names six CUDA libs: `libcudart.so.12`, `libcublas.so.12`,
+#        `libcublasLt.so.12`, `libcurand.so.10`, `libcufft.so.11`,
+#        `libcudnn.so.9`. The sparrow-engine GPU image-decode path additionally
+#        loads `libnvjpeg.so.12` via dlopen — missing nvJPEG fails at first
+#        inference with `SparrowEngineError::NvjpegUnavailable`
+#        (sparrow-engine-gpu/src/models/{yolo,tiled,classifier}.rs).
+#     2. Is cuDNN ≥9.10 reachable? (canonical project floor — cuDNN 9.8 has
 #        the asymmetric-padding ConvFwd engine bug that breaks SpeciesNet on
 #        sm_89; sources cited in `probe_cudnn_check` below).
-#     2. Is the GPU compute-capability ≥sm_80? (FP16 production cells need
+#     3. Is the GPU compute-capability ≥sm_80? (FP16 production cells need
 #        Ampere Tensor Cores; T4 and earlier silently fall back to FP32 at
 #        2-3× the latency of advertised perf).
 #
@@ -21,9 +31,15 @@
 #           ok)          : ;;                                # silent install
 #           sm_warn)     warn "FP16 perf will be degraded" ;;
 #           cudnn_warn)  warn "SpeciesNet will fail until cuDNN ≥9.10" ;;
-#           cudnn_err)   die 11 "cuDNN <9.10 — block install" ;;
+#           cudnn_err)   die 11 "GPU runtime sidecar missing — block install" ;;
 #       esac
 #
+#   The `cudnn_err` verdict is the hard-fail bucket for ANY missing GPU
+#   runtime prerequisite (cuDNN, cuBLAS, cuFFT, nvJPEG, etc.); the verdict
+#   name is preserved for back-compat with `sparrow-engine-install.sh`'s
+#   case-handling. The verdict REASON string carries which specific library
+#   is the actual culprit.
+#
 #   Direct invocation:
 #       bash installer/probe_gpu_quality.sh    # stdout = quality verdict
 #
@@ -40,6 +56,14 @@
 #   docs/design/phase4.1-install-selector/round_02/scripts-architect_proposal.md § 1.2.1
 #   docs/design/phase4.1-install-selector/round_01/scripts-architect_proposal.md § 1.2.1 (canonical pseudocode)
 #
+# DLL surface ground truth (2026-05-27, RP-20):
+#   ORT 1.25.1 CUDA provider DT_NEEDED CUDA libs (readelf -d
+#   libonnxruntime_providers_cuda.so):
+#     libcublasLt.so.12, libcublas.so.12, libcurand.so.10, libcufft.so.11,
+#     libcudart.so.12, libcudnn.so.9
+#   Engine-side nvJPEG (dlopen at first JpegDecoder::new):
+#     libnvjpeg.so.12
+#
 # cuDNN ≥9.10 floor citation (verified 2026-05-08):
 #   - sparrow-engine/scripts/ort-env.sh:167-168 — "cuDNN: we require 9.10+ for SpeciesNet
 #     on sm_89 (cuDNN 9.8 has a Conv engine bug with asymmetric padding —
@@ -51,10 +75,120 @@
 # This script must NEVER `exit` from a sourced context — caller may have
 # sourced it. Use `return` from inside the function.
 
+# ---------------------------------------------------------------------------
+# Helper: search for a CUDA runtime sidecar library by exact basename.
+#
+# Looks in the canonical Linux locations sparrow-engine-gpu's wrapper
+# script (installer/homebrew/sparrow-engine-gpu.rb caveats) advertises,
+# plus the user-set $LD_LIBRARY_PATH. Returns the first matching absolute
+# path on stdout, or empty string if not found.
+#
+# Search order (mirrors the brew GPU wrapper's auto-discovery so the
+# probe's "yes / no" verdict agrees with what `spe-gpu` will actually
+# resolve at runtime):
+#   1. Each entry of $LD_LIBRARY_PATH (colon-separated).
+#   2. ~/.sparrow-engine/cuda-sidecars/lib/python*/site-packages/nvidia/<pkg>/lib
+#      (Python sidecar venv pattern from user-manual.md §2.5 Option B).
+#   3. /usr/lib/python3/dist-packages/torch/lib (Lambda Stack / PyTorch-bundled).
+#   4. /usr/lib/python3/dist-packages/tensorflow (Lambda Stack / TF-bundled).
+#   5. /usr/local/cuda/lib64 (NVIDIA CUDA toolkit / system install).
+#   6. /usr/lib/x86_64-linux-gnu (Ubuntu apt nvidia-cudnn / system).
+#   7. /usr/lib64 + /usr/lib (RHEL / fallback).
+# ---------------------------------------------------------------------------
+_search_required_lib() {
+    _basename="$1"
+    # 1. LD_LIBRARY_PATH (colon-split). Quoting `$LD_LIBRARY_PATH:` and using
+    #    parameter expansion avoids `set -u` blowups when the var is unset.
+    _OLDIFS="$IFS"
+    IFS=':'
+    for _d in ${LD_LIBRARY_PATH:-}; do
+        IFS="$_OLDIFS"
+        if [ -n "$_d" ] && [ -e "$_d/$_basename" ]; then
+            printf '%s\n' "$_d/$_basename"
+            return 0
+        fi
+        IFS=':'
+    done
+    IFS="$_OLDIFS"
+    # 2-7. Deterministic dir list. Glob expansion handles the Python sidecar
+    #      venv variants. `find -maxdepth 4` bounds the cost.
+    for _glob in \
+        "$HOME/.sparrow-engine/cuda-sidecars"/lib/python*/site-packages/nvidia/*/lib \
+        /usr/lib/python3/dist-packages/torch/lib \
+        /usr/lib/python3/dist-packages/tensorflow \
+        /usr/local/cuda/lib64 \
+        /usr/lib/x86_64-linux-gnu \
+        /usr/lib64 \
+        /usr/lib; do
+        # Word-splitting on the glob is intentional — wildcard expansion
+        # produces multiple dirs.
+        for _d in $_glob; do
+            [ -d "$_d" ] || continue
+            if [ -e "$_d/$_basename" ]; then
+                printf '%s\n' "$_d/$_basename"
+                return 0
+            fi
+        done
+    done
+    return 1
+}
+
 probe_gpu_quality() {
     SPARROW_ENGINE_GPU_QUALITY=""
     SPARROW_ENGINE_GPU_QUALITY_REASON=""
 
+    # 0. Hard-required GPU runtime sidecar libraries (RP-20). Each missing
+    #    library prevents `spe-gpu` from running. Collect ALL missing names
+    #    into a list so the user fixes them in one cycle rather than
+    #    one-per-rerun.
+    _required_libs="
+        libcudart.so.12
+        libcublas.so.12
+        libcublasLt.so.12
+        libcurand.so.10
+        libcufft.so.11
+        libnvjpeg.so.12
+    "
+    _missing_libs=""
+    for _lib in $_required_libs; do
+        if ! _search_required_lib "$_lib" >/dev/null; then
+            if [ -z "$_missing_libs" ]; then
+                _missing_libs="$_lib"
+            else
+                _missing_libs="$_missing_libs $_lib"
+            fi
+        fi
+    done
+
+    if [ -n "$_missing_libs" ]; then
+        # Build install-hint with the matching pip-sidecar package names.
+        # Mapping (basename → wheel) is determined from the canonical
+        # NVIDIA cu12 wheel naming convention.
+        _hint=""
+        for _miss in $_missing_libs; do
+            case "$_miss" in
+                libcudart.so.12)   _pkg="nvidia-cuda-runtime-cu12" ;;
+                libcublas.so.12)   _pkg="nvidia-cublas-cu12" ;;
+                libcublasLt.so.12) _pkg="nvidia-cublas-cu12" ;;
+                libcurand.so.10)   _pkg="nvidia-curand-cu12" ;;
+                libcufft.so.11)    _pkg="nvidia-cufft-cu12" ;;
+                libnvjpeg.so.12)   _pkg="nvidia-nvjpeg-cu12" ;;
+                *)                 _pkg="(unknown)" ;;
+            esac
+            _hint="${_hint}    - ${_miss}  (pip pkg: ${_pkg})
+"
+        done
+        SPARROW_ENGINE_GPU_QUALITY="cudnn_err"
+        SPARROW_ENGINE_GPU_QUALITY_REASON="GPU runtime sidecar(s) missing — \`spe-gpu\` will fail at first inference with a libdl 'cannot open shared object file' error.
+$_hint
+Install via ONE of (see docs/user-manual.md §2.5):
+  Option A — system CUDA: sudo apt install nvidia-cuda-toolkit nvidia-cudnn libnvjpeg
+  Option B — Python sidecar wheels (no root): uv pip install --target ~/.sparrow-engine/cuda-sidecars nvidia-cudnn-cu12 nvidia-cublas-cu12 nvidia-curand-cu12 nvidia-cufft-cu12 nvidia-nvjpeg-cu12 nvidia-cuda-runtime-cu12"
+        export SPARROW_ENGINE_GPU_QUALITY SPARROW_ENGINE_GPU_QUALITY_REASON
+        printf '%s\n' "fail"
+        return 0
+    fi
+
     # 1. cuDNN check — search the engine-canonical paths in priority order.
     #    Mirrors `sparrow-engine/scripts/ort-env.sh::pick_newest_cudnn_dir` (lines
     #    179-198). Two filename patterns can carry the version:
