Publish 300W bottom 30pct fan qualification

Author: Lightheartdevs

hardware-tests/tower2-dual-qwen27-no-gap-2026-07-29/README.md

@@ -73,6 +73,7 @@ software-power-cap active fraction.
 
 | Run | Bottom mean / max / fan | Top mean / max / fan | Throttling |
 |---|---|---|---|
+| [`ng-faniso-own-loadbottom-f30-p300-v3host-bump-r1`](ng-faniso-own-loadbottom-f30-p300-v3host-bump-r1/) | 53.58 C / 57 C / 30.0%, 1,201 RPM | 40.34 C / 42 C / 50.0%, 1,679 RPM | 300 W bottom-loaded 30% fan qualification pass; safety-only/non-inferential; all thermal/brake events and counters zero |
 | [`ng-fan-b40-t60-sym300-v3host-15m-r5`](ng-fan-b40-t60-sym300-v3host-15m-r5/) | 53.82 C / 57 C / 40.0%, 1,439 RPM | 67.16 C / 70 C / 60.0%, 1,917 RPM | V3HOST 40/60 internally validated at n=3; matched block 3; counters zero |
 | [`ng-fan-b60-t40-sym300-v3host-15m-r5`](ng-fan-b60-t40-sym300-v3host-15m-r5/) | 53.81 C / 57 C / 60.0%, 1,917 RPM | 67.38 C / 71 C / 40.0%, 1,439 RPM | V3HOST 60/40 internally validated at n=3; sub-second GPU0 boundary transient retained; counters zero |
 | [`ng-fan-b60-t40-sym300-v3host-15m-r4`](ng-fan-b60-t40-sym300-v3host-15m-r4/) | 53.71 C / 57 C / 60.0%, 1,917 RPM | 67.28 C / 71 C / 40.0%, 1,439 RPM | V3HOST 60/40 population n=2/3; matched block 2 shows clock redistribution with unchanged total; counters zero |

hardware-tests/tower2-dual-qwen27-no-gap-2026-07-29/VALIDATION_REGISTRY.csv

@@ -50,3 +50,4 @@ run_id,cell_id,replicate,configuration,bottom_power_w,top_power_w,load_pattern,d
 2026-07-30T19-19-56Z,NG-FAN-B60-T40-SYM300-V3HOST-15M,2,no-gap-fixed-fan-v3host,300,300,loaded-loaded,900,pass,true,false,no calibrated ambient or local-inlet probes; raw campaign replicate label R4; tightly matched reversed-order crossover block 2,,ng-fan-b60-t40-sym300-v3host-15m-r4
 2026-07-30T19-55-42Z,NG-FAN-B60-T40-SYM300-V3HOST-15M,3,no-gap-fixed-fan-v3host,300,300,loaded-loaded,900,pass,true,false,no calibrated ambient or local-inlet probes; raw campaign replicate label R5; sub-second GPU0 request-boundary transient passed frozen loaded-power gate,,ng-fan-b60-t40-sym300-v3host-15m-r5
 2026-07-30T20-31-39Z,NG-FAN-B40-T60-SYM300-V3HOST-15M,3,no-gap-fixed-fan-v3host,300,300,loaded-loaded,900,pass,true,false,no calibrated ambient or local-inlet probes; raw campaign replicate label R5; tightly matched crossover block 3; both V3HOST policies now n=3,,ng-fan-b40-t60-sym300-v3host-15m-r5
+2026-07-30T21-22-21Z,NG-FANISO-OWN-LOADBOTTOM-F30-P300-V3HOST-BUMP,1,no-gap-fan-mechanism-qualification,300,0,loaded-idle,120,qualification-pass,false,false,two-minute safety qualification is not an inferential replicate; initial process exit was a corrected schema-key mismatch; preserved telemetry passes commit 5a10c2f gate,,ng-faniso-own-loadbottom-f30-p300-v3host-bump-r1

hardware-tests/tower2-dual-qwen27-no-gap-2026-07-29/dual-vllm-qwen27-30m.sh

@@ -655,6 +655,7 @@ ABORT_FILE="$OUT/.abort"
 STOPPED_FILE="$OUT/stopped-containers.txt"
 STOPPED_USER_SERVICES_FILE="$OUT/stopped-user-services.txt"
 SUMMARY_JSON="$OUT/summary.json"
+QUALIFICATION_RESULT_JSON="$OUT/qualification-result.json"
 CHECKSUMS="$OUT/SHA256SUMS"
 GPU0_PAYLOAD="$OUT/gpu0-payload.json"
 GPU1_PAYLOAD="$OUT/gpu1-payload.json"
@@ -1455,7 +1456,7 @@ python3 "$(dirname "$0")/summarize-dual-vllm.py" \
   "$OUT/gpu1-vllm-metrics-before.txt" "$OUT/gpu1-vllm-metrics-after.txt"
 
 if ((QUALIFICATION_ONLY == 1)); then
-  jq -e '
+  if jq -e '
     .workload_isolation.success_delta_matches_controlled_log == true and
     .workload_isolation.controlled_errors_all_gpus_all_phases == 0 and
     .quality_gates.fan_telemetry_pass == true and
@@ -1477,10 +1478,31 @@ if ((QUALIFICATION_ONLY == 1)); then
       .sampled_counter_deltas_us.hw_thermal_slowdown_counter_us == 0 and
       .sampled_counter_deltas_us.hw_power_brake_counter_us == 0
     )
-  ' "$SUMMARY_JSON" >/dev/null || {
+  ' "$SUMMARY_JSON" >/dev/null; then
+    jq -n \
+      --arg evaluated_at "$(date -u +%FT%T.%3NZ)" \
+      '{
+        schema_version:"1.0",
+        qualification_only:true,
+        passed:true,
+        inferential_replicate:false,
+        evaluated_at:$evaluated_at,
+        gate:"isolation + loaded/idle power + fan telemetry/tracking + independent NVML + zero thermal/brake events and counters"
+      }' > "$QUALIFICATION_RESULT_JSON"
+  else
+    jq -n \
+      --arg evaluated_at "$(date -u +%FT%T.%3NZ)" \
+      '{
+        schema_version:"1.0",
+        qualification_only:true,
+        passed:false,
+        inferential_replicate:false,
+        evaluated_at:$evaluated_at,
+        gate:"isolation + loaded/idle power + fan telemetry/tracking + independent NVML + zero thermal/brake events and counters"
+      }' > "$QUALIFICATION_RESULT_JSON"
     echo "Qualification validation gate failed; see ${SUMMARY_JSON}" >&2
     exit 1
-  }
+  fi
 else
   jq -e '
     .workload_isolation.success_delta_matches_controlled_log == true and

hardware-tests/tower2-dual-qwen27-no-gap-2026-07-29/ng-faniso-own-loadbottom-f30-p300-v3host-bump-r1/REPORT.md

@@ -0,0 +1,31 @@
+# Own-fan 30% bottom-loaded 300 W safety bump
+
+- Run: `2026-07-30T21-22-21Z-ng-faniso-own-loadbottom-f30-p300-v3host-bump-r1`
+- Cell: `NG-FANISO-OWN-LOADBOTTOM-F30-P300-V3HOST-BUMP`
+- Purpose: qualify the first 30% loaded-fan condition before 15-minute cells
+- Measured window: 120 seconds after 120 seconds loaded warmup
+- Result: **qualification pass; safety-only and never an inferential replicate**
+
+| Metric | GPU0 / bottom loaded | GPU1 / top model-resident idle |
+|---|---:|---:|
+| Mean / maximum board power | 299.994 / 300.04 W | 21.944 / 25.81 W |
+| Mean / maximum temperature | 53.584 / 57 C | 40.342 / 42 C |
+| Mean fan duty | 30.0% | 50.0% |
+| Mean physical fan RPM | 1,200.586 | 1,678.934 |
+| Mean graphics clock | 1,035.3 MHz | 180.5 MHz |
+| GPU utilization | 100% | 0% |
+| SW/HW thermal or HW-brake active samples | 0 | 0 |
+
+The V3HOST preflight held continuously for 303 seconds and ended at 29/31 C
+GPUs, 54.4 C CPU Tctl, and 39.9 C hottest NVMe. GPU0 then held its 300 W cap
+at 30% fan with 28 C of observed margin to the 85 C emergency cutoff. The idle
+neighbor stayed below its 50 W isolation limit. Workload isolation, fan target
+tracking, four-fan RPM telemetry, independent NVML clocks, sampled slowdown
+counters, and automatic cleanup all passed.
+
+The original run process returned nonzero because the first implementation of
+the qualification command requested a hardware power-brake key not emitted in
+`counter_deltas_us`. The authoritative sampled key,
+`sampled_counter_deltas_us.hw_power_brake_counter_us`, exists and is zero for
+both GPUs. Commit `5a10c2f` corrected that schema mismatch; the preserved raw
+run then passed the corrected prospective gate without modification or rerun.

hardware-tests/tower2-dual-qwen27-no-gap-2026-07-29/ng-faniso-own-loadbottom-f30-p300-v3host-bump-r1/SHA256SUMS

@@ -0,0 +1,28 @@
+e71f9c089c569f2b774740bb9acccc1fdaf83663a5a1a6b0650fe674d6c86dc2  containers-before.txt
+8773e1057b90902916611b06edbe8a8fd235cd86e1c413e9f0e3c25c0d549b86  events.log
+527a11bc67df269dbd6ede370ed1075c4d7a1999f1da0b3b423871063c70dc08  gpu0-payload.json
+b4bdba3af0d845bb88e26f22dbbbde58bc5614174ba1abd9b9f53a7935c3d053  gpu0-vllm.log
+7f6d1e55ffa5676b0fc0c16e0baefed7299f358e112d81a9a78bc1ba18fa4d49  gpu1-payload.json
+8003d1067aa19e7c9460978393da6d955de1608cb981da5b3bc2bc2fae3e1bd4  gpu1-vllm-metrics-after.txt
+696f51b9bfccde655bf7fc6aef2d4e6b8eb15dacc4e91ba1a43c2323fbfd23d4  gpu1-vllm-metrics-before.txt
+6c52a9b98bc74d07857378ce05eb34b2ea7d4314249f1fb9c511500c681a3ae1  gpu-fan-telemetry.csv
+b7c51f01d6fb178663cea8918c5238701c21bdc96c8d375338f7278e0544b34e  gpu-nvml-clock.csv
+4d72e83985c9a5ee2d92f51f6b914cc98a640183df8fd44f6e6bdcf0380cbf97  gpu-processes-after.csv
+b6bad80dc12a9adc477f6ada36116b2b328954df64879bea32eb8b7c62542338  gpu-processes-before.csv
+b8bb3fc83af1a7ffed2250fc9494288fafc5f9f3503e18f1ca648f0f97459177  gpu-processes-isolated.csv
+d27f08187a0528ee4f96241616cf71941412255397777a0ff43ab607749fffe0  gpu-telemetry.csv
+643a68c0a5be8d1223374812db1128b78d99f81cd4c515a061effbb11316af74  host-before.txt
+b684e39fe67f77a6143502b618c49eaec6c976e454173a6fabd2d8738a86e539  host-telemetry.csv
+e8993fa688d903d0cebeb53836850b0837eb9b8ac5957e79a8a503d086a13d61  nvidia-after.txt
+923410e4d624d994763033f46c89c80807cbfe83aff55220ab552eba51111515  nvidia-after.xml
+e3bb3ebd0f3a9321901306d407abb12193612d452fb5ba565c5a4ebfed1bf419  nvidia-before.txt
+2e49c9e0adc758bb589fbd9474586ec446252641854d6e6de7f013810b239cec  nvidia-before.xml
+b871c89db680101657c867f401474a009bc05dc0d7df7e9c14908a886fb467e2  preflight-telemetry.csv
+0495287e775232107dab93d4cf423ac527d21c75ceb38a470c967002d9e546c6  requests.csv
+fc3c79966ac60461244f4687035b3b303a61f1b51e332c2b4d059dc9ba0f648b  run-config.json
+fd1bd9664947cbcf6545b1d18ec056c6c0b1c55607f3e87dcaaba04d5f34a7d1  run.log
+18c751db1d16580d8a0a9ab353b94b98e88173aa578649e6a40706431f87dc20  sanctuary-inspect.json
+36e6967245e9e87e106ddcf82979a6b6949e36ef87c49f9642f7168ab4828352  stopped-containers.txt
+0f68911c9969203fa83cf49b0838f97b2d5846776a3d01cd1654fa238ed11e04  stopped-user-services.txt
+65072632ac9f4b5400ff5f6731feb9210732bf27d779ce1913de3093f1cf54ce  summary.json
+96b1088362e73535d9f1c799f7efc42161493ba323bcb215a30a45f78adad95f  user-services-before.txt