Record 400W 120-point fan qualification

Author: Lightheartdevs

hardware-tests/tower2-dual-qwen27-no-gap-2026-07-29/README.md

@@ -351,6 +351,14 @@ were zero, so this is a safety-envelope failure rather than observed NVIDIA
 thermal throttling. The prospectively amended replacement raises the matched
 fan budget to 120 points (EQ60, B70T50, B50T70) without raising the cutoff.
 
+[`analysis/400W_STATIC_FAN_120POINT_QUALIFICATION.md`](analysis/400W_STATIC_FAN_120POINT_QUALIFICATION.md)
+records the replacement safety block. EQ60, B50T70, and B70T50 all passed at
+400/400 W with exact fan tracking and zero thermal/brake counter deltas. The
+hottest top observation was 76 C, leaving 9 C to the unchanged cutoff. The
+directional policies were only 0.133 C apart in top mean temperature, so the
+bumps establish safety—not an optimal allocation; three 15-minute Latin-order
+blocks remain required.
+
 ## Read order
 
 1. Each run directory's `REPORT.md` gives the human-readable result.

hardware-tests/tower2-dual-qwen27-no-gap-2026-07-29/STATIC_FAN_POWER_SPINE_PROTOCOL.md

@@ -119,6 +119,12 @@ RPM, power, completeness, plateau, and counter gates remain unchanged. If the
 120-point budget cannot maintain a safe plateau, the next pre-registered
 fallback is the matched 140-point EQ70/B80T60/B60T80 set; no cutoff is raised.
 
+The 120-point qualification completed on 2026-07-31. All three policies
+passed, with a hottest observed top temperature of 76 C and zero within-run
+thermal/brake counter deltas. This authorizes the pre-registered 15-minute
+Latin-order blocks; it does not count toward `n` or establish steady-state
+safety.
+
 ## Runner
 
 [`run-static-fan-power-block.sh`](run-static-fan-power-block.sh) implements

hardware-tests/tower2-dual-qwen27-no-gap-2026-07-29/VALIDATION_REGISTRY.csv

@@ -87,3 +87,6 @@ run_id,cell_id,replicate,configuration,bottom_power_w,top_power_w,load_pattern,d
 2026-07-31T12-40-06Z,NG-FAN-B60T40-SYM400-V3HOST-BUMP,1,no-gap-fixed-fan-v3host,400,400,loaded-loaded,120,qualification-pass,false,false,two-minute safety qualification is not an inferential replicate; static 400 W qualification sequence 3 of 3,,ng-fan-b60t40-sym400-v3host-bump-r1
 2026-07-31T12-58-43Z,NG-FAN-EQ50-SYM400-V3HOST-15M,1,no-gap-fixed-fan-v3host,400,400,loaded-loaded,900,pass,true,false,no calibrated ambient or local-inlet probes; maximum top temperature 84 C leaves only 1 C to static-spine cutoff,,ng-fan-eq50-sym400-v3host-15m-r1
 2026-07-31T13-30-58Z,NG-FAN-B60T40-SYM400-V3HOST-15M,1,no-gap-fixed-fan-v3host,400,400,loaded-loaded,900,safety-aborted,false,false,top reached 85 C cutoff after approximately 4 measured minutes; incomplete and non-steady; zero within-run thermal counter deltas,,ng-fan-b60t40-sym400-v3host-15m-r1
+2026-07-31T14-04-04Z,NG-FAN-EQ60-SYM400-V3HOST-BUMP,1,no-gap-fixed-fan-v3host,400,400,loaded-loaded,120,qualification-pass,false,false,two-minute safety qualification is not an inferential replicate; 120-point replacement sequence 1 of 3,,ng-fan-eq60-sym400-v3host-bump-r1
+2026-07-31T14-26-42Z,NG-FAN-B50T70-SYM400-V3HOST-BUMP,1,no-gap-fixed-fan-v3host,400,400,loaded-loaded,120,qualification-pass,false,false,two-minute safety qualification is not an inferential replicate; 120-point replacement sequence 2 of 3,,ng-fan-b50t70-sym400-v3host-bump-r1
+2026-07-31T14-44-25Z,NG-FAN-B70T50-SYM400-V3HOST-BUMP,1,no-gap-fixed-fan-v3host,400,400,loaded-loaded,120,qualification-pass,false,false,two-minute safety qualification is not an inferential replicate; 120-point replacement sequence 3 of 3,,ng-fan-b70t50-sym400-v3host-bump-r1

hardware-tests/tower2-dual-qwen27-no-gap-2026-07-29/VALIDATION_STATUS.md

@@ -40,6 +40,9 @@ Two validation levels are tracked:
 | NG-FAN-EQ50-SYM400-V3HOST-15M | 1 | 3 | 0 | R1 admissible but marginal at 84 C maximum; original 100-point budget retired at 400 W |
 | NG-FAN-B60T40-SYM400-V3HOST-15M | 0 | 3 | 0 | R1 safety-aborted at 85 C after ~4 measured minutes; do not repeat unchanged |
 | NG-FAN-B40T60-SYM400-V3HOST-15M | 0 | 3 | 0 | Not run because the R1 block correctly fail-stopped after B60T40 |
+| NG-FAN-EQ60-SYM400-V3HOST-15M | 0 | 3 | 0 | 120-point replacement qualified; three 15-minute Latin-order replicates pending |
+| NG-FAN-B50T70-SYM400-V3HOST-15M | 0 | 3 | 0 | 120-point replacement qualified; three 15-minute Latin-order replicates pending |
+| NG-FAN-B70T50-SYM400-V3HOST-15M | 0 | 3 | 0 | 120-point replacement qualified; three 15-minute Latin-order replicates pending |
 | NG-SYM-600 | 0 | 3 | 0 | Known failed pilot; do not repeat unchanged |
 
 `NG-SINGLE-T-250` is the first cell to reach three internally admissible replicates. Replicates 2, 4, and 5 were independently initialized execution blocks with cleanup/cooldown between them, but all occurred during one campaign session. The cell is therefore validated for the internal Tower2/no-gap model with an explicit within-session limitation. No cell is transferable yet.

hardware-tests/tower2-dual-qwen27-no-gap-2026-07-29/analysis/400W_STATIC_FAN_120POINT_QUALIFICATION.md

@@ -0,0 +1,30 @@
+# 400/400 W 120-point fan-budget qualification
+
+The prospectively amended 120-point fan budget passed all three independent
+two-minute safety bumps at 400/400 W. Every cell held both GPUs at 100%
+utilization and approximately 400 W, tracked its commanded fan settings, and
+recorded zero software/hardware thermal-slowdown or hardware-power-brake
+events and zero within-run thermal/brake counter deltas.
+
+| Policy | Bottom fan / RPM | Top fan / RPM | Bottom mean / max C | Top mean / max C | Bottom / top mean clock MHz |
+|---|---:|---:|---:|---:|---:|
+| EQ60 | 60% / 1,917 | 60% / 1,917 | 60.633 / 64 | 72.874 / 76 | 1,763.771 / 1,583.936 |
+| B50T70 | 50% / 1,679 | 70% / 2,158 | 58.327 / 62 | 70.218 / 75 | 1,788.282 / 1,611.296 |
+| B70T50 | 70% / 2,158 | 50% / 1,678 | 58.006 / 62 | 70.351 / 74 | 1,774.998 / 1,619.146 |
+
+The 120-point budget is substantially safer in the short window than the
+retired 100-point 400 W budget: the hottest top observation was 76 C, leaving
+9 C to the unchanged 85 C cutoff. All three cells remained thermally
+non-steady, however. The directional policies were nearly tied in top mean
+temperature (0.133 C apart), and their short-run clock/throughput ordering is
+not inferential. These bumps authorize, but cannot replace, the three
+pre-registered 15-minute Latin-order blocks.
+
+Artifacts:
+
+- [`400w-static-fan-120point-qualification.csv`](400w-static-fan-120point-qualification.csv)
+- [`400w-static-fan-120point-qualification.png`](400w-static-fan-120point-qualification.png)
+- [`../ng-fan-eq60-sym400-v3host-bump-r1/`](../ng-fan-eq60-sym400-v3host-bump-r1/)
+- [`../ng-fan-b50t70-sym400-v3host-bump-r1/`](../ng-fan-b50t70-sym400-v3host-bump-r1/)
+- [`../ng-fan-b70t50-sym400-v3host-bump-r1/`](../ng-fan-b70t50-sym400-v3host-bump-r1/)
+- [`../static-fan-power-sym400-fan120-qualify-r1/`](../static-fan-power-sym400-fan120-qualify-r1/)

hardware-tests/tower2-dual-qwen27-no-gap-2026-07-29/analysis/400w-static-fan-120point-qualification.csv

@@ -0,0 +1,4 @@
+policy,run_dir,bottom_fan_pct,top_fan_pct,bottom_fan_rpm_mean,top_fan_rpm_mean,bottom_power_mean_w,top_power_mean_w,bottom_temp_mean_c,top_temp_mean_c,bottom_temp_max_c,top_temp_max_c,bottom_clock_mean_mhz,top_clock_mean_mhz,bottom_requests_per_second,top_requests_per_second,bottom_request_duration_mean_s,top_request_duration_mean_s,thermal_or_brake_events,qualification_passed
+EQ60,ng-fan-eq60-sym400-v3host-bump-r1,60,60,1917.483,1917.384,399.992,399.991,60.633,72.874,64.0,76.0,1763.771,1583.936,1.6,1.3333,21.586,22.372,0,True
+B50T70,ng-fan-b50t70-sym400-v3host-bump-r1,50,70,1678.599,2157.574,399.99,399.992,58.327,70.218,62.0,75.0,1788.282,1611.296,1.6,1.3333,21.513,22.253,0,True
+B70T50,ng-fan-b70t50-sym400-v3host-bump-r1,70,50,2157.802,1678.492,399.992,399.992,58.006,70.351,62.0,74.0,1774.998,1619.146,1.6,1.6,21.553,22.217,0,True

hardware-tests/tower2-dual-qwen27-no-gap-2026-07-29/analysis/400w-static-fan-120point-qualification.png

@@ -12,11 +12,27 @@
 ROOT = Path(__file__).resolve().parent
 CAMPAIGN = ROOT.parent
 POWER = int(sys.argv[1]) if len(sys.argv) > 1 else 350
-POLICIES = (
-    ("EQ50", "eq50", 50, 50),
-    ("B40T60", "b40t60", 40, 60),
-    ("B60T40", "b60t40", 60, 40),
-)
+FAN_BUDGET = int(sys.argv[2]) if len(sys.argv) > 2 else 100
+POLICY_SETS = {
+    100: (
+        ("EQ50", "eq50", 50, 50),
+        ("B40T60", "b40t60", 40, 60),
+        ("B60T40", "b60t40", 60, 40),
+    ),
+    120: (
+        ("EQ60", "eq60", 60, 60),
+        ("B50T70", "b50t70", 50, 70),
+        ("B70T50", "b70t50", 70, 50),
+    ),
+    140: (
+        ("EQ70", "eq70", 70, 70),
+        ("B60T80", "b60t80", 60, 80),
+        ("B80T60", "b80t60", 80, 60),
+    ),
+}
+if FAN_BUDGET not in POLICY_SETS:
+    raise SystemExit(f"unsupported fan budget: {FAN_BUDGET}")
+POLICIES = POLICY_SETS[FAN_BUDGET]
 
 
 rows = []
@@ -63,7 +79,8 @@
         }
     )
 
-csv_path = ROOT / f"{POWER}w-static-fan-qualification.csv"
+suffix = "" if FAN_BUDGET == 100 else f"-{FAN_BUDGET}point"
+csv_path = ROOT / f"{POWER}w-static-fan{suffix}-qualification.csv"
 with csv_path.open("w", newline="", encoding="utf-8") as handle:
     writer = csv.DictWriter(handle, fieldnames=list(rows[0]))
     writer.writeheader()
@@ -100,7 +117,7 @@ def font(size, bold=False):
 )
 draw.text(
     (70, 98),
-    "Qwen3.6-27B AWQ-INT4 | 100% utilization | matched 100-point fan budget | three 120-second safety bumps",
+    f"Qwen3.6-27B AWQ-INT4 | 100% utilization | matched {FAN_BUDGET}-point fan budget | three 120-second safety bumps",
     font=font(20),
     fill=MUTED,
 )
@@ -186,7 +203,7 @@ def grouped_panel(box, title, bottom_key, top_key, y_min, y_max, decimals):
     "bottom_fan_rpm_mean",
     "top_fan_rpm_mean",
     1100,
-    2050,
+    2300 if FAN_BUDGET >= 120 else 2050,
     0,
 )
 
@@ -207,7 +224,7 @@ def grouped_panel(box, title, bottom_key, top_key, y_min, y_max, decimals):
     fill=MUTED,
 )
 
-png_path = ROOT / f"{POWER}w-static-fan-qualification.png"
+png_path = ROOT / f"{POWER}w-static-fan{suffix}-qualification.png"
 image.save(png_path)
 print(csv_path)
 print(png_path)

hardware-tests/tower2-dual-qwen27-no-gap-2026-07-29/analysis/plot-static-fan-power-qualification.py

@@ -0,0 +1,21 @@
+# 400/400 W B50T70 fixed-fan safety bump
+
+- Run: `2026-07-31T14-26-42Z-ng-fan-b50t70-sym400-v3host-bump-r1`
+- Cell: `NG-FAN-B50T70-SYM400-V3HOST-BUMP`
+- Policy: GPU0/bottom 50%, GPU1/top 70%
+- Result: **qualification pass; safety-only and non-inferential**
+
+| Metric | GPU0 / bottom | GPU1 / top |
+|---|---:|---:|
+| Mean board power | 399.990 W | 399.992 W |
+| Mean / maximum temperature | 58.327 / 62 C | 70.218 / 75 C |
+| Fan / mean RPM | 50% / 1,678.599 | 70% / 2,157.574 |
+| Mean graphics clock | 1,788.282 MHz | 1,611.296 MHz |
+| Requests/s | 1.6000 | 1.3333 |
+| Mean request duration | 21.513 s | 22.253 s |
+
+Both GPUs held 100% utilization and the requested power cap. Fan target
+tracking, workload isolation, primary telemetry, and the independent NVML
+clock stream passed. Thermal and hardware-power-brake event samples and
+counter deltas were zero. The cell was only 120 measured seconds and retained
+positive temperature slopes; it does not count toward `n`.

hardware-tests/tower2-dual-qwen27-no-gap-2026-07-29/ng-fan-b50t70-sym400-v3host-bump-r1/REPORT.md

@@ -0,0 +1,29 @@
+365fad97910d78aa966c3152005b1282f507b35311ccdb32806f0c1c00d6e51a  containers-before.txt
+25b33442ca605900320f44fbd9890e26cd76157ec1853adb281b647a3c1064b2  events.log
+527a11bc67df269dbd6ede370ed1075c4d7a1999f1da0b3b423871063c70dc08  gpu0-payload.json
+6a47b4446adf9a0e165a5a10a4c838e3250cb4a454daa92ad6f1ccf0800c2603  gpu0-vllm.log
+7f6d1e55ffa5676b0fc0c16e0baefed7299f358e112d81a9a78bc1ba18fa4d49  gpu1-payload.json
+6804f8daf409102d22198766042ea67fd199bc1f7fa96a75c53645541ec917fd  gpu1-vllm-metrics-after.txt
+a3b09ee89f7ff4d34cb0837a2bb57b3e8a6e26f81a880864302317d72ce38131  gpu1-vllm-metrics-before.txt
+e72b9d5d05bb3c18e3c8b8c24066280bdd8bf277082048e0c5f17f2347b24a3c  gpu-fan-telemetry.csv
+ac03886e0ca535f2d60af0bc429dd0cb0a1f9b08b271adcb8b394c44663423cd  gpu-nvml-clock.csv
+ae760b5ec9df2e93699f2f0dfdbf9fef4e239d09c1f388316a1684e8d16d53ac  gpu-processes-after.csv
+b81e28c89a720fbfdba2c8626a371ef666d2b9222e64b9ab5784b83777ee6ca6  gpu-processes-before.csv
+f82836b191c7465c95f42cc413460b475d9b63b848f069fd9dd60c62b2f65d72  gpu-processes-isolated.csv
+bf4fb57558b90d9d0b9232b8e7930b7ef38881e1a0749de68cf74c7b3302524e  gpu-telemetry.csv
+643a68c0a5be8d1223374812db1128b78d99f81cd4c515a061effbb11316af74  host-before.txt
+92aad62e34ce6f93ac84fbd70cc45116c780b7f1580cf1727e2757297faf18a4  host-telemetry.csv
+935c01a83c6d6bbfb4d9e7037da6f6a5519e8e75bf3cc56a5d5f8f453ec44add  nvidia-after.txt
+b5059bec4e3030f69248fe9473c532acbf52bd80dd0b340d10f806943ca0ef16  nvidia-after.xml
+ad7a77b78f75581a11b0a7e5598c37092477b508bd7a3a9b94e3104d5ec6bffa  nvidia-before.txt
+5968914a7cb8eb5805e170aa4c08456c6839b3036df60950a0524795841c28d1  nvidia-before.xml
+6de9bc2a1205fb1d7af34f6edefec84392cddbd8ebada95c0f2548d11a744826  preflight-telemetry.csv
+cd6f2574d429847d4d299d5dffa59b22dfcebe0318dc019a25f946c3240454a4  qualification-result.json
+ab9ba18f45433d7cd32d839647477aaae4648a26efdb64e5f465907595f710c0  requests.csv
+284697bf7aa1fab7afc39938f1a3f6dce461ca2dc753e71a19e2e8bfd8351f93  run-config.json
+e8726ef621a6cdc11b2c930b64e08c2a0b450e1f33d97d09b8827309264c2659  run.log
+a8c4ec97c7996bcaeae2c953f10947b3649c981cbd51fcc3aaa05828bf8eefa8  sanctuary-inspect.json
+36e6967245e9e87e106ddcf82979a6b6949e36ef87c49f9642f7168ab4828352  stopped-containers.txt
+0f68911c9969203fa83cf49b0838f97b2d5846776a3d01cd1654fa238ed11e04  stopped-user-services.txt
+634d14ce77ae1285c53ad9e57b93b7080a87ee123346f604c6881d56af6552ab  summary.json
+561a3b13b8a70e6fb25a40e88b3e910dbb225fef2edc271915a5a0510db0e9c7  user-services-before.txt