Widen kernel launch latency tolerance to cover scheduler variance

The latency benchmarks measure identical launch code at values that differ by
a uniform ~3.5 microseconds depending on which GPU and CPU cgroups SLURM
assigns to the job: builds 7080 and 7081 ran the same host launch path on the
same node and measured 16990 vs 20530 ns (GPU index 3 with CPU 3 vs GPU index
0 with CPU 0). An atol of 2000 ns cannot distinguish this assignment variance
from a real regression, so raise it to 4000 ns, shared by all three
benchmarks. A genuine per-launch regression like the one caught in round 17
(+7000 ns) would still fail.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: dennisYatunin

test/gpu/latency_benchmarks.jl

@@ -31,7 +31,12 @@ import LazyBroadcast: lazy
     latency = median(@benchmark $scalar_field_1 .= $scalar_field_1 .+ $scalar_field_2).time
     # update this value if the kernel launch time changes significantly and it is expected
     baseline_latency = 18000
-    @test latency ≈ baseline_latency atol = 2000
+    # The tolerance must cover scheduler-assignment variance: identical code
+    # measures launch latencies that differ by up to ~3.5 μs depending on which
+    # GPU and CPU cgroups SLURM assigns to the job (builds 7080 and 7081 ran on
+    # the same node with different GPU indices and measured 16990 and 20530 ns).
+    latency_atol = 4000
+    @test latency ≈ baseline_latency atol = latency_atol
     percent_change_latency =
         round(Int, (latency - baseline_latency) / baseline_latency * 100)
     @info "Latency: $latency ns, Percent change from baseline: $percent_change_latency%"
@@ -45,7 +50,7 @@ import LazyBroadcast: lazy
         ).time
     # update this value if the kernel launch time changes significantly and it is expected
     baseline_latency = 28500
-    @test latency ≈ baseline_latency atol = 2000
+    @test latency ≈ baseline_latency atol = latency_atol
     percent_change_latency =
         round(Int, (latency - baseline_latency) / baseline_latency * 100)
     @info "Latency: $latency ns, Percent change from baseline: $percent_change_latency%"
@@ -58,7 +63,7 @@ import LazyBroadcast: lazy
     latency = median(@benchmark $scalar_field_1 .= $lazy_sum_3).time
     # update this value if the kernel launch time changes significantly and it is expected
     baseline_latency = 46000
-    @test latency ≈ baseline_latency atol = 2000
+    @test latency ≈ baseline_latency atol = latency_atol
     percent_change_latency =
         round(Int, (latency - baseline_latency) / baseline_latency * 100)
     @info "Latency: $latency ns, Percent change from baseline: $percent_change_latency%"