Also compare batch measurements in nvbench_compare.py #263

bernhardmgruber · 2025-08-14T17:18:19Z

Fixes: #247

Cold and batch measurements can sometimes differ substantially, so we want to show both. An example is kernels using PDL (Programmatic Dependent Launch).

Here is a comparison of DeviceTransform with and without PDL (see also NVIDIA/cccl#5249):

# mul

## [0] NVIDIA B200

|  T{ct}  |  OffsetT{ct}  |  Elements{io}  |   Ref Time |   Ref Noise |   Cmp Time |   Cmp Noise |      Diff |   %Diff |  Status  |   B Ref Time |   B Cmp Time |    B Diff |   B %Diff |  B Status  |
|---------|---------------|----------------|------------|-------------|------------|-------------|-----------|---------|----------|--------------|--------------|-----------|-----------|------------|
|   I8    |      I32      |      2^16      |   5.729 us |      11.77% |   5.734 us |      11.65% |  0.005 us |   0.08% |   SAME   |     4.094 us |     1.622 us | -2.472 us |   -60.39% |    FAST    |
|   I8    |      I32      |      2^20      |   6.016 us |       5.61% |   6.130 us |       7.31% |  0.114 us |   1.90% |   SAME   |     4.101 us |     2.831 us | -1.270 us |   -30.97% |    FAST    |
|   I8    |      I32      |      2^24      |  13.739 us |       5.56% |  13.620 us |       5.84% | -0.119 us |  -0.87% |   SAME   |    10.271 us |     8.388 us | -1.883 us |   -18.33% |    FAST    |
|   I8    |      I32      |      2^28      | 114.526 us |       0.25% | 114.550 us |       0.28% |  0.024 us |   0.02% |   SAME   |   112.599 us |   110.039 us | -2.560 us |    -2.27% |    FAST    |
|   I8    |      I64      |      2^16      |   5.494 us |      14.80% |   5.512 us |      14.22% |  0.018 us |   0.33% |   SAME   |     4.094 us |     1.594 us | -2.500 us |   -61.06% |    FAST    |
|   I8    |      I64      |      2^20      |   6.059 us |       5.71% |   6.383 us |       9.59% |  0.324 us |   5.35% |   SAME   |     4.101 us |     2.826 us | -1.274 us |   -31.07% |    FAST    |
|   I8    |      I64      |      2^24      |  14.075 us |       2.96% |  14.041 us |       3.52% | -0.035 us |  -0.25% |   SAME   |    10.290 us |     8.436 us | -1.854 us |   -18.02% |    FAST    |
|   I8    |      I64      |      2^28      | 115.311 us |       0.80% | 115.433 us |       0.82% |  0.122 us |   0.11% |   SAME   |   112.952 us |   110.872 us | -2.079 us |    -1.84% |    FAST    |
|   I16   |      I32      |      2^16      |   5.346 us |      16.26% |   5.377 us |      15.65% |  0.031 us |   0.58% |   SAME   |     4.094 us |     1.599 us | -2.496 us |   -60.96% |    FAST    |
|   I16   |      I32      |      2^20      |   6.099 us |       6.33% |   6.380 us |       9.76% |  0.281 us |   4.61% |   SAME   |     4.099 us |     2.815 us | -1.285 us |   -31.34% |    FAST    |
|   I16   |      I32      |      2^24      |  16.382 us |       3.32% |  16.376 us |       3.23% | -0.006 us |  -0.04% |   SAME   |    12.288 us |    10.074 us | -2.214 us |   -18.02% |    FAST    |
|   I16   |      I32      |      2^28      | 161.785 us |       0.28% | 161.935 us |       0.44% |  0.150 us |   0.09% |   SAME   |   159.618 us |   156.623 us | -2.996 us |    -1.88% |    FAST    |
|   I16   |      I64      |      2^16      |   5.672 us |      12.72% |   5.756 us |      11.04% |  0.084 us |   1.48% |   SAME   |     4.094 us |     1.577 us | -2.518 us |   -61.49% |    FAST    |
|   I16   |      I64      |      2^20      |   6.129 us |       6.10% |   6.391 us |      10.08% |  0.262 us |   4.28% |   SAME   |     4.102 us |     2.816 us | -1.286 us |   -31.36% |    FAST    |
|   I16   |      I64      |      2^24      |  16.434 us |       3.20% |  16.467 us |       3.44% |  0.034 us |   0.21% |   SAME   |    12.289 us |    10.120 us | -2.169 us |   -17.65% |    FAST    |
|   I16   |      I64      |      2^28      | 163.663 us |       0.22% | 163.721 us |       0.16% |  0.058 us |   0.04% |   SAME   |   160.345 us |   158.210 us | -2.135 us |    -1.33% |    FAST    |
|   F32   |      I32      |      2^16      |   5.935 us |       6.94% |   5.907 us |       6.24% | -0.028 us |  -0.47% |   SAME   |     4.094 us |     1.595 us | -2.499 us |   -61.04% |    FAST    |
|   F32   |      I32      |      2^20      |   7.543 us |      10.91% |   7.497 us |      10.90% | -0.046 us |  -0.61% |   SAME   |     4.099 us |     2.856 us | -1.244 us |   -30.34% |    FAST    |
|   F32   |      I32      |      2^24      |  25.470 us |       3.65% |  25.395 us |       3.59% | -0.075 us |  -0.29% |   SAME   |    20.843 us |    19.593 us | -1.251 us |    -6.00% |    FAST    |
|   F32   |      I32      |      2^28      | 313.226 us |       0.32% | 313.291 us |       0.30% |  0.065 us |   0.02% |   SAME   |   311.189 us |   308.332 us | -2.857 us |    -0.92% |    SAME    |
|   F32   |      I64      |      2^16      |   5.545 us |      14.41% |   5.487 us |      14.25% | -0.058 us |  -1.05% |   SAME   |     4.094 us |     1.600 us | -2.494 us |   -60.92% |    FAST    |
|   F32   |      I64      |      2^20      |   7.494 us |      11.26% |   7.432 us |      11.17% | -0.062 us |  -0.83% |   SAME   |     4.099 us |     2.859 us | -1.241 us |   -30.26% |    FAST    |
|   F32   |      I64      |      2^24      |  25.602 us |       3.56% |  25.608 us |       3.53% |  0.006 us |   0.02% |   SAME   |    20.726 us |    19.591 us | -1.136 us |    -5.48% |    FAST    |
|   F32   |      I64      |      2^28      | 313.271 us |       0.33% | 313.254 us |       0.31% | -0.017 us |  -0.01% |   SAME   |   311.100 us |   308.282 us | -2.818 us |    -0.91% |    SAME    |
|   F64   |      I32      |      2^16      |   5.706 us |      11.69% |   5.719 us |      11.38% |  0.014 us |   0.24% |   SAME   |     4.094 us |     1.596 us | -2.498 us |   -61.01% |    FAST    |
|   F64   |      I32      |      2^20      |   8.084 us |       4.22% |   8.043 us |       4.93% | -0.041 us |  -0.51% |   SAME   |     4.108 us |     2.882 us | -1.226 us |   -29.85% |    FAST    |
|   F64   |      I32      |      2^24      |  45.629 us |       2.07% |  45.498 us |       1.93% | -0.131 us |  -0.29% |   SAME   |    43.046 us |    40.308 us | -2.738 us |    -6.36% |    FAST    |
|   F64   |      I32      |      2^28      | 620.092 us |       0.23% | 620.155 us |       0.19% |  0.063 us |   0.01% |   SAME   |   617.714 us |   614.871 us | -2.843 us |    -0.46% |    SAME    |
|   F64   |      I64      |      2^16      |   5.698 us |      11.99% |   5.698 us |      11.30% | -0.001 us |  -0.01% |   SAME   |     4.094 us |     1.616 us | -2.478 us |   -60.53% |    FAST    |
|   F64   |      I64      |      2^20      |   8.098 us |       4.25% |   8.032 us |       4.07% | -0.067 us |  -0.82% |   SAME   |     4.106 us |     2.896 us | -1.210 us |   -29.47% |    FAST    |
|   F64   |      I64      |      2^24      |  45.517 us |       2.01% |  45.637 us |       2.05% |  0.119 us |   0.26% |   SAME   |    43.031 us |    40.252 us | -2.779 us |    -6.46% |    FAST    |
|   F64   |      I64      |      2^28      | 620.032 us |       0.22% | 620.114 us |       0.22% |  0.082 us |   0.01% |   SAME   |   617.629 us |   614.842 us | -2.786 us |    -0.45% |    SAME    |
|  I128   |      I32      |      2^16      |   5.959 us |       4.88% |   5.917 us |       5.20% | -0.042 us |  -0.71% |   SAME   |     4.094 us |     1.596 us | -2.499 us |   -61.03% |    FAST    |
|  I128   |      I32      |      2^20      |  10.308 us |       4.86% |  10.378 us |       5.50% |  0.070 us |   0.67% |   SAME   |     6.141 us |     5.100 us | -1.042 us |   -16.97% |    FAST    |
|  I128   |      I32      |      2^24      |  83.570 us |       0.98% |  83.639 us |       0.89% |  0.068 us |   0.08% |   SAME   |    81.530 us |    78.615 us | -2.916 us |    -3.58% |    FAST    |
|  I128   |      I32      |      2^28      |   1.235 ms |       0.15% |   1.235 ms |       0.15% | -0.010 us |  -0.00% |   SAME   |     1.233 ms |     1.231 ms | -2.248 us |    -0.18% |    SAME    |
|  I128   |      I64      |      2^16      |   5.979 us |       4.55% |   5.995 us |       4.38% |  0.016 us |   0.27% |   SAME   |     4.094 us |     1.601 us | -2.493 us |   -60.90% |    FAST    |
|  I128   |      I64      |      2^20      |  10.320 us |       4.89% |  10.433 us |       6.27% |  0.113 us |   1.10% |   SAME   |     6.141 us |     5.091 us | -1.050 us |   -17.10% |    FAST    |
|  I128   |      I64      |      2^24      |  83.769 us |       0.74% |  83.803 us |       0.77% |  0.034 us |   0.04% |   SAME   |    81.544 us |    78.602 us | -2.942 us |    -3.61% |    FAST    |
|  I128   |      I64      |      2^28      |   1.234 ms |       0.15% |   1.234 ms |       0.14% | -0.059 us |  -0.00% |   SAME   |     1.233 ms |     1.231 ms | -1.907 us |    -0.15% |    SAME    |

# add

## [0] NVIDIA B200

|  T{ct}  |  OffsetT{ct}  |  Elements{io}  |   Ref Time |   Ref Noise |   Cmp Time |   Cmp Noise |       Diff |   %Diff |  Status  |   B Ref Time |   B Cmp Time |     B Diff |   B %Diff |  B Status  |
|---------|---------------|----------------|------------|-------------|------------|-------------|------------|---------|----------|--------------|--------------|------------|-----------|------------|
|   I8    |      I32      |      2^16      |   5.829 us |       9.25% |   5.838 us |       8.67% |   0.009 us |   0.16% |   SAME   |     4.094 us |     1.594 us |  -2.500 us |   -61.06% |    FAST    |
|   I8    |      I32      |      2^20      |   6.113 us |       6.09% |   6.388 us |      10.02% |   0.275 us |   4.49% |   SAME   |     4.098 us |     3.051 us |  -1.047 us |   -25.55% |    FAST    |
|   I8    |      I32      |      2^24      |  15.292 us |       6.13% |  15.207 us |       6.09% |  -0.085 us |  -0.56% |   SAME   |    11.264 us |     9.365 us |  -1.899 us |   -16.86% |    FAST    |
|   I8    |      I32      |      2^28      | 141.213 us |       0.11% | 141.350 us |       0.41% |   0.137 us |   0.10% |   SAME   |   137.296 us |   135.994 us |  -1.302 us |    -0.95% |    SAME    |
|   I8    |      I64      |      2^16      |   5.919 us |       7.47% |   5.926 us |       6.03% |   0.007 us |   0.12% |   SAME   |     4.094 us |     1.604 us |  -2.490 us |   -60.82% |    FAST    |
|   I8    |      I64      |      2^20      |   6.111 us |       5.90% |   6.422 us |      10.09% |   0.310 us |   5.07% |   SAME   |     4.098 us |     3.097 us |  -1.001 us |   -24.43% |    FAST    |
|   I8    |      I64      |      2^24      |  15.775 us |       5.14% |  15.685 us |       5.19% |  -0.091 us |  -0.57% |   SAME   |    11.875 us |     9.387 us |  -2.489 us |   -20.96% |    FAST    |
|   I8    |      I64      |      2^28      | 143.584 us |       0.49% | 143.338 us |       0.34% |  -0.246 us |  -0.17% |   SAME   |   141.093 us |   137.760 us |  -3.333 us |    -2.36% |    FAST    |
|   I16   |      I32      |      2^16      |   5.936 us |       6.48% |   5.841 us |       9.35% |  -0.095 us |  -1.60% |   SAME   |     4.094 us |     1.627 us |  -2.467 us |   -60.26% |    FAST    |
|   I16   |      I32      |      2^20      |   6.649 us |      11.53% |   6.616 us |      11.37% |  -0.033 us |  -0.49% |   SAME   |     4.098 us |     3.023 us |  -1.075 us |   -26.23% |    FAST    |
|   I16   |      I32      |      2^24      |  22.386 us |       1.29% |  22.510 us |       2.15% |   0.123 us |   0.55% |   SAME   |    16.525 us |    18.175 us |   1.650 us |     9.98% |    SLOW    |
|   I16   |      I32      |      2^28      | 237.871 us |       0.33% | 272.302 us |       0.11% |  34.431 us |  14.47% |   SLOW   |   233.873 us |   268.593 us |  34.719 us |    14.85% |    SLOW    |
|   I16   |      I64      |      2^16      |   5.623 us |      13.25% |   5.612 us |      13.14% |  -0.011 us |  -0.20% |   SAME   |     4.094 us |     1.604 us |  -2.490 us |   -60.82% |    FAST    |
|   I16   |      I64      |      2^20      |   6.673 us |      11.74% |   6.706 us |      11.73% |   0.033 us |   0.50% |   SAME   |     4.098 us |     3.101 us |  -0.997 us |   -24.32% |    FAST    |
|   I16   |      I64      |      2^24      |  22.365 us |       1.38% |  22.528 us |       1.83% |   0.163 us |   0.73% |   SAME   |    16.525 us |    18.208 us |   1.683 us |    10.19% |    SLOW    |
|   I16   |      I64      |      2^28      | 239.449 us |       0.17% | 272.324 us |       0.19% |  32.875 us |  13.73% |   SLOW   |   235.292 us |   268.593 us |  33.301 us |    14.15% |    SLOW    |
|   F32   |      I32      |      2^16      |   5.787 us |      10.26% |   5.779 us |      10.38% |  -0.008 us |  -0.13% |   SAME   |     4.094 us |     1.575 us |  -2.519 us |   -61.52% |    FAST    |
|   F32   |      I32      |      2^20      |   8.008 us |       3.51% |   7.970 us |       3.98% |  -0.038 us |  -0.47% |   SAME   |     4.098 us |     2.875 us |  -1.223 us |   -29.84% |    FAST    |
|   F32   |      I32      |      2^24      |  35.876 us |       2.71% |  38.696 us |       0.88% |   2.820 us |   7.86% |   SLOW   |    32.666 us |    34.847 us |   2.182 us |     6.68% |    SLOW    |
|   F32   |      I32      |      2^28      | 454.211 us |       0.26% | 539.121 us |       0.17% |  84.909 us |  18.69% |   SLOW   |   452.508 us |   535.671 us |  83.163 us |    18.38% |    SLOW    |
|   F32   |      I64      |      2^16      |   5.958 us |       4.79% |   5.942 us |       5.19% |  -0.017 us |  -0.28% |   SAME   |     4.094 us |     1.590 us |  -2.504 us |   -61.17% |    FAST    |
|   F32   |      I64      |      2^20      |   8.016 us |       3.59% |   8.033 us |       3.33% |   0.018 us |   0.22% |   SAME   |     4.098 us |     2.880 us |  -1.218 us |   -29.73% |    FAST    |
|   F32   |      I64      |      2^24      |  35.917 us |       2.54% |  38.719 us |       0.85% |   2.802 us |   7.80% |   SLOW   |    32.694 us |    34.865 us |   2.171 us |     6.64% |    SLOW    |
|   F32   |      I64      |      2^28      | 453.938 us |       0.27% | 539.019 us |       0.16% |  85.081 us |  18.74% |   SLOW   |   452.062 us |   535.638 us |  83.577 us |    18.49% |    SLOW    |
|   F64   |      I32      |      2^16      |   6.006 us |       4.47% |   5.973 us |       4.50% |  -0.033 us |  -0.55% |   SAME   |     4.094 us |     1.594 us |  -2.501 us |   -61.08% |    FAST    |
|   F64   |      I32      |      2^20      |  10.086 us |       2.80% |  10.059 us |       3.17% |  -0.027 us |  -0.26% |   SAME   |     6.141 us |     5.197 us |  -0.944 us |   -15.37% |    FAST    |
|   F64   |      I32      |      2^24      |  65.164 us |       1.05% |  71.574 us |       0.51% |   6.411 us |   9.84% |   SLOW   |    59.868 us |    68.208 us |   8.340 us |    13.93% |    SLOW    |
|   F64   |      I32      |      2^28      | 909.760 us |       0.20% |   1.073 ms |       0.05% | 163.440 us |  17.97% |   SLOW   |   905.075 us |     1.070 ms | 164.766 us |    18.20% |    SLOW    |
|   F64   |      I64      |      2^16      |   5.962 us |       4.90% |   5.957 us |       5.03% |  -0.005 us |  -0.09% |   SAME   |     4.094 us |     1.583 us |  -2.511 us |   -61.34% |    FAST    |
|   F64   |      I64      |      2^20      |  10.092 us |       2.78% |  10.089 us |       2.82% |  -0.003 us |  -0.03% |   SAME   |     6.141 us |     5.192 us |  -0.950 us |   -15.46% |    FAST    |
|   F64   |      I64      |      2^24      |  65.317 us |       0.74% |  71.592 us |       0.53% |   6.275 us |   9.61% |   SLOW   |    59.828 us |    68.206 us |   8.378 us |    14.00% |    SLOW    |
|   F64   |      I64      |      2^28      | 909.178 us |       0.19% |   1.073 ms |       0.06% | 164.056 us |  18.04% |   SLOW   |   904.768 us |     1.070 ms | 165.069 us |    18.24% |    SLOW    |
|  I128   |      I32      |      2^16      |   5.997 us |       4.65% |   5.947 us |       5.01% |  -0.050 us |  -0.83% |   SAME   |     4.094 us |     1.593 us |  -2.501 us |   -61.10% |    FAST    |
|  I128   |      I32      |      2^20      |  14.195 us |       1.87% |  14.194 us |       1.96% |  -0.001 us |  -0.00% |   SAME   |     8.188 us |     9.372 us |   1.184 us |    14.46% |    SLOW    |
|  I128   |      I32      |      2^24      | 120.742 us |       0.76% | 138.904 us |       0.47% |  18.162 us |  15.04% |   SLOW   |   116.723 us |   134.950 us |  18.227 us |    15.62% |    SLOW    |
|  I128   |      I32      |      2^28      |   1.811 ms |       0.15% |   2.142 ms |       0.02% | 330.608 us |  18.25% |   SLOW   |     1.807 ms |     2.140 ms | 333.116 us |    18.44% |    SLOW    |
|  I128   |      I64      |      2^16      |   5.979 us |       5.28% |   5.991 us |       4.69% |   0.012 us |   0.21% |   SAME   |     4.094 us |     1.594 us |  -2.500 us |   -61.07% |    FAST    |
|  I128   |      I64      |      2^20      |  14.176 us |       1.96% |  14.138 us |       2.08% |  -0.038 us |  -0.27% |   SAME   |     8.188 us |     9.378 us |   1.190 us |    14.53% |    SLOW    |
|  I128   |      I64      |      2^24      | 120.699 us |       0.75% | 138.979 us |       0.35% |  18.280 us |  15.15% |   SLOW   |   116.648 us |   134.950 us |  18.303 us |    15.69% |    SLOW    |
|  I128   |      I64      |      2^28      |   1.811 ms |       0.15% |   2.142 ms |       0.02% | 330.955 us |  18.28% |   SLOW   |     1.805 ms |     2.140 ms | 334.179 us |    18.51% |    SLOW    |

# triad

## [0] NVIDIA B200

|  T{ct}  |  OffsetT{ct}  |  Elements{io}  |   Ref Time |   Ref Noise |   Cmp Time |   Cmp Noise |       Diff |   %Diff |  Status  |   B Ref Time |   B Cmp Time |     B Diff |   B %Diff |  B Status  |
|---------|---------------|----------------|------------|-------------|------------|-------------|------------|---------|----------|--------------|--------------|------------|-----------|------------|
|   I8    |      I32      |      2^16      |   5.814 us |       9.73% |   5.905 us |       6.76% |   0.091 us |   1.57% |   SAME   |     4.094 us |     1.602 us |  -2.493 us |   -60.88% |    FAST    |
|   I8    |      I32      |      2^20      |   6.128 us |       6.23% |   6.365 us |       9.87% |   0.237 us |   3.87% |   SAME   |     4.098 us |     3.039 us |  -1.059 us |   -25.83% |    FAST    |
|   I8    |      I32      |      2^24      |  16.207 us |       1.78% |  16.208 us |       1.79% |   0.001 us |   0.01% |   SAME   |    12.294 us |    10.362 us |  -1.932 us |   -15.72% |    FAST    |
|   I8    |      I32      |      2^28      | 149.994 us |       0.59% | 150.658 us |       0.63% |   0.664 us |   0.44% |   SAME   |   147.459 us |   145.001 us |  -2.458 us |    -1.67% |    FAST    |
|   I8    |      I64      |      2^16      |   6.006 us |       4.56% |   5.974 us |       4.71% |  -0.032 us |  -0.53% |   SAME   |     4.094 us |     1.613 us |  -2.481 us |   -60.60% |    FAST    |
|   I8    |      I64      |      2^20      |   6.083 us |       5.23% |   6.343 us |       9.49% |   0.260 us |   4.27% |   SAME   |     4.098 us |     3.083 us |  -1.015 us |   -24.76% |    FAST    |
|   I8    |      I64      |      2^24      |  16.208 us |       1.73% |  16.172 us |       1.79% |  -0.037 us |  -0.23% |   SAME   |    12.294 us |    10.623 us |  -1.671 us |   -13.59% |    FAST    |
|   I8    |      I64      |      2^28      | 153.088 us |       0.50% | 153.434 us |       0.21% |   0.346 us |   0.23% |   SLOW   |   149.586 us |   147.658 us |  -1.928 us |    -1.29% |    FAST    |
|   I16   |      I32      |      2^16      |   5.718 us |      11.90% |   5.721 us |      11.60% |   0.003 us |   0.05% |   SAME   |     4.094 us |     1.605 us |  -2.489 us |   -60.81% |    FAST    |
|   I16   |      I32      |      2^20      |   6.924 us |      13.05% |   7.008 us |      12.94% |   0.084 us |   1.21% |   SAME   |     4.098 us |     2.979 us |  -1.119 us |   -27.30% |    FAST    |
|   I16   |      I32      |      2^24      |  22.350 us |       1.38% |  22.718 us |       2.59% |   0.369 us |   1.65% |   SLOW   |    16.474 us |    18.079 us |   1.605 us |     9.74% |    SLOW    |
|   I16   |      I32      |      2^28      | 239.548 us |       0.12% | 272.308 us |       0.13% |  32.761 us |  13.68% |   SLOW   |   235.182 us |   268.432 us |  33.249 us |    14.14% |    SLOW    |
|   I16   |      I64      |      2^16      |   5.888 us |       8.48% |   5.782 us |       9.90% |  -0.106 us |  -1.80% |   SAME   |     4.094 us |     1.592 us |  -2.502 us |   -61.12% |    FAST    |
|   I16   |      I64      |      2^20      |   7.166 us |      12.98% |   7.195 us |      12.81% |   0.030 us |   0.41% |   SAME   |     4.098 us |     3.083 us |  -1.015 us |   -24.78% |    FAST    |
|   I16   |      I64      |      2^24      |  22.381 us |       1.20% |  22.704 us |       2.70% |   0.322 us |   1.44% |   SLOW   |    17.928 us |    18.183 us |   0.255 us |     1.42% |    SLOW    |
|   I16   |      I64      |      2^28      | 241.571 us |       0.12% | 272.374 us |       0.17% |  30.803 us |  12.75% |   SLOW   |   237.152 us |   268.438 us |  31.286 us |    13.19% |    SLOW    |
|   F32   |      I32      |      2^16      |   5.989 us |       4.50% |   5.950 us |       4.88% |  -0.039 us |  -0.66% |   SAME   |     4.094 us |     1.594 us |  -2.500 us |   -61.06% |    FAST    |
|   F32   |      I32      |      2^20      |   8.021 us |       3.47% |   7.989 us |       3.89% |  -0.032 us |  -0.39% |   SAME   |     4.098 us |     2.865 us |  -1.233 us |   -30.09% |    FAST    |
|   F32   |      I32      |      2^24      |  36.517 us |       1.70% |  38.782 us |       0.80% |   2.266 us |   6.20% |   SLOW   |    32.731 us |    34.859 us |   2.128 us |     6.50% |    SLOW    |
|   F32   |      I32      |      2^28      | 453.094 us |       0.24% | 539.057 us |       0.16% |  85.962 us |  18.97% |   SLOW   |   452.672 us |   535.696 us |  83.024 us |    18.34% |    SLOW    |
|   F32   |      I64      |      2^16      |   5.973 us |       4.63% |   5.960 us |       5.83% |  -0.014 us |  -0.23% |   SAME   |     4.094 us |     1.606 us |  -2.488 us |   -60.77% |    FAST    |
|   F32   |      I64      |      2^20      |   8.038 us |       3.43% |   8.009 us |       3.70% |  -0.028 us |  -0.35% |   SAME   |     4.098 us |     2.870 us |  -1.228 us |   -29.96% |    FAST    |
|   F32   |      I64      |      2^24      |  36.686 us |       0.97% |  38.769 us |       0.86% |   2.083 us |   5.68% |   SLOW   |    32.731 us |    34.859 us |   2.127 us |     6.50% |    SLOW    |
|   F32   |      I64      |      2^28      | 452.862 us |       0.19% | 539.153 us |       0.17% |  86.292 us |  19.05% |   SLOW   |   452.065 us |   535.694 us |  83.629 us |    18.50% |    SLOW    |
|   F64   |      I32      |      2^16      |   5.980 us |       4.76% |   5.963 us |       4.62% |  -0.016 us |  -0.27% |   SAME   |     4.094 us |     1.609 us |  -2.486 us |   -60.71% |    FAST    |
|   F64   |      I32      |      2^20      |  10.062 us |       2.89% |  10.054 us |       2.78% |  -0.009 us |  -0.09% |   SAME   |     6.141 us |     5.194 us |  -0.947 us |   -15.43% |    FAST    |
|   F64   |      I32      |      2^24      |  64.131 us |       1.48% |  71.512 us |       0.48% |   7.381 us |  11.51% |   SLOW   |    59.679 us |    68.255 us |   8.576 us |    14.37% |    SLOW    |
|   F64   |      I32      |      2^28      | 907.488 us |       0.24% |   1.073 ms |       0.05% | 165.704 us |  18.26% |   SLOW   |   904.912 us |     1.070 ms | 164.891 us |    18.22% |    SLOW    |
|   F64   |      I64      |      2^16      |   5.975 us |       4.58% |   5.921 us |       5.35% |  -0.054 us |  -0.90% |   SAME   |     4.094 us |     1.616 us |  -2.478 us |   -60.53% |    FAST    |
|   F64   |      I64      |      2^20      |  10.087 us |       2.88% |  10.049 us |       3.08% |  -0.039 us |  -0.39% |   SAME   |     6.141 us |     5.190 us |  -0.951 us |   -15.48% |    FAST    |
|   F64   |      I64      |      2^24      |  64.740 us |       1.54% |  71.541 us |       0.52% |   6.801 us |  10.50% |   SLOW   |    59.532 us |    68.144 us |   8.612 us |    14.47% |    SLOW    |
|   F64   |      I64      |      2^28      | 907.463 us |       0.25% |   1.073 ms |       0.04% | 165.692 us |  18.26% |   SLOW   |   904.402 us |     1.070 ms | 165.408 us |    18.29% |    SLOW    |
|  I128   |      I32      |      2^16      |   5.976 us |       4.80% |   5.969 us |       5.03% |  -0.007 us |  -0.11% |   SAME   |     4.094 us |     1.596 us |  -2.498 us |   -61.02% |    FAST    |
|  I128   |      I32      |      2^20      |  14.157 us |       1.98% |  14.101 us |       2.22% |  -0.056 us |  -0.40% |   SAME   |     8.188 us |     9.375 us |   1.187 us |    14.50% |    SLOW    |
|  I128   |      I32      |      2^24      | 119.885 us |       0.79% | 138.877 us |       0.45% |  18.992 us |  15.84% |   SLOW   |   116.405 us |   134.968 us |  18.563 us |    15.95% |    SLOW    |
|  I128   |      I32      |      2^28      |   1.812 ms |       0.11% |   2.142 ms |       0.02% | 329.563 us |  18.18% |   SLOW   |     1.806 ms |     2.140 ms | 333.777 us |    18.48% |    SLOW    |
|  I128   |      I64      |      2^16      |   5.983 us |       4.41% |   6.001 us |       5.10% |   0.017 us |   0.29% |   SAME   |     4.094 us |     1.578 us |  -2.516 us |   -61.46% |    FAST    |
|  I128   |      I64      |      2^20      |  14.136 us |       2.12% |  14.144 us |       2.09% |   0.008 us |   0.06% |   SAME   |     8.188 us |     9.380 us |   1.192 us |    14.56% |    SLOW    |
|  I128   |      I64      |      2^24      | 120.039 us |       0.76% | 138.907 us |       0.43% |  18.869 us |  15.72% |   SLOW   |   116.317 us |   134.990 us |  18.673 us |    16.05% |    SLOW    |
|  I128   |      I64      |      2^28      |   1.812 ms |       0.11% |   2.142 ms |       0.03% | 329.903 us |  18.21% |   SLOW   |     1.805 ms |     2.140 ms | 334.969 us |    18.56% |    SLOW    |

# nstream

## [0] NVIDIA B200

|  T{ct}  |  OffsetT{ct}  |  Elements{io}  |   Ref Time |   Ref Noise |   Cmp Time |   Cmp Noise |       Diff |   %Diff |  Status  |   B Ref Time |   B Cmp Time |     B Diff |   B %Diff |  B Status  |
|---------|---------------|----------------|------------|-------------|------------|-------------|------------|---------|----------|--------------|--------------|------------|-----------|------------|
|   I8    |      I32      |      2^16      |   5.989 us |       4.55% |   5.949 us |       5.01% |  -0.039 us |  -0.66% |   SAME   |     4.094 us |     1.634 us |  -2.460 us |   -60.09% |    FAST    |
|   I8    |      I32      |      2^20      |   6.342 us |       9.26% |   6.419 us |      10.11% |   0.077 us |   1.21% |   SAME   |     4.098 us |     3.117 us |  -0.981 us |   -23.94% |    FAST    |
|   I8    |      I32      |      2^24      |  19.445 us |       4.85% |  19.424 us |       4.68% |  -0.021 us |  -0.11% |   SAME   |    14.343 us |    13.192 us |  -1.151 us |    -8.02% |    FAST    |
|   I8    |      I32      |      2^28      | 212.863 us |       0.10% | 211.458 us |       0.43% |  -1.405 us |  -0.66% |   FAST   |   209.033 us |   205.241 us |  -3.792 us |    -1.81% |    FAST    |
|   I8    |      I64      |      2^16      |   6.009 us |       4.13% |   5.967 us |       4.81% |  -0.042 us |  -0.70% |   SAME   |     4.094 us |     1.645 us |  -2.449 us |   -59.82% |    FAST    |
|   I8    |      I64      |      2^20      |   6.334 us |       8.91% |   6.337 us |       9.49% |   0.003 us |   0.05% |   SAME   |     4.098 us |     3.126 us |  -0.972 us |   -23.71% |    FAST    |
|   I8    |      I64      |      2^24      |  19.865 us |       4.15% |  19.762 us |       4.17% |  -0.102 us |  -0.51% |   SAME   |    14.343 us |    13.214 us |  -1.129 us |    -7.87% |    FAST    |
|   I8    |      I64      |      2^28      | 215.113 us |       0.26% | 214.918 us |       0.08% |  -0.195 us |  -0.09% |   FAST   |   212.031 us |   208.688 us |  -3.342 us |    -1.58% |    FAST    |
|   I16   |      I32      |      2^16      |   6.009 us |       4.15% |   5.982 us |       4.60% |  -0.027 us |  -0.45% |   SAME   |     4.094 us |     1.614 us |  -2.480 us |   -60.58% |    FAST    |
|   I16   |      I32      |      2^20      |   7.932 us |       6.13% |   7.939 us |       5.75% |   0.006 us |   0.08% |   SAME   |     4.098 us |     3.114 us |  -0.984 us |   -24.01% |    FAST    |
|   I16   |      I32      |      2^24      |  26.983 us |       2.80% |  28.431 us |       1.60% |   1.448 us |   5.37% |   SLOW   |    22.510 us |    23.762 us |   1.252 us |     5.56% |    SLOW    |
|   I16   |      I32      |      2^28      | 322.852 us |       0.30% | 362.068 us |       0.20% |  39.215 us |  12.15% |   SLOW   |   319.790 us |   357.594 us |  37.804 us |    11.82% |    SLOW    |
|   I16   |      I64      |      2^16      |   5.987 us |       4.39% |   5.932 us |       4.98% |  -0.055 us |  -0.92% |   SAME   |     4.094 us |     1.619 us |  -2.475 us |   -60.44% |    FAST    |
|   I16   |      I64      |      2^20      |   7.946 us |       6.01% |   7.860 us |       6.95% |  -0.086 us |  -1.08% |   SAME   |     4.098 us |     3.140 us |  -0.958 us |   -23.38% |    FAST    |
|   I16   |      I64      |      2^24      |  27.225 us |       3.23% |  28.478 us |       1.04% |   1.253 us |   4.60% |   SLOW   |    22.522 us |    23.750 us |   1.228 us |     5.45% |    SLOW    |
|   I16   |      I64      |      2^28      | 327.108 us |       0.26% | 361.990 us |       0.22% |  34.882 us |  10.66% |   SLOW   |   324.185 us |   357.615 us |  33.430 us |    10.31% |    SLOW    |
|   F32   |      I32      |      2^16      |   5.977 us |       4.80% |   5.978 us |       4.67% |   0.000 us |   0.01% |   SAME   |     4.094 us |     1.632 us |  -2.462 us |   -60.13% |    FAST    |
|   F32   |      I32      |      2^20      |   8.471 us |       7.81% |   8.566 us |       8.58% |   0.095 us |   1.12% |   SAME   |     4.514 us |     3.815 us |  -0.699 us |   -15.48% |    FAST    |
|   F32   |      I32      |      2^24      |  46.447 us |       1.83% |  49.517 us |       1.50% |   3.070 us |   6.61% |   SLOW   |    41.011 us |    46.008 us |   4.997 us |    12.19% |    SLOW    |
|   F32   |      I32      |      2^28      | 598.549 us |       0.15% | 717.324 us |       0.12% | 118.774 us |  19.84% |   SLOW   |   592.635 us |   713.743 us | 121.108 us |    20.44% |    SLOW    |
|   F32   |      I64      |      2^16      |   5.988 us |       4.55% |   5.959 us |       4.85% |  -0.029 us |  -0.49% |   SAME   |     4.094 us |     1.614 us |  -2.480 us |   -60.57% |    FAST    |
|   F32   |      I64      |      2^20      |   8.514 us |       8.41% |   8.492 us |       8.07% |  -0.022 us |  -0.26% |   SAME   |     4.715 us |     3.826 us |  -0.889 us |   -18.86% |    FAST    |
|   F32   |      I64      |      2^24      |  46.541 us |       1.71% |  49.648 us |       1.55% |   3.107 us |   6.68% |   SLOW   |    41.069 us |    45.971 us |   4.902 us |    11.94% |    SLOW    |
|   F32   |      I64      |      2^28      | 599.451 us |       0.14% | 717.347 us |       0.13% | 117.896 us |  19.67% |   SLOW   |   593.715 us |   713.745 us | 120.029 us |    20.22% |    SLOW    |
|   F64   |      I32      |      2^16      |   5.958 us |       4.65% |   5.965 us |       4.72% |   0.007 us |   0.11% |   SAME   |     4.094 us |     1.625 us |  -2.469 us |   -60.31% |    FAST    |
|   F64   |      I32      |      2^20      |  11.491 us |       7.71% |  11.504 us |       7.48% |   0.013 us |   0.11% |   SAME   |     6.141 us |     5.222 us |  -0.919 us |   -14.96% |    FAST    |
|   F64   |      I32      |      2^24      |  84.027 us |       0.80% |  84.034 us |       0.92% |   0.007 us |   0.01% |   SAME   |    77.789 us |    74.972 us |  -2.817 us |    -3.62% |    FAST    |
|   F64   |      I32      |      2^28      |   1.184 ms |       0.05% |   1.184 ms |       0.05% |   0.029 us |   0.00% |   SAME   |     1.176 ms |     1.173 ms |  -2.920 us |    -0.25% |    SAME    |
|   F64   |      I64      |      2^16      |   5.966 us |       4.90% |   5.963 us |       4.86% |  -0.003 us |  -0.06% |   SAME   |     4.094 us |     1.637 us |  -2.457 us |   -60.02% |    FAST    |
|   F64   |      I64      |      2^20      |  11.513 us |       7.79% |  11.494 us |       7.78% |  -0.019 us |  -0.17% |   SAME   |     6.141 us |     5.216 us |  -0.925 us |   -15.06% |    FAST    |
|   F64   |      I64      |      2^24      |  84.030 us |       0.82% |  84.060 us |       0.76% |   0.030 us |   0.04% |   SAME   |    77.804 us |    74.994 us |  -2.810 us |    -3.61% |    FAST    |
|   F64   |      I64      |      2^28      |   1.184 ms |       0.05% |   1.184 ms |       0.06% |   0.075 us |   0.01% |   SAME   |     1.176 ms |     1.173 ms |  -2.845 us |    -0.24% |    SAME    |
|  I128   |      I32      |      2^16      |   6.066 us |       5.68% |   6.017 us |       5.84% |  -0.049 us |  -0.81% |   SAME   |     4.094 us |     1.592 us |  -2.502 us |   -61.11% |    FAST    |
|  I128   |      I32      |      2^20      |  16.358 us |       2.88% |  16.406 us |       2.93% |   0.048 us |   0.29% |   SAME   |     8.188 us |     9.399 us |   1.211 us |    14.79% |    SLOW    |
|  I128   |      I32      |      2^24      | 156.487 us |       0.62% | 156.500 us |       0.61% |   0.014 us |   0.01% |   SAME   |   150.507 us |   147.953 us |  -2.554 us |    -1.70% |    FAST    |
|  I128   |      I32      |      2^28      |   2.356 ms |       0.04% |   2.356 ms |       0.04% |  -0.046 us |  -0.00% |   SAME   |     2.349 ms |     2.347 ms |  -1.491 us |    -0.06% |    SAME    |
|  I128   |      I64      |      2^16      |   6.084 us |       6.74% |   6.073 us |       6.19% |  -0.010 us |  -0.17% |   SAME   |     4.094 us |     1.607 us |  -2.487 us |   -60.74% |    FAST    |
|  I128   |      I64      |      2^20      |  16.395 us |       2.82% |  16.393 us |       2.69% |  -0.002 us |  -0.01% |   SAME   |     8.188 us |     9.410 us |   1.222 us |    14.92% |    SLOW    |
|  I128   |      I64      |      2^24      | 156.555 us |       0.64% | 156.506 us |       0.62% |  -0.049 us |  -0.03% |   SAME   |   150.525 us |   147.965 us |  -2.560 us |    -1.70% |    FAST    |
|  I128   |      I64      |      2^28      |   2.356 ms |       0.04% |   2.356 ms |       0.04% |  -0.012 us |  -0.00% |   SAME   |     2.349 ms |     2.347 ms |  -1.875 us |    -0.08% |    SAME    |

# Summary

- Total Matches: 160
  - Pass    (diff <= min_noise): 130
  - Unknown (infinite noise):    0
  - Failure (diff > min_noise):  190

The table becomes a bit unwieldy. We could consider dropping the Diff and B Diff columns to improve the situation. Alternatively, we could emit two rows per benchmark.

Fixes: NVIDIA#247

alliepiper · 2025-08-20T21:57:58Z

scripts/nvbench_compare.py

+                if has_batch_data:
+                    if (
+                        abs(frac_diff_batch) <= 0.01
+                    ):  # TODO(bgruber): what value to use here?


I have no idea, let's get some input internally on that.

Maybe just pick a sensible default and let the user override with command-line opts?

We should use min_noise estimated from abs(frac_diff_batch) <= min_noise if available, like we do for cold measurements, and use yellow tint if min_noise is not available.

Perhaps in case when has_batch_data is True, min_noise should always be available.

alliepiper · 2025-08-20T22:01:46Z

I like the idea of splitting them to a new line, I think it'd be cleaner.

Or making them into separate tables? That way you could still quickly scan a column to check for outliers. That'd be harder if the timings were alternating cold/batch.

bernhardmgruber added 2 commits August 14, 2025 19:18

Also compare batch measurements in nvbench_compare.py

58d1f8c

Fixes: NVIDIA#247

Fix

d140027

alliepiper reviewed Aug 20, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Also compare batch measurements in nvbench_compare.py #263

Also compare batch measurements in nvbench_compare.py #263

Uh oh!

bernhardmgruber commented Aug 14, 2025 •

edited

Loading

Uh oh!

alliepiper Aug 20, 2025

Uh oh!

alliepiper Aug 20, 2025

Uh oh!

oleksandr-pavlyk Oct 2, 2025

Uh oh!

alliepiper commented Aug 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Also compare batch measurements in nvbench_compare.py #263

Are you sure you want to change the base?

Also compare batch measurements in nvbench_compare.py #263

Uh oh!

Conversation

bernhardmgruber commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alliepiper Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

alliepiper Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

oleksandr-pavlyk Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

alliepiper commented Aug 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bernhardmgruber commented Aug 14, 2025 •

edited

Loading