group_reporting stops the "All clients" aggregation report from being output in client mode

[mrkbutty]

Please acknowledge the following before creating a ticket

• [y ] I have read the GitHub issues section of REPORTING-BUGS.

Description of the bug:
The "All clients" aggregation section is missing from reports when specifying more than one job.

Environment: RHEL 9.4 with one master server and 2x remote clients

fio version: fio 3.38

Reproduction steps
Using command: fio --client=myhost1 /tmp/test.fio --client=myhost2 /tmp/test.fio
Using job file:
[global]
numjobs=2
time_based=1
ramp_time=5
runtime=10
direct=1
overwrite=1
ioengine=posixaio
group_reporting=1
unified_rw_reporting=both # or none, mixed or both
disable_clat=1
disable_slat=1
slat_percentiles=0
clat_percentiles=0
disk_util=0
[8k Random Read Miss1]
bs=8k
rw=randrw
iodepth=32
filename=/dev/nvme1n1

I get a report including myhost1, myhost2 and the "All clients" report.

However, if I append this section to the above job file (to add another disk):
[8k Random Read Miss2]
bs=8k
rw=randrw
iodepth=32
filename=/dev/nvme1n2

I still get the individual server reports myhost1 & myhost2 but I no longer get the "All clients" report.

[vincentkfu]

I can't reproduce your issue. There is a (short) "All clients" section at the very bottom. In 3.39 is a fix that allows latency reporting in the "All clients" section. Is that what you are missing?

[global]
ioengine=null
filesize=1T
time_based=1
runtime=3s

[test1]

[test2]

$ ./fio/fio-3.38 --client=localhost 1881.fio --client=localhost 1881.fio
hostname=meeki, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-3.38, flags=1
hostname=meeki, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-3.38, flags=1
<meeki> test1: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
<meeki> test1: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
<meeki> test2: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
<meeki> test2: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
<meeki> Starting 2 processes
<meeki> Starting 2 processes
<meeki> (f=0): [R(2)][85.7%][r=214G][r=53.5M IOPS][eta 00m:01s]
<meeki> 
test1: (groupid=0, jobs=1): err= 0: pid=37670: Wed Mar 12 12:23:40 2025
  read: IOPS=13.2M, BW=50.2GiB/s (53.9GB/s)(151GiB/3000msec)
    clat (nsec): min=0, max=15108, avg=11.58, stdev=11.93
     lat (nsec): min=10, max=23093, avg=19.16, stdev=19.19
    clat percentiles (nsec):
     |  1.00th=[    9],  5.00th=[   10], 10.00th=[   10], 20.00th=[   10],
     | 30.00th=[   10], 40.00th=[   10], 50.00th=[   10], 60.00th=[   10],
     | 70.00th=[   10], 80.00th=[   10], 90.00th=[   20], 95.00th=[   20],
     | 99.00th=[   20], 99.50th=[   20], 99.90th=[   30], 99.95th=[   30],
     | 99.99th=[   30]
   bw (  MiB/s): min=48608, max=52045, per=49.83%, avg=51405.85, stdev=1375.35, samples=6
   iops        : min=12443888, max=13323694, avg=13159915.33, stdev=352047.86, samples=6
  lat (nsec)   : 2=0.51%, 10=2.32%, 20=83.42%, 50=13.75%, 100=0.01%
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=99.63%, sys=0.00%, ctx=39, majf=0, minf=5
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=39479746,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
test2: (groupid=0, jobs=1): err= 0: pid=37672: Wed Mar 12 12:23:40 2025
  read: IOPS=13.2M, BW=50.5GiB/s (54.3GB/s)(152GiB/3000msec)
    clat (nsec): min=0, max=20118, avg=11.32, stdev=10.15
     lat (nsec): min=10, max=20128, avg=18.87, stdev=15.25
    clat percentiles (nsec):
     |  1.00th=[    9],  5.00th=[   10], 10.00th=[   10], 20.00th=[   10],
     | 30.00th=[   10], 40.00th=[   10], 50.00th=[   10], 60.00th=[   10],
     | 70.00th=[   10], 80.00th=[   10], 90.00th=[   20], 95.00th=[   20],
     | 99.00th=[   20], 99.50th=[   20], 99.90th=[   21], 99.95th=[   30],
     | 99.99th=[   30]
   bw (  MiB/s): min=48776, max=52484, per=50.17%, avg=51755.24, stdev=1462.71, samples=6
   iops        : min=12486664, max=13435998, avg=13249342.67, stdev=374453.29, samples=6
  lat (nsec)   : 2=0.76%, 10=2.55%, 20=84.90%, 50=11.80%, 100=0.01%
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=99.70%, sys=0.00%, ctx=16, majf=0, minf=5
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=39748028,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=101GiB/s (108GB/s), 50.2GiB/s-50.5GiB/s (53.9GB/s-54.3GB/s), io=302GiB (325GB), run=3000-3000msec
test1: (groupid=0, jobs=1): err= 0: pid=37669: Wed Mar 12 12:23:40 2025
  read: IOPS=13.2M, BW=50.4GiB/s (54.1GB/s)(151GiB/3000msec)
    clat (nsec): min=0, max=12503, avg=11.57, stdev=10.00
     lat (nsec): min=10, max=16721, avg=19.11, stdev=15.72
    clat percentiles (nsec):
     |  1.00th=[    9],  5.00th=[   10], 10.00th=[   10], 20.00th=[   10],
     | 30.00th=[   10], 40.00th=[   10], 50.00th=[   10], 60.00th=[   10],
     | 70.00th=[   10], 80.00th=[   10], 90.00th=[   20], 95.00th=[   20],
     | 99.00th=[   20], 99.50th=[   20], 99.90th=[   21], 99.95th=[   30],
     | 99.99th=[   30]
   bw (  MiB/s): min=48879, max=52289, per=49.94%, avg=51598.37, stdev=1335.17, samples=6
   iops        : min=12513156, max=13386120, avg=13209205.33, stdev=341754.54, samples=6
  lat (nsec)   : 2=0.50%, 10=2.32%, 20=83.53%, 50=13.65%, 100=0.01%
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=99.67%, sys=0.00%, ctx=24, majf=0, minf=6
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=39627616,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
test2: (groupid=0, jobs=1): err= 0: pid=37671: Wed Mar 12 12:23:40 2025
  read: IOPS=13.2M, BW=50.5GiB/s (54.2GB/s)(152GiB/3000msec)
    clat (nsec): min=0, max=13435, avg=11.33, stdev= 9.31
     lat (nsec): min=10, max=16891, avg=18.88, stdev=14.76
    clat percentiles (nsec):
     |  1.00th=[    9],  5.00th=[   10], 10.00th=[   10], 20.00th=[   10],
     | 30.00th=[   10], 40.00th=[   10], 50.00th=[   10], 60.00th=[   10],
     | 70.00th=[   10], 80.00th=[   10], 90.00th=[   20], 95.00th=[   20],
     | 99.00th=[   20], 99.50th=[   20], 99.90th=[   21], 99.95th=[   30],
     | 99.99th=[   30]
   bw (  MiB/s): min=48747, max=52464, per=50.06%, avg=51722.82, stdev=1461.26, samples=6
   iops        : min=12479432, max=13430960, avg=13241042.33, stdev=374083.07, samples=6
  lat (nsec)   : 2=0.74%, 10=2.57%, 20=84.83%, 50=11.86%, 100=0.01%
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=99.70%, sys=0.00%, ctx=29, majf=0, minf=6
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=39723127,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=101GiB/s (108GB/s), 50.4GiB/s-50.5GiB/s (54.1GB/s-54.2GB/s), io=303GiB (325GB), run=3000-3000msec
All clients: (groupid=0, jobs=4): err= 0: pid=0: Wed Mar 12 12:23:40 2025
  read: IOPS=52.9M, BW=202Gi (217G)(605GiB/3000msec)
    clat (nsec): min=0, max=20118, avg=11.45, stdev=10.39
     lat (nsec): min=10, max=23093, avg=19.00, stdev=16.32
   bw (  MiB/s): min=195011, max=209284, per=50.00%, avg=206482.28, stdev=1314.56, samples=24
   iops        : min=49923140, max=53576772, avg=52859505.67, stdev=336506.11, samples=24
  lat (nsec)   : 2=0.63%, 10=2.44%, 20=84.17%, 50=12.76%, 100=0.01%
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=99.67%, sys=0.00%, ctx=108, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=158578517,0,0,0 short=0,0,0,0 dropped=0,0,0,0

[vincentkfu]

Oh I tried your job and now see the issue.

vincent@meeki:~/fio-dev/1881$ ./fio/fio-3.38 --client=localhost 1881-2.fio --client=localhost 1881-2.fio
hostname=meeki, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-3.38, flags=1
hostname=meeki, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-3.38, flags=1
<meeki> 8k Random Read Miss1: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=posixaio, iodepth=32
<meeki> 8k Random Read Miss1: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=posixaio, iodepth=32
<meeki> ...
<meeki> ...
<meeki> 8k Random Read Miss2: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=posixaio, iodepth=32
<meeki> 8k Random Read Miss2: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=posixaio, iodepth=32
<meeki> ...
<meeki> ...
<meeki> Starting 4 processes
<meeki> Starting 4 processes
<meeki> (f=4): [r(2),_(2)][100.0%][r=3016M][r=377k IOPS][eta 00m:00s]
8k Random Read Miss1: (groupid=0, jobs=4): err= 0: pid=38528: Wed Mar 12 12:40:15 2025
  read: IOPS=181k, BW=1410MiB/s (1479MB/s)(13.8GiB/10001msec)
     lat (usec): min=35, max=49793, avg=707.14, stdev=841.69
   bw (  MiB/s): min=  886, max= 1503, per=100.00%, avg=1410.52, stdev=48.67, samples=80
   iops        : min=113440, max=192442, avg=180546.30, stdev=6230.21, samples=80
  mixed: IOPS=0, BW=0 (0)(0B/10001msec)
     lat (usec): min=35, max=49793, avg=707.14, stdev=841.69
   bw (  MiB/s): min=  886, max= 1503, per=100.00%, avg=1410.52, stdev=48.67, samples=80
   iops        : min=113440, max=192442, avg=180546.30, stdev=6230.21, samples=80
  cpu          : usr=2.71%, sys=1.90%, ctx=454123, majf=0, minf=145
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=7.0%, 16=68.0%, 32=25.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.7%, 8=1.4%, 16=1.4%, 32=1.5%, 64=0.0%, >=64=0.0%
     issued rwts: total=1805339,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1410MiB/s (1479MB/s), 1410MiB/s-1410MiB/s (1479MB/s-1479MB/s), io=13.8GiB (14.8GB), run=10001-10001msec
  MIXED: bw=1410MiB/s (1479MB/s), 1410MiB/s-1410MiB/s (1479MB/s-1479MB/s), io=13.8GiB (14.8GB), run=10001-10001msec
<meeki> 
8k Random Read Miss1: (groupid=0, jobs=4): err= 0: pid=38527: Wed Mar 12 12:40:15 2025
  read: IOPS=180k, BW=1410MiB/s (1479MB/s)(13.8GiB/10001msec)
     lat (usec): min=149, max=48923, avg=707.09, stdev=840.55
   bw (  MiB/s): min=  888, max= 1503, per=100.00%, avg=1410.41, stdev=48.55, samples=80
   iops        : min=113728, max=192448, avg=180532.80, stdev=6214.63, samples=80
  mixed: IOPS=0, BW=0 (0)(0B/10001msec)
     lat (usec): min=149, max=48923, avg=707.09, stdev=840.55
   bw (  MiB/s): min=  888, max= 1503, per=100.00%, avg=1410.41, stdev=48.55, samples=80
   iops        : min=113728, max=192448, avg=180532.80, stdev=6214.63, samples=80
  cpu          : usr=2.81%, sys=1.82%, ctx=453490, majf=0, minf=146
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=8.1%, 16=66.8%, 32=25.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.4%, 8=1.1%, 16=0.6%, 32=1.9%, 64=0.0%, >=64=0.0%
     issued rwts: total=1805165,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1410MiB/s (1479MB/s), 1410MiB/s-1410MiB/s (1479MB/s-1479MB/s), io=13.8GiB (14.8GB), run=10001-10001msec
  MIXED: bw=1410MiB/s (1479MB/s), 1410MiB/s-1410MiB/s (1479MB/s-1479MB/s), io=13.8GiB (14.8GB), run=10001-10001msec

Can you do some troubleshooting to figure out the minimum combination of options that trigger the problem?

[mrkbutty]

I'll try and come back to you...

[mrkbutty]

I used the following job file:

# cat test2.fio
[global]
ioengine=null
filesize=1M
time_based=1
runtime=3s
group_reporting=1
[test1]
filename=/tmp/fiotest1
[test2]
filename=/tmp/fiotest2

Soon as I added "group_reporting" the issue occurs:

# /scripts/fio/fio-3.38/fio --client hlg201 test2.fio --client hlg202 test2.fio
hostname=hlg202, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-3.38, flags=1
hostname=hlg201, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-3.38, flags=1
<hlg202> test1: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
<hlg201> test1: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
<hlg202> test2: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
<hlg201> test2: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
<hlg201> Starting 2 processes
<hlg202> Starting 2 processes
<hlg201> f=4): [R(2)][-.-%][r=91.0G][r=22.7M IOPS][eta 00m:00s]
test1: (groupid=0, jobs=2): err= 0: pid=1827075: Wed Mar 12 23:09:07 2025
  read: IOPS=9862k, BW=37.6GiB/s (40.4GB/s)(113GiB/3000msec)
    clat (nsec): min=9, max=77023, avg=26.45, stdev=71.32
     lat (nsec): min=29, max=92472, avg=47.83, stdev=113.03
    clat percentiles (nsec):
     |  1.00th=[   10],  5.00th=[   20], 10.00th=[   20], 20.00th=[   20],
     | 30.00th=[   20], 40.00th=[   20], 50.00th=[   30], 60.00th=[   30],
     | 70.00th=[   30], 80.00th=[   30], 90.00th=[   31], 95.00th=[   31],
     | 99.00th=[   50], 99.50th=[   60], 99.90th=[   70], 99.95th=[  100],
     | 99.99th=[  470]
   bw (  MiB/s): min=33874, max=48933, per=100.00%, avg=38523.82, stdev=3182.32, samples=12
   iops        : min=8671854, max=12527042, avg=9862110.33, stdev=814670.15, samples=12
  lat (nsec)   : 10=0.16%, 20=1.78%, 50=96.38%, 100=1.63%, 250=0.04%
  lat (nsec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.01%
  cpu          : usr=98.15%, sys=0.75%, ctx=162, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=29586332,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=37.6GiB/s (40.4GB/s), 37.6GiB/s-37.6GiB/s (40.4GB/s-40.4GB/s), io=113GiB (121GB), run=3000-3000msec
<hlg202>
test1: (groupid=0, jobs=2): err= 0: pid=58704: Wed Mar 12 23:09:07 2025
  read: IOPS=9983k, BW=38.1GiB/s (40.9GB/s)(114GiB/3001msec)
    clat (nsec): min=9, max=49682, avg=26.14, stdev=31.45
     lat (nsec): min=29, max=70782, avg=47.08, stdev=50.11
    clat percentiles (nsec):
     |  1.00th=[   10],  5.00th=[   20], 10.00th=[   20], 20.00th=[   20],
     | 30.00th=[   20], 40.00th=[   20], 50.00th=[   30], 60.00th=[   30],
     | 70.00th=[   30], 80.00th=[   30], 90.00th=[   31], 95.00th=[   31],
     | 99.00th=[   50], 99.50th=[   50], 99.90th=[   61], 99.95th=[   71],
     | 99.99th=[  191]
   bw (  MiB/s): min=34695, max=49828, per=100.00%, avg=39010.91, stdev=3150.03, samples=12
   iops        : min=8882246, max=12756090, avg=9986815.33, stdev=806397.93, samples=12
  lat (nsec)   : 10=0.17%, 20=1.92%, 50=96.78%, 100=1.10%, 250=0.02%
  lat (nsec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=99.42%, sys=0.00%, ctx=7, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=29960447,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=38.1GiB/s (40.9GB/s), 38.1GiB/s-38.1GiB/s (40.9GB/s-40.9GB/s), io=114GiB (123GB), run=3001-3001msec

[mrkbutty]

Further update, I found that adding "new_group" solved my issue ...

[global]
ioengine=null
time_based=1
runtime=3
size=1M
group_reporting
new_group

[test1]

[test2]

However, I do end up with 24 groups as I am defining one job per device, i.e. I have a filename=/dev/xxxx for all 24 devices on each of my 4x fio server hosts. It does group 0 for the first device on each host, then group1 for the second device and so on, strange grouping and not exactly what I want but it works.

It does work though!