nrfiles=2 in rw=randwrite writes only in single file

[Tulsishah]

[global]
ioengine=libaio
direct=1
fadvise_hint=0
verify=0
rw=read
bs=1M
iodepth=64
invalidate=1
ramp_time=10s
runtime=60s
startdelay=2m
time_based=1
nrfiles=1
thread=1
fsync=1
openfiles=1
group_reporting=1
allrandrepeat=1
filename_format=$jobname.$jobnum.$filenum

[3mb_rand_write]
stonewall
startdelay=310
directory=gcs/3mb
filesize=3M
rw=randwrite
nrfiles=2
fsync=1
numjobs=1

writes 3mb on single file(not 2 files) in case of random writes.

May I know the reason for the same?

[vincentkfu]

This appears to be a bug in fio. Probably this is related to the other issue you filed as well.

vincent@fio-dev:~/fio-dev/1676$ ./fio-canonical/fio 1676.fio --debug=io &> 1676.log
vincent@fio-dev:~/fio-dev/1676$ head 1676.log -n 100
fio: set debug option io
io       4466  load ioengine libaio
3mb_rand_write: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
fio-3.36-19-gbdf99
Starting 1 thread
3mb_rand_write: Laying out IO files (2 files / total 6MiB)
io       4466  declare unneeded cache 3mb_rand_write.0.0: 0/3145728
io       4466  declare unneeded cache 3mb_rand_write.0.1: 0/3145728
io       4468  declare unneeded cache 3mb_rand_write.0.0: 0/3145728
io       4468  fill: io_u 0x5580635816c0: off=0x200000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: off=0x200000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: ret=0
io       4468  queue: io_u 0x5580635816c0: off=0x200000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  calling ->commit(), depth 1
io       4468  io_u 0x558063581500, setting file failed
io       4468  get_io_u failed
io       4468  io_u_queued_complete: min=1
io       4468  getevents: 1
io       4468  complete: io_u 0x5580635816c0: off=0x200000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  fill: io_u 0x5580635816c0: off=0x200000,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: off=0x200000,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: ret=0
io       4468  queue: io_u 0x5580635816c0: off=0x200000,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  complete: io_u 0x5580635816c0: off=0x200000,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  io_u 0x5580635816c0, setting file failed
io       4468  get_io_u failed
io       4468  fill: io_u 0x5580635816c0: off=0x0,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: off=0x0,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: ret=0
io       4468  queue: io_u 0x5580635816c0: off=0x0,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  calling ->commit(), depth 1
io       4468  io_u 0x558063581500, setting file failed
io       4468  get_io_u failed
io       4468  io_u_queued_complete: min=1
io       4468  getevents: 1
io       4468  complete: io_u 0x5580635816c0: off=0x0,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  fill: io_u 0x5580635816c0: off=0x0,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: off=0x0,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: ret=0
io       4468  queue: io_u 0x5580635816c0: off=0x0,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  complete: io_u 0x5580635816c0: off=0x0,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  io_u 0x5580635816c0, setting file failed
io       4468  get_io_u failed
io       4468  fill: io_u 0x5580635816c0: off=0x100000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: off=0x100000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: ret=0
io       4468  queue: io_u 0x5580635816c0: off=0x100000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  calling ->commit(), depth 1
io       4468  io_u 0x558063581500, setting file failed
io       4468  get_io_u failed
io       4468  io_u_queued_complete: min=1
io       4468  getevents: 1
io       4468  complete: io_u 0x5580635816c0: off=0x100000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  fill: io_u 0x5580635816c0: off=0x100000,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: off=0x100000,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: ret=0
io       4468  queue: io_u 0x5580635816c0: off=0x100000,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  complete: io_u 0x5580635816c0: off=0x100000,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  io_u 0x5580635816c0, setting file failed
io       4468  get_io_u failed
io       4468  fill: io_u 0x5580635816c0: off=0x100000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: off=0x100000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: ret=0
io       4468  queue: io_u 0x5580635816c0: off=0x100000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  calling ->commit(), depth 1
io       4468  io_u 0x558063581500, setting file failed
io       4468  get_io_u failed
io       4468  io_u_queued_complete: min=1
io       4468  getevents: 1
io       4468  complete: io_u 0x5580635816c0: off=0x100000,len=0x100000,ddir=1,file=3mb_rand_write.0.0

[Tulsishah]

When can we expect a fix for this issue?

[vincentkfu]

There is no timeline.

To do a good job of fixing this the steps would be:

1. Establish a set of requirements for fio's behavior involving multiple files and the options affecting this.
2. Write a test script to make sure fio meets these requirements.
3. Write a patch to fix this issue that causes no regressions.

One thing that might help is if someone tried older fio versions to see if this is a longstanding or a recent problem. If this is a regression, then it might help to identify the patch introducing this issue.
It would also help to have a simpler job file that can be used to produce this issue (https://stackoverflow.com/help/minimal-reproducible-example).

[Tulsishah]

I tested fio versions 3.34 and 3.35, and both exhibited the reported issues.(#1676, #1677)

[Tulsishah]

This issue also occurs in rw=write mode. However, using timebased=0 it works as expected.