Setting loops > 1 and unlink_each_loop=1 causes I/O to fail

[jan--f]

This job file produces the expected output. Results look plausible (i.e. io volume fits the job file, bandwidth seems appropriate)

[global]
  size=1g
  loops=3
  unlink_each_loop=1

[worker0_1m_readwrite]
  rw=readwrite
  blocksize=1m

However fio does print an error message: worker0_1m_readwrite: No I/O performed by psync, perhaps try --debug=io option for details?

Adding unlink_each_loop=1 to the job specification causes fio to not do any I/O and two error messages are reported:

worker0_1m_readwrite: No I/O performed by psync, perhaps try --debug=io option for details?
fio: pid=7096, err=5/file:backend.c:490, func=full resid, error=Input/output error

Furthermore the io stats reported in the results are bogus, as in io volume is not what was specified.

[sitsofe]

@jan--f Which version of fio?

[sitsofe]

OK it happens with master fio. The following is enough to reproduce the problem:

$ fio --size=1m --loops=2 --unlink_each_loop=1 --name=noio --rw=read
[...]
noio: Laying out IO file (1 file / 1MiB)
noio: No I/O performed by psync, perhaps try --debug=io option for details?
fio: pid=31525, err=2/file:filesetup.c:708, func=open(noio.0.0), error=No such file or directory

The following is a varient:

$ fio --size=1m --loops=2 --unlink_each_loop=1 --name=noio --rw=readwrite 
[...]
noio: Laying out IO file (1 file / 1MiB)
noio: No I/O performed by psync, perhaps try --debug=io option for details?
fio: pid=31519, err=5/file:backend.c:490, func=full resid, error=Input/output error

@jan--f is the expectation that fio will re-layout the file on each loop pass after unlinking?

[jan--f]

@sitsofe my test was done with fio-2.20.
I'm actually not sure what I should expect other then fio making io. I came across this by accident...so maybe its a user error as well. In that case a doc update might be good. I'm not clear what unlink_each_loop is supposed to do.

[MohanrajBabu]

@sitsofe @axboe
This issue still persist.

Document says
unlink_each_loop=bool
Unlink job files after each iteration or loop. Default: false.

Linux distro and kernel version
Ubuntu 18.04
4.15.0-29-generic

fio version
fio-3.12-18-g92a1

fio issues IO for only one loop.

fio --name=job1 --size=10m --loops=2 --unlink_each_loop=1

job1: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12-18-g92a1
Starting 1 process
fio: pid=37, err=2/file:filesetup.c:701, func=open(job1.0.0), error=No such file or directory

job1: (groupid=0, jobs=1): err= 2 (file:filesetup.c:701, func=open(job1.0.0), error=No such file or directory): pid=37: Sun Nov 25 16:46:24 2018
  read: IOPS=7975, BW=31.2MiB/s (32.7MB/s)(10.0MiB/321msec)
    clat (nsec): min=1337, max=86697k, avg=124063.73, stdev=2327184.99
     lat (nsec): min=1378, max=86697k, avg=124118.14, stdev=2327186.03
    clat percentiles (nsec):
     |  1.00th=[    1384],  5.00th=[    1448], 10.00th=[    1480],
     | 20.00th=[    1512], 30.00th=[    1560], 40.00th=[    1576],
     | 50.00th=[    1608], 60.00th=[    1640], 70.00th=[    1688],
     | 80.00th=[    2040], 90.00th=[    2576], 95.00th=[    3088],
     | 99.00th=[ 3129344], 99.50th=[ 4079616], 99.90th=[38535168],
     | 99.95th=[65273856], 99.99th=[86507520]
  lat (usec)   : 2=79.69%, 4=16.95%, 10=1.37%, 20=0.23%, 50=0.04%
  lat (usec)   : 100=0.08%, 250=0.04%, 500=0.20%, 750=0.20%
  lat (msec)   : 4=0.66%, 10=0.35%, 20=0.08%, 50=0.04%, 100=0.08%
  cpu          : usr=1.25%, sys=1.25%, ctx=42, majf=0, minf=18
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=31.2MiB/s (32.7MB/s), 31.2MiB/s-31.2MiB/s (32.7MB/s-32.7MB/s), io=10.0MiB (10.5MB), run=321-321msec

Disk stats (read/write):
  sda: ios=10/0, merge=0/0, ticks=180/0, in_queue=216, util=56.90%

[sitsofe]

@MohanrajBabu sure, I can well believe this is still broken :-) However, we need to work out what the expected behaviour should be before a fix can be put in:

[...] is the expectation that fio will re-layout the file on each loop pass after unlinking?

[rapiz1]

I actually expect the re-layout. Perhaps this could use an option.

[wangyihanlarry]

the issue still persist, are there any EST fix date for this problem?

fio: pid=1784327, err=2/file:backend.c:557, func=unlink_all_files, error=No such file or directory
fio: pid=1784314, err=2/file:backend.c:557, func=unlink_all_files, error=No such file or directory
fio: pid=1784334, err=2/file:backend.c:557, func=unlink_all_files, error=No such file or directory
fio: pid=1784308, err=2/file:backend.c:557, func=unlink_all_files, error=No such file or directory
fio: pid=1784328, err=2/file:backend.c:557, func=unlink_all_files, error=No such file or directory
fio: pid=1784331, err=2/file:backend.c:557, func=unlink_all_files, error=No such file or directory
fio: pid=1784311, err=2/file:backend.c:557, func=unlink_all_files, error=No such file or directory
fio: pid=1784310, err=2/file:backend.c:557, func=unlink_all_files, error=No such file or directory
fio: pid=1784325, err=2/file:backend.c:557, func=unlink_all_files, error=No such file or directory
fio: pid=1784309, err=2/file:backend.c:557, func=unlink_all_files, error=No such file or directory
fio: pid=1784329, err=2/file:backend.c:557, func=unlink_all_files, error=No such file or directory
fio: pid=1784305, err=2/file:backend.c:557, func=unlink_all_files, error=No such file or directory
Jobs: 8 (f=9): [X(1),F(1),M(1),X(4),M(2),X(1),_(10),X(1),w(1),X(3),w(1),X(1),F(1),w(1),X(1)][2.7%][r=100MiB/s,w=3