read job with nrfiles>9 causes unlink

[llamafilm]

I am trying to write 10 files in a job, and then read them in another job. So I have set filename_format to match and this works with fewer than 10 files per job. But when nrfiles=10, the read job unlinks one file. To see this more clearly, I have set allow_file_create=0. In the below output, notice the output of ls at the end, it's missing one file. I can't think of any reason for this so I guess it's a bug. This is fio version 3.5.

root@freenas ~ # fio --name=write_file --rw=write --bs=1M --size=2G --end_fsync=1 --filename_format=/octa/file.\$jobnum.\$filenum --nrfiles=10
write_file: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.5
Starting 1 process
write_file: Laying out IO files (10 files / total 2047MiB)
Jobs: 1 (f=1): [F(1)][-.-%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:00s]
write_file: (groupid=0, jobs=1): err= 0: pid=75333: Thu Oct  3 12:54:13 2019
  write: IOPS=868, BW=869MiB/s (911MB/s)(2040MiB/2348msec)
    clat (usec): min=200, max=1564, avg=227.49, stdev=82.32
     lat (usec): min=210, max=1574, avg=240.05, stdev=94.30
    clat percentiles (usec):
     |  1.00th=[  204],  5.00th=[  206], 10.00th=[  206], 20.00th=[  208],
     | 30.00th=[  208], 40.00th=[  210], 50.00th=[  219], 60.00th=[  219],
     | 70.00th=[  221], 80.00th=[  221], 90.00th=[  225], 95.00th=[  237],
     | 99.00th=[  668], 99.50th=[  873], 99.90th=[  947], 99.95th=[  947],
     | 99.99th=[ 1565]
  lat (usec)   : 250=97.21%, 500=0.49%, 750=1.42%, 1000=0.83%
  lat (msec)   : 2=0.05%
  cpu          : usr=2.34%, sys=20.40%, ctx=36, majf=0, minf=0
  IO depths    : 1=100.5%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2040,0,10 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=869MiB/s (911MB/s), 869MiB/s-869MiB/s (911MB/s-911MB/s), io=2040MiB (2139MB), run=2348-2348msec

root@freenas ~ # ls /octa
file.0.0	file.0.2	file.0.4	file.0.6	file.0.8
file.0.1	file.0.3	file.0.5	file.0.7	file.0.9

root@freenas ~ # fio --name=read_file --rw=read --bs=1M --size=2G --filename_format=/octa/file.\$jobnum.\$filenum --allow_file_create=0 --nrfiles=10 --debug=file
fio: set debug option file
file     102664 add file /octa/file.0.0
file     102664 resize file array to 11 files
file     102664 file 0x801a040d0 "/octa/file.0.0" added at 0
file     102664 add file /octa/file.0.1
file     102664 file 0x801a04290 "/octa/file.0.1" added at 1
file     102664 add file /octa/file.0.2
file     102664 file 0x801a04450 "/octa/file.0.2" added at 2
file     102664 add file /octa/file.0.3
file     102664 file 0x801a04610 "/octa/file.0.3" added at 3
file     102664 add file /octa/file.0.4
file     102664 file 0x801a047d0 "/octa/file.0.4" added at 4
file     102664 add file /octa/file.0.5
file     102664 file 0x801a04990 "/octa/file.0.5" added at 5
file     102664 add file /octa/file.0.6
file     102664 file 0x801a04b50 "/octa/file.0.6" added at 6
file     102664 add file /octa/file.0.7
file     102664 file 0x801a04d10 "/octa/file.0.7" added at 7
file     102664 add file /octa/file.0.8
file     102664 file 0x801a04ed0 "/octa/file.0.8" added at 8
file     102664 add file /octa/file.0.9
file     102664 file 0x801a05090 "/octa/file.0.9" added at 9
read_file: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.5
Starting 1 process
file     102664 setup files
file     102664 get file size for 0x801a040d0/0//octa/file.0.0
file     102664 get file size for 0x801a04290/1//octa/file.0.1
file     102664 get file size for 0x801a04450/2//octa/file.0.2
file     102664 get file size for 0x801a04610/3//octa/file.0.3
file     102664 get file size for 0x801a047d0/4//octa/file.0.4
file     102664 get file size for 0x801a04990/5//octa/file.0.5
file     102664 get file size for 0x801a04b50/6//octa/file.0.6
file     102664 get file size for 0x801a04d10/7//octa/file.0.7
file     102664 get file size for 0x801a04ed0/8//octa/file.0.8
file     102664 get file size for 0x801a05090/9//octa/file.0.9
read_file: Laying out IO files (10 files / total 2047MiB)
file     102664 layout unlink /octa/file.0.0
file     102664 open file /octa/file.0.0, flags 401
fio: file creation disallowed by allow_file_create=0


Run status group 0 (all jobs):

root@freenas ~ # ls /octa
file.0.1	file.0.3	file.0.5	file.0.7	file.0.9
file.0.2	file.0.4	file.0.6	file.0.8

[sitsofe]

@llamafilm nice report! Just to check - after pass 1 all files are of the correct size?

[sitsofe]

Thinking about this we're going to hit rounding issues because we want to divide size by the number of files:

>>> 2048*1024*1024/10.0
214748364.8

I guess the question becomes what do we do? Work out the rounded down sizes and then make the files at that size? Do the same but round up? I guess it doesn't matter so long as we're consistent...

[llamafilm]

Thank you. I just checked, and all 10 files are exactly 213,909,504 Bytes. That's a pretty large rounding error, over 800K.
I actually just repeated the test with nrfiles=9 and I get the same problem, it unlinks the first file. That's weird, I could have sworn I did that earlier today and it worked.

[sitsofe]

OK that makes more sense. Looking at

fio/filesetup.c

Line 772 in 3c029ac

f->real_file_size = td->o.size / td->o.nr_files;

shows how we work out the file size when the file doesn't exist. However if the section that checks whether the file is already the right size (roughly

fio/filesetup.c

Line 1035 in 3c029ac

/*

?) does a different calculation we're going to want to unlink it and "extend" it to the right size (which will be the "wrong" smaller size but then we don't check again afterwards).

[sitsofe]

BTW: I think the error might be that large because it wants to make each file's size a multiple of the block size.

[sitsofe]

BTW if the files that we want already exist at a size equal to or greater than what we want they won't need to be recreated. Maybe you had leftover big files from something else when you didn't see the issue earlier?

[llamafilm]

Yeah, you're probably right that I had larger files left over from a previous job.
Since the block size is 1MiB, rounding to 214,748,160 Bytes would be closer. But I don't think the exact size is so important, as long as the reader and the writer calculate it consistently.