Help: How to enable multithreading with libext for FIO testing?

[Luca-Cliver]

I wonder how can I enable multithreading. I failed to find some guide in readme. I want to use multithread with libext to run fio, so may I get some help here?
Really sorry for take up your time!

[wkozaczuk]

Hi,

You do not need to do anything special to enable multi-threading on the OSv libext side. Please follow this doc to setup and run OSv with an ext disk mounted - https://github.com/cloudius-systems/osv/tree/master/modules/libext#readme.

As far as fio goes, there is nothing special you have to do. Here are the examples of me running fio with libext:

./modules/libext/test.sh '/fio --name=fiotest --directory=/data/randread --numjobs=4 --size=1M --rw=randread --time_based=1 --runtime=10'
###### Running /fio --name=fiotest --directory=/data/randread --numjobs=4 --size=1M --rw=randread --time_based=1 --runtime=10
OSv v0.57.0-169-g7ceff0f5
eth0: 192.168.122.15
Booted up in 203.74 ms
Cmdline: /fio --name=fiotest --directory=/data/randread --numjobs=4 --size=1M --rw=randread --time_based=1 --runtime=10
fiotest: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.29-dirty
Starting 4 threads
Jobs: 4 (f=4): [r(4)][100.0%][r=323MiB/s][r=82.7k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=48: Tue Jul  9 19:23:09 2024
  read: IOPS=20.0k, BW=78.2MiB/s (82.0MB/s)(782MiB/10000msec)
    clat (usec): min=15, max=4308, avg=49.21, stdev=23.78
     lat (usec): min=15, max=4308, avg=49.30, stdev=23.78
    clat percentiles (usec):
     |  1.00th=[   28],  5.00th=[   34], 10.00th=[   37], 20.00th=[   42],
     | 30.00th=[   46], 40.00th=[   49], 50.00th=[   50], 60.00th=[   52],
     | 70.00th=[   53], 80.00th=[   54], 90.00th=[   58], 95.00th=[   63],
     | 99.00th=[   76], 99.50th=[   86], 99.90th=[  182], 99.95th=[  285],
     | 99.99th=[  979]
   bw (  KiB/s): min=74840, max=87528, per=25.00%, avg=79960.42, stdev=3960.16, samples=19
   iops        : min=18710, max=21882, avg=19989.95, stdev=990.10, samples=19
  lat (usec)   : 20=0.01%, 50=51.39%, 100=48.41%, 250=0.14%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.02%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=399.98%, sys=0.00%, ctx=0, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=200097,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
fiotest: (groupid=0, jobs=1): err= 0: pid=49: Tue Jul  9 19:23:09 2024
  read: IOPS=20.1k, BW=78.6MiB/s (82.4MB/s)(786MiB/10001msec)
    clat (usec): min=17, max=6119, avg=48.82, stdev=24.86
     lat (usec): min=17, max=6119, avg=48.94, stdev=24.93
    clat percentiles (usec):
     |  1.00th=[   28],  5.00th=[   33], 10.00th=[   36], 20.00th=[   42],
     | 30.00th=[   46], 40.00th=[   49], 50.00th=[   50], 60.00th=[   51],
     | 70.00th=[   53], 80.00th=[   54], 90.00th=[   58], 95.00th=[   61],
     | 99.00th=[   76], 99.50th=[   87], 99.90th=[  167], 99.95th=[  249],
     | 99.99th=[  914]
   bw (  KiB/s): min=73644, max=87449, per=25.15%, avg=80432.89, stdev=3962.57, samples=19
   iops        : min=18411, max=21862, avg=20108.11, stdev=990.63, samples=19
  lat (usec)   : 20=0.01%, 50=52.29%, 100=47.51%, 250=0.15%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.02%
  lat (msec)   : 2=0.01%, 10=0.01%
  cpu          : usr=399.98%, sys=0.00%, ctx=0, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=201232,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
fiotest: (groupid=0, jobs=1): err= 0: pid=50: Tue Jul  9 19:23:09 2024
  read: IOPS=20.0k, BW=78.2MiB/s (82.0MB/s)(782MiB/10001msec)
    clat (usec): min=20, max=4300, avg=49.02, stdev=22.69
     lat (usec): min=20, max=4300, avg=49.14, stdev=22.69
    clat percentiles (usec):
     |  1.00th=[   28],  5.00th=[   32], 10.00th=[   36], 20.00th=[   42],
     | 30.00th=[   46], 40.00th=[   49], 50.00th=[   50], 60.00th=[   51],
     | 70.00th=[   53], 80.00th=[   55], 90.00th=[   59], 95.00th=[   62],
     | 99.00th=[   77], 99.50th=[   87], 99.90th=[  178], 99.95th=[  302],
     | 99.99th=[  988]
   bw (  KiB/s): min=75688, max=85884, per=25.02%, avg=80022.47, stdev=3140.16, samples=19
   iops        : min=18922, max=21471, avg=20005.53, stdev=785.07, samples=19
  lat (usec)   : 50=52.27%, 100=47.51%, 250=0.16%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.02%
  lat (msec)   : 2=0.01%, 10=0.01%
  cpu          : usr=399.98%, sys=0.00%, ctx=0, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=200277,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
fiotest: (groupid=0, jobs=1): err= 0: pid=51: Tue Jul  9 19:23:09 2024
  read: IOPS=19.8k, BW=77.4MiB/s (81.1MB/s)(774MiB/10001msec)
    clat (usec): min=20, max=1680, avg=49.47, stdev=20.16
     lat (usec): min=20, max=1681, avg=49.58, stdev=20.17
    clat percentiles (usec):
     |  1.00th=[   28],  5.00th=[   33], 10.00th=[   37], 20.00th=[   42],
     | 30.00th=[   46], 40.00th=[   49], 50.00th=[   50], 60.00th=[   52],
     | 70.00th=[   53], 80.00th=[   55], 90.00th=[   59], 95.00th=[   63],
     | 99.00th=[   79], 99.50th=[   89], 99.90th=[  178], 99.95th=[  277],
     | 99.99th=[  938]
   bw (  KiB/s): min=70944, max=85840, per=24.74%, avg=79143.00, stdev=3701.37, samples=19
   iops        : min=17736, max=21460, avg=19785.58, stdev=925.30, samples=19
  lat (usec)   : 50=51.24%, 100=48.53%, 250=0.18%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.02%
  lat (msec)   : 2=0.01%
  cpu          : usr=399.98%, sys=0.00%, ctx=0, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=198122,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=312MiB/s (328MB/s), 77.4MiB/s-78.6MiB/s (81.1MB/s-82.4MB/s), io=3124MiB (3276MB), run=10000-10001msec

./modules/libext/test.sh '/fio --name=fiotest --directory=/data/randwrite --numjobs=4 --size=1M --rw=randwrite --time_based=1 --runtime=10'
###### Running /fio --name=fiotest --directory=/data/randwrite --numjobs=4 --size=1M --rw=randwrite --time_based=1 --runtime=10
OSv v0.57.0-169-g7ceff0f5
eth0: 192.168.122.15
Booted up in 209.69 ms
Cmdline: /fio --name=fiotest --directory=/data/randwrite --numjobs=4 --size=1M --rw=randwrite --time_based=1 --runtime=10
fiotest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.29-dirty
Starting 4 threads
fiotest: Laying out IO file (1 file / 1MiB)
fiotest: Laying out IO file (1 file / 1MiB)
fiotest: Laying out IO file (1 file / 1MiB)
fiotest: Laying out IO file (1 file / 1MiB)
Jobs: 4 (f=4): [w(4)][100.0%][w=247MiB/s][w=63.3k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=48: Tue Jul  9 19:23:35 2024
  write: IOPS=15.1k, BW=59.2MiB/s (62.0MB/s)(592MiB/10000msec); 0 zone resets
    clat (usec): min=32, max=52133, avg=65.07, stdev=142.07
     lat (usec): min=32, max=52134, avg=65.21, stdev=142.07
    clat percentiles (usec):
     |  1.00th=[   45],  5.00th=[   52], 10.00th=[   56], 20.00th=[   58],
     | 30.00th=[   60], 40.00th=[   61], 50.00th=[   62], 60.00th=[   64],
     | 70.00th=[   65], 80.00th=[   68], 90.00th=[   74], 95.00th=[   88],
     | 99.00th=[  121], 99.50th=[  141], 99.90th=[  249], 99.95th=[  502],
     | 99.99th=[ 1106]
   bw (  KiB/s): min=44880, max=63664, per=24.91%, avg=60413.05, stdev=4240.15, samples=19
   iops        : min=11220, max=15916, avg=15103.11, stdev=1060.02, samples=19
  lat (usec)   : 50=3.66%, 100=93.56%, 250=2.68%, 500=0.05%, 750=0.02%
  lat (usec)   : 1000=0.02%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 100=0.01%
  cpu          : usr=399.98%, sys=0.00%, ctx=0, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,151445,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
fiotest: (groupid=0, jobs=1): err= 0: pid=49: Tue Jul  9 19:23:35 2024
  write: IOPS=15.2k, BW=59.4MiB/s (62.3MB/s)(594MiB/10001msec); 0 zone resets
    clat (usec): min=33, max=36324, avg=64.81, stdev=116.42
     lat (usec): min=33, max=36324, avg=64.95, stdev=116.42
    clat percentiles (usec):
     |  1.00th=[   44],  5.00th=[   51], 10.00th=[   55], 20.00th=[   58],
     | 30.00th=[   59], 40.00th=[   61], 50.00th=[   62], 60.00th=[   64],
     | 70.00th=[   65], 80.00th=[   68], 90.00th=[   74], 95.00th=[   88],
     | 99.00th=[  122], 99.50th=[  143], 99.90th=[  251], 99.95th=[  498],
     | 99.99th=[ 1090]
   bw (  KiB/s): min=43248, max=64689, per=25.03%, avg=60702.47, stdev=4954.98, samples=19
   iops        : min=10812, max=16172, avg=15175.47, stdev=1238.70, samples=19
  lat (usec)   : 50=4.72%, 100=92.51%, 250=2.67%, 500=0.05%, 750=0.02%
  lat (usec)   : 1000=0.02%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=399.99%, sys=0.00%, ctx=0, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,152037,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
fiotest: (groupid=0, jobs=1): err= 0: pid=50: Tue Jul  9 19:23:35 2024
  write: IOPS=15.0k, BW=58.8MiB/s (61.6MB/s)(588MiB/10001msec); 0 zone resets
    clat (usec): min=31, max=50021, avg=65.53, stdev=140.61
     lat (usec): min=31, max=50022, avg=65.66, stdev=140.61
    clat percentiles (usec):
     |  1.00th=[   45],  5.00th=[   52], 10.00th=[   56], 20.00th=[   58],
     | 30.00th=[   60], 40.00th=[   61], 50.00th=[   62], 60.00th=[   64],
     | 70.00th=[   65], 80.00th=[   68], 90.00th=[   75], 95.00th=[   89],
     | 99.00th=[  122], 99.50th=[  145], 99.90th=[  249], 99.95th=[  523],
     | 99.99th=[ 1029]
   bw (  KiB/s): min=44144, max=63576, per=24.77%, avg=60063.95, stdev=4791.39, samples=19
   iops        : min=11036, max=15894, avg=15015.84, stdev=1197.83, samples=19
  lat (usec)   : 50=3.21%, 100=93.96%, 250=2.73%, 500=0.05%, 750=0.02%
  lat (usec)   : 1000=0.02%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 100=0.01%
  cpu          : usr=399.99%, sys=0.00%, ctx=0, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,150461,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
fiotest: (groupid=0, jobs=1): err= 0: pid=51: Tue Jul  9 19:23:35 2024
  write: IOPS=15.2k, BW=59.5MiB/s (62.4MB/s)(595MiB/10001msec); 0 zone resets
    clat (usec): min=34, max=59037, avg=64.68, stdev=156.08
     lat (usec): min=34, max=59037, avg=64.83, stdev=156.09
    clat percentiles (usec):
     |  1.00th=[   44],  5.00th=[   51], 10.00th=[   55], 20.00th=[   58],
     | 30.00th=[   59], 40.00th=[   61], 50.00th=[   62], 60.00th=[   64],
     | 70.00th=[   65], 80.00th=[   68], 90.00th=[   74], 95.00th=[   86],
     | 99.00th=[  118], 99.50th=[  143], 99.90th=[  251], 99.95th=[  449],
     | 99.99th=[  955]
   bw (  KiB/s): min=47208, max=65259, per=25.10%, avg=60875.32, stdev=4275.98, samples=19
   iops        : min=11802, max=16314, avg=15218.68, stdev=1068.96, samples=19
  lat (usec)   : 50=4.35%, 100=93.14%, 250=2.40%, 500=0.05%, 750=0.02%
  lat (usec)   : 1000=0.02%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 100=0.01%
  cpu          : usr=399.99%, sys=0.00%, ctx=0, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,152385,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=237MiB/s (248MB/s), 58.8MiB/s-59.5MiB/s (61.6MB/s-62.4MB/s), io=2368MiB (2484MB), run=10000-10001msec

[Luca-Cliver]

Sincerely appreciate your response. But I didn't use the .sh files here for testing; instead, I used the contents under the apps directory. I ran the following two commands:
./scripts/build image=libext,fio &
./scripts/run.py --execute='--mount-fs=ext,/dev/vblk1,/data /hello' --second-disk-image ./ext_images/ext4.img
If I want to launch a multi-threaded task, do I just need to set the -c option in the scripts manually?