Skip to content

storageperf fail with default time limits on h3 instances due to disk speed limitations #356

@jonathanspw

Description

@jonathanspw

From https://cloud.google.com/compute/docs/compute-optimized-machines#h3_limitations

Persistent Disk and Google Cloud Hyperdisk performance is capped at 15,000 IOPS and 240 MBps throughput.

This means the initial fio file write takes about an hour to complete (500GB @ 240MB/s) before the real tests even begin.

With the default timeout of 20m for this, the storageperf tests for h3 instances will time out unless the perfect speed is maintained for a whole hour.

I think a good solution would be write a smaller amount of data for h3 instances, perhaps 20G or 50G instead of 500 for the initial pre-test write.

root       82318  0.0  0.0 1244224 23324 ?       Sl   14:45   0:00              \_ /etc/image_test2047922026/image_test -test.v -test.timeout 20m -test.run TestRandomReadIOPS|TestSequentialReadIOPS|TestRandomWriteIOPS|TestSequentialWriteIOPS
root       82516  0.4  0.0 239496  4928 ?        Sl   14:46   0:08                  \_ fio --name=fill_disk --direct=1 --verify=0 --randrepeat=0 --bs=128K --iodepth=64 --rw=randwrite --iodepth_batch_submit=64 --iodepth_batch_complete_max=64 --ioengine=libaio --filesize=500G --filename=/dev/nvme0n2
root       82686  3.5  0.0 182592  9564 ?        Ss   14:46   1:01                      \_ fio --name=fill_disk --direct=1 --verify=0 --randrepeat=0 --bs=128K --iodepth=64 --rw=randwrite --iodepth_batch_submit=64 --iodepth_batch_complete_max=64 --ioengine=libaio --filesize=500G --filename=/dev/nvme0n2
[root@h3standard88pdbalanced-storageperf-6j2t8 jonathan]# iostat -x 2
Linux 5.14.0-570.49.1.el9_6.x86_64 (h3standard88pdbalanced-storageperf-6j2t8)   10/01/2025      _x86_64_        (88 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.02    0.00    0.04    0.00    0.00   99.94

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
nvme0n1          0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
nvme0n2          0.00      0.00     0.00   0.00    0.00     0.00 1920.00 245760.00     0.00   0.00   33.30   128.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   63.94 100.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions