Virtualbox 7.2.6 + OpenZFS 2.4.1 in Ubuntu 26.04. The death of L1ARC?

[BertN45]

The problem" The L1ARC did not fill at all (max say 0.4 GiB of 8.0 GiB) and consequently L2ARC stayed behind. VMs running on nvme was just acceptable, VMs running from cached HDD were bad, boot times > 1 minute and response times were disastrous. I did not have the problem with VirtualBox 7.2.6 + OpenZFS 2.2.2 in Ubuntu 24.04.

Solution: Enable the Host IO cache in the storage section of VirtualBox.

History: I'm 80, I use Ubuntu since 2008, VirtualBox since 2010 and OpenZFS since 2018. From caching I understood after say 2010, that if the VM enabled the use of the Host IO cache,it would increase the chance of corruption in case of crashes (for me in the Caribbean 2 to 12 power-fails per day). So my Host IO cache in VirtualBox was off. I kept the settings after introducing OpenZFS, because I already cached 2 times, once in the VM and once in the L1ARC. Those settings kept me happy till April 2026 when I upgraded to 26.04 and OpenZFS 2.4.1.

Explanation: The Issue as I understand it: If the Host IO cache is disabled in the VM storage section, OpenZFS/VBox seems to use direct/asynchronous IOs in 2.4.1. In the past OpenZFS 2.2.2 it was no issue, because they used always synchronous IO that went to the L1ARC. However to speed up professional apps run in VMs, they bypassed the L1ARC now, saving CPU caching time and go directly to nvme or HDD. If you select the Host IO Cache they assume synchronous/sequential IO and cache it in the L1ARC cache to be collected and written once in say 5 seconds.

So now I have 3 caches, ~1 GiB in the VM Page/IO cache; ~2 GiB in the Host Page/IO cache and 5.75 GiB in L1ARC (uncompressed 9.5 GiB :). Note that all 3 caches are basically free memory, that will be used, if needed for other apps or VMs.

[amotin]

ZFS 2.4 now respects O_DIRECT flag, if set by application. If that is what Virtualbox does, then it is a question to them about the motivation. If needed, it can be overridden on dataset level with direct property.

[BertN45]

All my direct flags in the datasets are set to standard (default). That seems to be the smart default, OpenZFS chooses for datapools and thus its datasets, for I expect compatibility with older versions of OpenZFS.

The primary values/behaviors associated with Direct I/O in ZFS include:

• Standard (Buffered/Default): I/O passes through the ARC (Adaptive Replacement Cache).
• Direct (1 or on): Bypasses the ARC and the operating system page cache, forcing direct I/O between the application and the ZFS Virtual Device (VDEV) layer.
• Disabled (0 or off): Explicitly disables direct I/O, ensuring data is cached.

So it is an annoying bug in OpenZFS, because it seems to ignore the standard value of the direct flag in the dataset, destroying default compatibility with older releases. If you run nvme or raid z configurations, your users might not really notice the difference.
But I'm solo with a relative slow nvme and one HDD with one sata cache device, you will notice a difference. Besides I display in conky: ARC Size and Hit Rate of both L1ARC and L2ARC. So I directly noticed the difference in ARC behavior.

[amotin]

standard means following POSIX specifications and application requests. It means cached I/O unless application sets O_DIRECT. If it is a bug, then not ZFS, but application.

[BertN45]

About OPenZFS and POSIX: Although designed to be POSIX-compliant, the high performance, copy-on-write nature of ZFS can lead to non-standard behavior in specific situations.

POSIX does not specify anything about L1ARC! The specs I showed you stated clearly the "standard" means I/O passes through the ARC. It does not pass through the ARC, so it is inconsistent with documentation.
I'm sorry that you, don't seem to understand that inconsistency.

[amotin]

Direct I/O support is a new feature of OpenZFS 2.4. It may indeed bypass the ARC when requested and possible. O_DIRECT is a POSIX standard way to request it, which ZFS finally implements now. If you see some documentation inconsistency, please show where. The zfsprops man page states: "The standard behavior for Direct I/O requests is to bypass the ARC when possible."

[BertN45]

You were right and the AI generated reply I found was wrong.

However if you have more property values, then I (and AI) would expect that you would have :

• always use direct IO to the disk.
• disable the use direct IO to the disk and always go through the ARC
• request use direct IO if requested by the IO operation.
• default, use the old behavior of 2.2.2 as the default one.

I'm almost sure that VirtualBox has not been changed in many years with respect to direct IO. They only talk about using the Host IO Cache in the kernel for the VM IOs, you can enable it or disable it. In the past it only worked on the Host IO cache, but now due to the change in OpenZFS 2.4.1, as an undocumented side effect it also handles the ARC.

So to get back to the old situation, in the 2 datapools (and datasets?) I have to disable the direct IO property and slowly I have to stop the redundant caching in the Host IO cache again. Slowly because with 4 running VMs and 2 hands on a Ryzen 5 5600GT and 32GB DDR4, I can afford to waste some CPU time and memory :)

[BertN45]

I tried to use the direct=disabled property and that one was completely ignored.

I'm happy that my solution in VirtualBox enabling the Host IO Cache worked. I expect that the warning about file corruption given by VBox about using the Host IO Cache is not true in the case of OpenZFS.