Zero-block cache ignored?

[martona]

First of all huge thanks for creating and maintaining this project, it's fantastic!

I do have an issue I'm not sure what to do about. Upon startup, s3backer lists nonzero blocks and sets up the bitfield cache; from the stats file this appears to work as expected (showing the case of a brand-new block device on an empty bucket):

2025-06-15 17:31:57 INFO: starting non-zero block survey
2025-06-15 17:31:57 INFO: mounting /mnt/s3backer
2025-06-15 17:31:58 INFO: non-zero block survey completed (0 non-zero blocks reported)

http_normal_blocks_read      0
http_normal_blocks_written   0
http_zero_blocks_read        0
http_zero_blocks_written     0
http_empty_blocks_read       0
http_empty_blocks_written    0
http_gets                    16
http_puts                    1
http_deletes                 0
...
zero_block_cache_size        67108864 blocks
zero_block_cache_read_hits   0
zero_block_cache_write_hits  0
...

Then I go and create a filesystem, write a few gigabytes, and stats is showing this:

http_normal_blocks_read      25
http_normal_blocks_written   15344
http_zero_blocks_read        13080
http_zero_blocks_written     2
http_empty_blocks_read       0
http_empty_blocks_written    0
http_gets                    41
http_puts                    15345
http_deletes                 2
...
zero_block_cache_size        67095784 blocks
zero_block_cache_read_hits   12
zero_block_cache_write_hits  3
...

Am I wrong to expect 0 in http_zero_blocks_read , and a high value in cache_xxx_hits?

I wouldn't care about a few thousand unnecessary network reads (amazon will kill me on storage fees, the cost of the initial requests is just noise) but they seem to strongly impact performance as the worker threads seem to flap between "we're all checking for zero blocks" (not good) and "we're busy doing PUTs)" (great).

I'm seeing this in nbd mode as well as legacy. s3backer is built from a git pull as of a few days ago. I can provide more details and help troubleshoot if needed, but I wanted to confirm first that I'm not misinterpreting what the behavior should be.

[martona]

Just to provide some color on this, this is what I'm seeing on the network as the threads (seemingly all together) keep switching between listing zero blocks and writing to s3:

[archiecobbs]

Question...

In your first comment there is this:

2025-06-15 17:31:57 INFO: starting non-zero block survey
2025-06-15 17:31:57 INFO: mounting /mnt/s3backer
2025-06-15 17:31:58 INFO: non-zero block survey completed (0 non-zero blocks reported)

So in that scenario you are starting from scratch, and so all S3 blocks are zero (i.e., nonexistent) and the survey completes immediately.

Then there is this:

this is what I'm seeing on the network as the threads (seemingly all together) keep switching between listing zero blocks and writing to s3:

Is this the same scenario as the first comment, where you are starting from scratch? If so then the behavior doesn't make sense - because the zero block survey should have already completed immediately long ago.

However, if you are starting with an existing S3 bucket that has a lot of blocks leftover from previous work, and you start s3backer up, then the zero block survey is going to take some time. If you then create a filesystem before the zero block survey completes, then it would not be surprising that the HTTP layer is going to have to read some zero blocks - in situations where the survey hasn't yet gotten to them.

There may still be some weird behavior with thread scheduling or something but let's address this first question first.

[martona]

I agree that it doesn't seem to make much sense, which is why I thought it was worth it to bug you with it. It's a blank 0-block bucket, block cache file auto-created upon startup from scratch, the 50-thread --listBlocks completes immediately with the log entries posted. Yet as I start writing to it, s3backer starts increasing the "http_zero_blocks_read" counter and cache hits are basically nonexistent.

I agree we can table the thread scheduling for now; I think I may have conflated two issues and the flapping is more between handling network input to block cache, and then writing block cache to s3 blocks. While this probably has greater impact on performance than zero-block verification (network input stalls while output peaks, and vica versa) it's probably harder to pinpoint.

My config:

#/etc/s3backer.configfile
--ssl
--encrypt=aes-128-cbc
--passwordFile=/etc/s3backer.passwordfile
--compress=zstd
--compress-level=1
--accessFile=/etc/s3backer.accessfile
--blockCacheFile=/blockcache/s3backer.cache
--blockCacheSize=524000
--blockCacheThreads=32
--blockCacheWriteDelay=15000
--blockCacheRecoverDirtyBlocks
--blockCacheNumProtected=1024
--size=1024T
--statsFileMirror=/tmp/s3backer.stats
--storageClass=STANDARD
--blockSize=4M
--listBlocks
--listBlocksThreads=50

Blockcache setup:

zpool create -o ashift=12 -o autotrim=on -O atime=off -O \
    compression=off -O redundant_metadata=most -O xattr=sa \
    -O primarycache=metadata -O recordsize=4M -O sync=disabled \
    -O checksum=sha256  blockcache /dev/nvme1n1

s3backer command line (this one uses legacy mode but the behavior is identical with --nbd):

s3backer -f --configFile=/etc/s3backer.configfile martona-s3backer/zfsrecv /mnt/s3backer

s3backer filesystem:

zpool create -o ashift=12 -o autotrim=on -O atime=off \
    -O compression=off -O redundant_metadata=most \
    -O xattr=sa -O primarycache=metadata -O recordsize=4M \
    -O sync=disabled -O checksum=sha256 -O encryption=aes-256-gcm \
    -O keyformat=passphrase -O keylocation=file:///etc/ec2-zfsrecv.s3.zfskey \
   s3 /mnt/s3backer/file

Edit: formatting

[archiecobbs]

I'll admit I don't completely know what's happening but here are a few thoughts.

Yet as I start writing to it, s3backer starts increasing the "http_zero_blocks_read" counter and cache hits are basically nonexistent.

Regarding zero_block_cache_read_hits being a low value: A cache hit in the zero block cache, in this situation, would reflect the filesystem reading a block from the uninitialized disk that it has never written to (or at least, never written anything other than zeroes to). There's no reason it would ever do that, because it already knows the data is garbage. So this number being low is not surprising.

Once your filesystem is initialized, most blocks that it's going to access are not all zeroes, so again zero cache hits would be unlikely. That's probably true in most cases actually: many filesystems "compress" zero blocks using extents or whatever...

Regarding http_zero_blocks_read not being a low value: I'm less sure about what's happening here. One thing is that your S3 block size (4M) is a lot larger than 4096, so this could be a result of fragmentation: when the kernel is reading/writing from multiple portions of an S3 block at the same time, some caching has to be suppressed.

[martona]

It's reasonable to suspect write amplification and fragmentation, but I'm trying to stream in encrypted zfs snapshots, which should be almost purely sequential writes of incompressible data, especially to a brand new empty filesystem. From the # of blocks written at the end of a job I can confirm that this is the case.

The strange part is the 15k http blocks written, and 13k empty blocks read. If I interpret this correctly, s3backer read almost every empty block from s3 that it intended to write to.

[archiecobbs]

The strange part is the 15k http blocks written, and 13k empty blocks read. If I interpret this correctly, s3backer read almost every empty block from s3 that it intended to write to.

I'm not sure why that's happening.

One thing to try would be to conduct the experiment with the kernel block size = s3backer block size (e.g., 4096) and seeing if the problem disappears.

There may be some fragmentation corner cases that results from the fact that the "zero cache" and the "block cache" are not unified (as they should be).

[abzmr]

This pattern closely resembles my experience when doing mostly big write batches to btrfs on LUKS on NBD on s3backer on Backblaze. [Thanks Archie, s3backer is very much appreciated!]

I patched in some detailed statistics to figure out what is going on: #244

Turns out that in my case, using 128kB blocksize, a large majority of NBD access is actually 128kB (yay!) but nearly always wrongly aligned (boo!). So almost all NBD writes consist of two partial block writes -- which I suspect will be largely consecutive, so the first partial block is in the block cache already from the previous write, and the last partial block is not in the block cache and has never been used, i.e. is zero.

A partial block write is handled by the block cache. To end up with a full block to send to s3 later, the block cache will need to have the remaining part of that block as well. (Theoretically it could wait for additional writes to the same block, but that's a very tricky thing to do...)

The issue here is that the block cache sits below the zero cache. So the block cache can not reach any zero cache knowledge, and has no choice but to read from s3 directly. So for every "known zero" block that is partially written by FUSE/NBD, you'll see one http_zero_blocks_read.

(This is a GET request, which on Backblaze is chargeable even when an object doesn't exist. That's what triggered me.)

@martona 's graph showing bulk-switching between zero-GETs and useful-PUTs is easily explained: First zero-GETs happen during FUSE/NBD write operations that fill the s3backer cache. Then when s3backer cache has filled up, things gets finally written out to s3 in bulk.

My suggestion would be to invert the sequence and move the block cache above the zero cache. This is mostly trivial in s3backer_create_store(), but there might be some interfaces can't handle their different counterparts. In particular, the block cache should probably be taught to not store zero blocks.

In my case, I found that a much larger blocksize will reduce the pain enough so that I won't worry about it any more :)

[abzmr]

And reading further into the issues, this is just the same as #201
(That fix was reverted; indeed seems to have either race potential without locking, or deadlock potential with locking...)

[truxton]

I think @abzmr suggestion "move the block cache above the zero cache" is not the same as #201
I also see writes to a fresh s3backer involving GETs , perhaps because I'm using large block size.
The placement of the zero cache above the block cache is done in s3b_config.c and it is very easy
to invert the two (just move the two code blocks with "store = block_cache_store" and "store = zero_cache_store").
The abstraction of cache layers is clean, so this placement is easy. With this change, I get the desired behaviour
that writing to a fresh filesystem only involves PUT, no GET. And the speed is better.

[archiecobbs]

Hmm... why not have it both ways? We can have a zero cache both above and below the block cache.

This is implemented in branch lower-zero-cache helps (untested). See if that helps...

[truxton]

Thanks! One change is needed : util.c needs "config->lower_zero_cache.log = log;"
With that, things are mostly working. In my test, I create a zfs filesystem on top of fuse, using large blocks.
I write ~100 GB to the file system, and the s3backer block cache is 1.7 TB.
On the main branch writing the files happens at ~419 MB/sec
On the new "lower-zero-cache" branch, writing the files happens at ~562 MB/sec
The trouble happens when I delete the test files and do "zpool trim", then the block cache expands to its max size,
and zfs remains at "0% trimmed" even when the number of objects in the s3 bucket does drop down to a low value.
With the master branch (only one zero cache above the block cache), this problem doesnt happen,
the block cache remains a constant size (~100 GB) and the zfs trim operation progresses.
I'm not sure what is bloating the block cache when there is a zero cache above and below it,
and the number of s3 objects is low. I'll debug further...

[archiecobbs]

One change is needed : util.c needs "config->lower_zero_cache.log = log;"

Thanks for that - should be fixed now.

zfs remains at "0% trimmed"

Hmm, not sure why that would happen. The bulk_zero operation is "edited" by a zero cache layer to remove blocks that it already knows are zero, but I don't see how that would cause that behavior... let me know what you find.

[truxton]

Since only the lower zero cache does the zero-block-survey, the upper zero cache is essentially useless, and does not protect the block cache from filling up with zeros that the fs is writing (as part of the trim operation).

I changed it so that the upper zero block cache does the zero block survey, and I implemented
zero_cache_survey_non_zero to invoke the lower layer instead of returning ENOTSUP, and now the
lower layer hooks the callback for its own benefit before calling the survey callback from the
layer above. Now I am observing good behaviour, the block cache does not get bloated,
the zfs trim operation progresses normally, and there are no http GET calls at all
(at least until the large block cache would fill up).

Here is a patch. Please let me know if I should make a git commit somewhere instead of attaching the patch.

0001-lower-zero-cache.patch