storage: ballast file has no effect under zfs with compression=on #78606

Describe the problem

zfs does not support fallocate, so we write zeroes. On zfs with compression=on, zero blocks are not written to disk and do not take up disk space. So the ballast file has no effect.

roachprod create -n 1 --filesystem zfs $USER-zfs
roachprod stage cockroach
roachprod ssh $USER-zfs:1
$ sudo zfs set compression=on data1
$ ./cockroach debug ballast -z 600G /mnt/data1/ballast-compress
# ^-- ca 200s, i.e. 3GB/s, i.e. not really hitting disk
$ ls -lah /mnt/data1/ballast-compress 
-rw-rw-r-- 1 ubuntu ubuntu 559G Mar 28 13:44 /mnt/data1/ballast-compress
$ df -h /mnt/data1/
Filesystem      Size  Used Avail Use% Mounted on
data1           363G  128K  363G   1% /mnt/data1
$ zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
data1   374G  2.70M   374G        -         -     0%     0%  1.00x    ONLINE  -

To Reproduce

What did you do? Describe in your own words.

If possible, provide steps to reproduce the behavior:

1. Set up CockroachDB cluster ...
2. Send SQL ... / CLI command ...
3. Look at UI / log file / client app ...
4. See error

Expected behavior

The goal of a ballast file is not to create a file of the given size, but to create a file that frees disk space when deleted. So we would want a file that, when deleted, allows new files of approximately the same logical size to be created after.

We would achieve this by writing random data into the ballast files instead of zeroes.

Additional data / screenshots

https://github.com/cockroachlabs/support/issues/1503 (internal)

Environment:

• linux, zfs compression=on

Additional context

cc @nicktrav

Jira issue: CRDB-14195