Align the filesystem to 512 bytes and allow paddings to allow mounting via loop devices.

[knmgr]

When trying to set up a dwarfs file as a loop device, it does not align to the default 512B sector boundary. However, if I pad the file to multiples of 512B, it also fail to mount.
This affects automounting utilities like udevil, which may be used in GUI file managers and such. Another use case is when the dwarfs is written to a dedicated partition (in a firmware image, I imagine), which is likely to have paddings after the filesystem itself.

0 ~:ls -l test.dwarfs
-rw-r--r-- 1 root root 45834020970 03-28 11:28 test.dwarfs
0 ~:mkdir test
0 ~:mount -t fuse.dwarfs test.dwarfs test
0 ~:umount test
0 ~:udevil mount --verbose -t fuse.dwarfs test.dwarfs 
ROOT: /sbin/losetup /dev/loop0 /root/test.dwarfs
losetup: /root/test.dwarfs: Warning: file does not end on a 512-byte sector boundary; the remaining end of the file will be ignored.
ROOT: /bin/mount -v -t fuse.dwarfs -o nosuid,noexec,nodev,lazytime,ro /dev/loop0 /run/mount/test.dwarfs
E 11:26:54.892551 error initializing file system: dwarfs::runtime_error: [fs_section.cpp:68] truncated section data: 45834020970 > 45834020864
      mount exit status = 1
ROOT: /sbin/losetup -d /dev/loop0
0 ~:echo $((45834020970%512))
106
0 ~:echo $((45834020970-106+512))
45834021376
0 ~:truncate -s 45834021376 test.dwarfs 
0 ~:udevil mount --verbose -t fuse.dwarfs test.dwarfs 
ROOT: /sbin/losetup /dev/loop0 /root/test.dwarfs
ROOT: /bin/mount -v -t fuse.dwarfs -o nosuid,noexec,nodev,lazytime,ro /dev/loop0 /run/mount/test.dwarfs
E 11:29:02.360347 error initializing file system: dwarfs::runtime_error: [fs_section.cpp:52] truncated section header: 45834021354 + 64 > 45834021376
      mount exit status = 1
ROOT: /sbin/losetup -d /dev/loop0

Also, please consider adding a filesystem label and UUID, so the mountpoint can be displayed better in GUI file managers.

[mhx]

When trying to set up a dwarfs file as a loop device, it does not align to the default 512B sector boundary. However, if I pad the file to multiples of 512B, it also fail to mount.

That is correct. If you pad it, you need to pass -o imagesize={original size} when mounting.

I agree this is suboptimal. Historically, writing a DwarFS image was always done "streaming", i.e. once data was written, you couldn't go back and change something. As of v0.8.0, you could also stream to stdout and pipe the image into another tool. That means it wasn't really an easy option to embed the "total image size" at a known location, hence it was never done. That's why DwarFS currently relies on knowing the image size either from the file size, or from an explicit argument.

There are several ways to address this, but I'm currently favouring the following, which would also address your second request:

• Add an (optional) "superblock" section as the first section in the DwarFS image. This section will be a regular DwarFS section with a new section type. It will be special in that it is fixed size and must be uncompressed.
• Add an (optional) "padding" section that in theory can be inserted anywhere between the "superblock" and the section index sections. This section has variable size, but is also uncompressed.

The "superblock" section would hold the following:

• A 64-bit image size value, in little-endian byte order. A value of zero means "unset".
• A 16-byte UUID. A value of all-zeroes means "unset".
• A 64-byte filesystem label, zero padded.

Along with the header magic/version, this would allow tools/libraries to easily read size / UUID, and label.

It would also easily be possible to change the label, or set the UUID / image size. E.g., you might want to assign a UUID only after you've determined that the image isn't bit-identical to a previous version. Or you might want to stream the image from mkdwarfs somewhere and still want to update the image size later.

This is actually a backwards-compatible change: older DwarFS tools would simply skip over the superblock they don't recognize (and likely log a warning that they've discovered an unknown section).

However, it is still not possible to add arbitrary padding even with the superblock in place. While new tools could pick up the image size and act accordingly, older tools still assume that the image size is equal to the file size. To address this, if you want you image to be a multiple of the sector size (e.g. 512 bytes), you can insert a "padding" section just before the section index. That way, from the POV of an old tool, the section index is still at the very end of the image, and the padding section before it is one that can, again, simply be ignored.

How this is going to be exposed by mkdwarfs is something I'll have to think about.

I'm inclined to make mkdwarfs output the superblock by default to help with adoption. It will still be able to force it to not generate a superblock. Adding a superblock after writing the image will require a full re-write. But modifying an existing superblock can be done in place (e.g. for changing the label or finalizing the size of a streamed image).

The padding section will be optional and only be inserted if explicitly requested. However, it will be possible to pad an existing image almost in-place since that will only require the section index to be updated/rewritten.

Let me know what you think. I'm happy to consider these for the next minor release.

[knmgr]

once data was written, you couldn't go back and change something.

I think that the "streaming" approach has its limitations. As mentioned in #322, it would be nice to use a "non-streaming mode" when the output is a regular file on local disk (which is the majority of use cases). I wonder whether it is possible to remove or reduce the input/output buffering when the streaming requirement is gone, in order to save memory.

To address this, if you want you image to be a multiple of the sector size (e.g. 512 bytes), you can insert a "padding" section just before the section index.

The padding is nearly always at the end. In my firmware image example, the partition would be slightly larger than the filesystem and you would dd the image to it and dd the new image over it when updating. In other words, you do not know how many bytes would be padded in advance; with the original size stored, it should be able to ignore the "tail".

The "superblock" section would hold the following:

The superblock would also contain the sector size for the loop device, which would either be 0 (no concern for raw disk, treated as 512 in loop), 512 ("traditional" and default), 2048 (CDs/DVDs) or 4096 (advanced format, and also the page size). Since the length of remaining data is written to the section header, it would be possible to align each section to sector sizes. (It may lead to performance increases with small blocks and "fast" compressions, but I imagine it would be too low to matter.)

[mhx]

once data was written, you couldn't go back and change something.

I think that the "streaming" approach has its limitations. As mentioned in #322, it would be nice to use a "non-streaming mode" when the output is a regular file on local disk (which is the majority of use cases). I wonder whether it is possible to remove or reduce the input/output buffering when the streaming requirement is gone, in order to save memory.

For that to work, I would have to know where to write a block, but that is only known when all previous block sizes are known, at which point I can just write everything. Sure, blocks could theoretically be written in random order with an additional index to resolve them, but then you're lacking bit-identical output across multiple runs.

To address this, if you want you image to be a multiple of the sector size (e.g. 512 bytes), you can insert a "padding" section just before the section index.

The padding is nearly always at the end. In my firmware image example, the partition would be slightly larger than the filesystem and you would dd the image to it and dd the new image over it when updating. In other words, you do not know how many bytes would be padded in advance; with the original size stored, it should be able to ignore the "tail".

So with my proposed changes, there are three possible scenarios in addition to the current state:

1. A superblock is added to the image, but the image isn't padded to the sector size. In this case, only DwarFS FUSE drivers that know how to read the superblock will be able to read the image when mounted via a loop device.
2. A padding section is added to the image, but no superblock. The image can be read by any DwarFS driver that will ignore unknown sections (>=0.7.4).
3. Both superblock and padding are used. New DwarFS tools can make use of the superblock, but old tools would still be able to read the image.

(1) is probably going to be the new default along with a --no-superblock switch if you want to keep the image as small as possible or you don't need the superblock features. An --empty-uuid switch would help with producing bit-identical images. An --init-superblock switch would populate the UUID and size fields if they are empty. A --label switch would set/update the file system label.

The "superblock" section would hold the following:

The superblock would also contain the sector size for the loop device, which would either be 0 (no concern for raw disk, treated as 512 in loop), 512 ("traditional" and default), 2048 (CDs/DVDs) or 4096 (advanced format, and also the page size). Since the length of remaining data is written to the section header, it would be possible to align each section to sector sizes. (It may lead to performance increases with small blocks and "fast" compressions, but I imagine it would be too low to matter.)

I'm not convinced this is worth the effort. This would require a file system version increment, as old tools would not be able to read the image. It would also increase the complexity without offering much benefit in return. There's a good chance that if you align the start of a section to a sector boundary that the end of the section is also pushed across a sector boundary, so ultimately the same number of sectors have to be read to consume the whole section.

[knmgr]

I would have to know where to write a block, but that is only known when all previous block sizes are known, at which point I can just write everything.

You can output the result to temporary files first, and append the file to the image when it is possible. Of course this makes the compression slower but it avoids the memory waste situation where lots of fast blocks in memory are waiting for a few slow ones.
(This was also one of my justifications for separate block files, but I assume that proposal is already out of the window.)

A superblock is added to the image, but the image isn't padded to the sector size.

With the superblock, at least pad the image to multiples of 512 bytes, otherwise losetup will ignore the last sector, before the FUSE driver can read the image.