syzkaller: improve ceph coverage

[tarasmadan]

Current coverage is 4%.

[tarasmadan]

CC @dubeyko

[dvyukov]

We generate seeds for different fs types here:
https://github.com/google/syzkaller/blob/master/tools/syz-imagegen/imagegen.go
There are no ceph ones. @dubeyko you mentioned something about ceph mounting being more complicated than just having an image. How exactly is ceph mounted?

[dubeyko]

The main complexity takes place in deployment. Usually, it needs to deploy 3-nodes Ceph cluster. From my point of view, the easiest way is to use MicroCeph framework (Ubuntu) [1] for Ceph cluster deployment. Then, the Ceph kernel client can be mounted through kernel client or FUSE client. [1] https://canonical-microceph.readthedocs-hosted.com/stable/

…

On Fri, Dec 12, 2025 at 14:25 Dmitry Vyukov ***@***.***> wrote: *dvyukov* left a comment (google/syzkaller#6505) <#6505 (comment)> We generate seeds for different fs types here: https://github.com/google/syzkaller/blob/master/tools/syz-imagegen/imagegen.go There are no ceph ones. @dubeyko <https://github.com/dubeyko> you mentioned something about ceph mounting being more complicated than just having an image. How exactly is ceph mounted? — Reply to this email directly, view it on GitHub <#6505 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASX7GECHSZ2NBNAAVYQANL4BJGWFAVCNFSM6AAAAACOZZNTJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNBUHE3DSMBXGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[a-nogikh]

I'd say that it would definitely not be a typical setup for syzbot: instead of deploying user-space daemons, we strive to focus on the raw communication with the kernel. The more idiomatic approach would be to teach syzkaller to generate userspace programs that trick the kernel into believing they are ceph client and ceph daemons themselves. Though I admit that that might tricky to do.

If we deploy a full Ceph cluster on a VM, we would need to somehow address the following issues:

1. We'll be de facto also fuzzing the userspace code, so syzkaller also needs to be somehow taught to detect crashes of those applications.
2. Userspace daemons will accumulate a lot of state, which will complicate the generation of reproducers.
3. Overall, to reproduce such a bug locally, the user(s) will have to either do the complex pre-setup themselves or use the specific disk images. This contrasts with our current approach of self-contained reproducers.

(2) can be theoretically addressed by using snapshot-based fuzzing, but I don't really have good ideas re. what to do with the rest.

Maybe we can select some parts of the kernel-side ceph functionality that could be fuzzed in isolation and focus on them.

[dubeyko]

I think I see your point. But let me take a deeper look. I assume that you mean reproducibility. My current vision is based on providing some script that could execute such steps: (1) create some number of virtual drives, (2) download VM image, (3) download kernel sources, (4) build kernel, (5) start VM, (6) deploy Ceph cluster in the VM by means of MicroCeph, for example, (7) mount CephFS through kernel client or FUSE, (8) do fuzzing or reproduce the issue. This sequence looks reasonable to me. And it could be reproducible too. What do you think? Does it sound good?

…

On Sat, Dec 13, 2025 at 18:48 Aleksandr Nogikh ***@***.***> wrote: *a-nogikh* left a comment (google/syzkaller#6505) <#6505 (comment)> I'd say that it would definitely not be a typical setup for syzbot: instead of deploying user-space daemons, we strive to focus on the raw communication with the kernel. The more idiomatic approach would be to teach syzkaller to generate userspace programs that trick the kernel into believing they are ceph client and ceph daemons themselves. Though I admit that that might tricky to do. If we deploy a full Ceph cluster on a VM, we would need to somehow address the following issues: 1. We'll be de facto also fuzzing the userspace code, so syzkaller also needs to be somehow taught to detect crashes of those applications. 2. Userspace daemons will accumulate a lot of state, which will complicate the generation of reproducers. 3. Overall, to reproduce such a bug locally, the user(s) will have to either do the complex pre-setup themselves or use the specific disk images. This contrasts with our current approach of self-contained reproducers. (2) can be theoretically addressed by using snapshot-based fuzzing, but I don't really have good ideas re. what to do with the rest. Maybe we can select some parts of the kernel-side ceph functionality that could be fuzzed in isolation and focus on them. — Reply to this email directly, view it on GitHub <#6505 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASX7GDJW2C5GH65LLV4ZTT4BPOF3AVCNFSM6AAAAACOZZNTJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNBZGE3DKMZVHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dvyukov]

This sequence looks reasonable to me. And it could be reproducible too. What do you think? Does it sound good?

This does not exactly fit into the syzkaller workflow. As a fuzzer syzkaller operates on a level on individual tests. You can think of a test as a C test program started inside of an already running test machine. The test itself is not (and can't) do anything with images/VMs/sources. It needs to be something kselftest-like.

[dubeyko]

I think I don't quite follow what the syzkaller workflow is. Could you please explain it in more detail? So, what is your vision of CephFS testing? How should it run finally? I need to understand how CephFS should fit to the syzkaller workflow. Thanks.

…

On Tue, Dec 16, 2025 at 12:46 AM Dmitry Vyukov ***@***.***> wrote: *dvyukov* left a comment (google/syzkaller#6505) <#6505 (comment)> This sequence looks reasonable to me. And it could be reproducible too. What do you think? Does it sound good? This does not exactly fit into the syzkaller workflow. As a fuzzer syzkaller operates on a level on individual tests. You can think of a test as a C test program started inside of an already running test machine. The test itself is not (and can't) do anything with images/VMs/sources. It needs to be something kselftest-like. — Reply to this email directly, view it on GitHub <#6505 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASX7GFTOPH3PGBJN23JWU34B7BG3AVCNFSM6AAAAACOZZNTJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNJZGQZTIMBRGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dvyukov]

I think I don't quite follow what the syzkaller workflow is. Could you please explain it in more detail?

You can think of it as kselftest. A standalone C program that runs inside of an already booted test machine.

So, what is your vision of CephFS testing? How should it run finally? I need to understand how CephFS should fit to the syzkaller workflow. Thanks.

Can CephFS be tested from a kselftest? If you can show such type of test, we can think how to translate it to syzkaller.

[dubeyko]

As far as I can see, we are stuck in the same loop. :) So, let me start again from the beginning. Any file system needs to be mounted at first. If it's a local file system, then we need to have a storage device or storage device's partition. We must execute the mkfs tool on this partition or storage device itself and we can mount this file system instance. As a result, finally, you can execute kselftests, xfstests, or do any other way of testing. Distributed file systems are slightly different beasts. CephFS hasn't mkfs tool and you cannot mount a local storage device or storage device's partition. CephFS kernel client uses network connection to interact with Ceph cluster. So, if you need to mount the CephFS kernel client, then you must deploy the Ceph cluster first. Because, otherwise, nothing can answer your mount request. So, it needs to deploy the Ceph cluster first. You can have it running in the background or you can deploy it for every test-case. A minimalistic Ceph cluster contains three OSD nodes + three MDS nodes. Object Store Daemon (OSD) is a user-space process that processes read/write requests from clients (CephFS kernel client, for example) to persistent memory. Metadata Server (MDS) is a user-space process that manages metadata structures in DRAM and it requests OSD to store metadata persistently. So, the first step is to start several OSD and MDS nodes that can create quorum and, finally, serve client requests (mount, read, write, etc). So, CephFS mount operation is the request of CephFS client to any running MDS of the existing Ceph cluster. If mount operation was successful, then we can do kselftests, xfstests, or any other testing of CephFS kernel client. And this is why I mentioned Ceph cluster deployment several times already. I hope it helps. Have I clarified my point and answered the questions?

…

On Tue, Dec 16, 2025 at 8:13 PM Dmitry Vyukov ***@***.***> wrote: *dvyukov* left a comment (google/syzkaller#6505) <#6505 (comment)> I think I don't quite follow what the syzkaller workflow is. Could you please explain it in more detail? You can think of it as kselftest. A standalone C program that runs inside of an already booted test machine. So, what is your vision of CephFS testing? How should it run finally? I need to understand how CephFS should fit to the syzkaller workflow. Thanks. Can CephFS be tested from a kselftest? If you can show such type of test, we can think how to translate it to syzkaller. — Reply to this email directly, view it on GitHub <#6505 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASX7GGRWRCDXGI4UC2HPRL4CDJ6XAVCNFSM6AAAAACOZZNTJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNRTGU2TONBSHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dvyukov]

CephFS kernel client uses network connection to interact with Ceph cluster.

Is it a single network connection? If yes, it looks similar to FUSE. We test FUSE by mimicking fuse server inside of the same test process. The fuzzer generates responses on that connection.

[dubeyko]

Usually, any Ceph cluster provides several network connections with the goal to manage network partitioning, guarantee reliability, etc. But, as far as I can see, we can use the single network connection for fuzzing the CephFS kernel client. CephFS can be mounted by means of the FUSE interface. So, mimicking a fuse server can provide a way of testing CephFS through the FUSE infrastructure. I am guessing, how complicated could be the mimicking of regular Ceph cluster's servers for CephFS kernel client? What should be done for this?

…

On Thu, Dec 18, 2025 at 12:13 AM Dmitry Vyukov ***@***.***> wrote: *dvyukov* left a comment (google/syzkaller#6505) <#6505 (comment)> CephFS kernel client uses network connection to interact with Ceph cluster. Is it a single network connection? If yes, it looks similar to FUSE. We test FUSE by mimicking fuse server inside of the same test process. The fuzzer generates responses on that connection. — Reply to this email directly, view it on GitHub <#6505 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASX7GBVOIUNFYXTVX4LXZL4CJO3TAVCNFSM6AAAAACOZZNTJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNRYHE2TSMBVGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dvyukov]

I am guessing, how complicated could be the mimicking of regular Ceph cluster's servers for CephFS kernel client?

You know better what's the protocol there :)

What should be done for this?

I am thinking:

• add a pseudo-syscall here
• the pseudo-syscall would create a server socket, start a thread to receive requests and send replies, call mount passing the server address
• the pseudo-syscall may also accept a set of random server replies from the fuzzer engine (alternatively replies may be passed via a separate pseuso-syscall)
• set of syzlang descriptions for binary format of server replies