How to "best" communicate cpumasks from an iterator eBPF prog to user space?

[thediveo]

First, many thanks 🙇🏼 (again) for this incredibly useful project that frees me from the horrors of libbpf and distro shared library hell!

While working with eBPF task iterators to see if they improve things in term of performance but also data fidelity way beyond procfs I'm wondering what a good or even "The Best" way of passing CPU masks from an eBPF iterator into user space?

I've seen the various kfuncs regarding cpu masks, but haven't come yet across a canonical example how to deal with sending cpu masks to user space in an iterator, and outside some eBPF map for eBPF kernel/userspace communication. In my cases I will need to send other data also alongside the cpu mask as per iterator iteration, so I would like to do this in a single bpf_seq_write, not least due to error handling in case bpf_seq_write returns an error and I would need to retry the same iteration later.

Now, struct cpumask is peculiar in that its size depends on how a particular kernel was built. In case of say, a stock Ubuntu 25.04 kernel the NR_CPUS is CONFIG_NR_CPUS defined as 8192. However, other kernels might be built with a different setup.

Do I basically need to build my eBPF iterator program to some size I deem sufficient for my user base, such as 8192 and copy the cpumask into my fixed size array, zeroing any surplus elements? Is there a better way to handle the situation?

What are the limitations of bpf_seq_write for the kernel-userspace communication from iterators?

[ti-mo]

In my cases I will need to send other data also alongside the cpu mask as per iterator iteration, so I would like to do this in a single bpf_seq_write, not least due to error handling in case bpf_seq_write returns an error and I would need to retry the same iteration later.

I haven't used bpf_seq_write in anger, but my understanding is it works just like perf and ringbuf maps, so from my point of view, the same concepts apply. Push data from bpf to user space in whatever format you like. In practice, this often means you either send fixed-length messages using a shared struct definition between Go/C to keep the decoding operation simple. Or, you devise your own little TLV-based protocol, or literally anything else you can come up with that best fits your use case.

Now, struct cpumask is peculiar in that its size depends on how a particular kernel was built. In case of say, a stock Ubuntu 25.04 kernel the NR_CPUS is CONFIG_NR_CPUS defined as 8192. However, other kernels might be built with a different setup.
Do I basically need to build my eBPF iterator program to some size I deem sufficient for my user base, such as 8192 and copy the cpumask into my fixed size array, zeroing any surplus elements? Is there a better way to handle the situation?

https://nakryiko.com/posts/bpf-core-reference-guide/#reading-bitfields-and-integers-of-varying-sizes may be of interest here. You'll want to lean on CO-RE as much as you can. See https://docs.ebpf.io/concepts/core and related pages (there are doc pages for individual CO-RE macros), the post on Andrii's blog hasn't really evolved over time iirc.

What are the limitations of bpf_seq_write for the kernel-userspace communication from iterators?

Depends what you mean by limitations; from bpf side it's almost exactly like writing to a file or socket in a typical user space application. You can make multiple writes per read() in user space, so you can combine input from multiple sources, for example a header you allocated on the bpf stack, followed by the bitmask, followed by some more data you've generated in the program.

Instead of plumbing through the whole cpumask to user space, I'd suggest putting as much logic into the bpf program itself. (I assume there's some info you want to extract from the cpumask) Only communicate the result of the computation to user space to avoid copying and decoding as much as possible.

[thediveo]

Ah, Andrii Nakryiko's Blog has been already very helpful to me; there's no way around CO-RE when dabbling with struct task_struct. I'm still revisiting this blog post in order to grok more of the finer and finest details. But I've found that it works quite beautifully for me already, the only stumbling stone being when to use CORE_READ, or even CORE_READ_USER, the latter when calling eBPF programs from user space as part of unit testing. Not least, my own affinity Go module helps me with pinning the eBPF program execution to the correct CPU with the testing thread...

In my case I actually need to plumb through to some extend, as this information is evaluated in multiple different contexts (maybe over time I'll have more specialized eBPF progs, but at the moment I already have some consuming applications on the existing layer that I want to speed up using the eBPF task iterator).

With your feedback I'm now seriously pondering to use the "cpu list" format for this instead, as this is what I'm already doing in the upper layers and in displaying to the user. The systems my software will be run on hopefully doesn't have sparse cpu masks with every second CPU offline or so, so in most cases it will be single/dual logical CPU assignments, as well as housekeeping CPUs doing "kettle" business on the workload.