How to correctly deal with bpf_seq_write errors?

[thediveo]

First, many thanks 🙇🏼 (again) for this incredibly useful project that frees me from the horrors of libbpf and distro shared library hell!

I'm struggling with the eBPF Docs on bpf_seq_write:

-EOVERFLOW if an overflow happened: The same object will be tried again.

What actions do I need to take here? The docs make it sound as if there would be some magic retry, but I doubt that's the case. Is the idea that on -EOVERFLOW to immediately return 1 from the iterator eBPF prog and then hopefully things have sorted them out themselves...?

When does -EOVERFLOW actually happen? Can it also happen when hitting some per-write limitation so that retrying would never work but be stupid?

[ti-mo]

👋 Thanks for the question. I'm curious about this myself since it seems un(der)documented as usual, but keep in mind this is purely a bpf-oriented question and not related to ebpf-go.

selftests/bpf/prog_tests/bpf_iter.c:test_overflow and progs/bpf_iter_test_kern4.c are worth giving a read. It contains two comments in particular:

	/* create two maps: bpf program will only do bpf_seq_write
	 * for these two maps. The goal is one map output almost
	 * fills seq_file buffer and then the other will trigger
	 * overflow and needs restart.
	 */

	/* bpf_seq_printf kernel buffer is 8 pages, so one map
	 * bpf_seq_write will mostly fill it, and the other map
	 * will partially fill and then trigger overflow and need
	 * bpf_seq_read restart.
	 */

Note that this conflates bpf_seq_printf and bpf_seq_write. I'm not sure which has the 8-page limit; printf or write? Something to investigate through experimentation or reading more kernel code. I'd expect E2BIG to happen in user space (not EOVERFLOW) if the caller's buffer provided to read() is too small for the output the program wants to generate, and the caller needs to resize the buffer.

My understanding is that:

• multiple bpf_seq_write() calls can be issued from a single bpf program invocation
• the user space read() on the seq_file doesn't return until the bpf program invocation finished
• bpf_seq_write output needs to be buffered somewhere during the call to read()
• there's not much you can do on the bpf side when write() returns EOVERFLOW other than exit with an error

I'm not sure if the kernel immediately returns EOVERFLOW to both callers to read() and bpf_seq_write(), or if the bpf prog needs to return the error code received from write(). Further write calls are pointless in any case; the bpf prog needs to return and a subsequent call to read() will be provided the same context. (task/map/prog/whatever) It also looks like all selftests ignore the write() retcode.

[thediveo]

I was a little hesitant to ask this pure eBPF question, but since this isn't Wikipedia, I'm glad I did! Thank you very much for your helpful response as it helps me taking a closer look on elixir into this area of kernel code. For instance, I missed that the read returns only when the eBPF prog returns, which maps sense considering otherwise random blocking or overflows.

[thediveo]

The conflation seems to be based on that both functions end up writing into the sequence's buffer; now, where is the size of that buffer set? If I'm on the correct track here then bpf_seq_read does the allocation of PAGESIZE<<3, as I haven't yet found a place where the size gets set beforehand. This buffer allocation seems to happen as part of the first read from user space.

[thediveo]

As for the eBPF iterator program, per https://elixir.bootlin.com/linux/v6.14.7/source/kernel/bpf/bpf_iter.c#L718 it is only allowed to return 0 and 1 ... as I independently verified (pun intended) when trying to get my iterator program past the verifier.

In contrast, bpf_seq_write() only ever return 0 or -EOVERFLOW ... and the calling eBPF prog then needs to decide on how to proceed or not.

[ti-mo]

@thediveo Thanks for investigating! Would you be willing to find a suitable spot for this on docs.ebpf.io and send a PR with some clarification?

[thediveo]

@ti-mo yes, as this will most probably benefit us all; I expect to go through multiple iterations of the PR while finding out how to get this information in. I'm wondering if we should provide some information on a dedicated page/article about how the iterator seq architecture works, like how the iterators work in a pull-mode in order to have some flow control and seq reader implementation does kind of stop-and-go batching, where it pulls the eBPF program for as long as there are further elements and the buffer hasn't yet overflow and then waits for user space to drain the buffer completely and only then starting the next batch.