Implementing StructOps Support

[takehaya]

StructOps are used in sched_ext, mptcp, and tcp congestion control. It appears that cilium/ebpf does not support them. Why hasn't this been addressed?

If possible, I would like to work on this. Could you guide me on where to start?

[lmb]

I'm happy to help, the problem is that I don't really know how struct_ops work. Maybe you can explain them to me?

• Which program type do they use?
• How are they attached?

[ti-mo]

Could you please share some error output so we're sure we're talking about the same thing? It's hard to help without any real information.

[shun159]

The following is an error message when loading a program that includes references to .rodata.

2024/10/01 21:59:42 loading objects: field SimpleSched1: map simple_sched1: failed to load struct_ops program: simple_enable cannot load program without loading its whole collection: instruction 0: unsatisfied map reference

However, I may have figured out the cause. It's probably that the program should be loaded from NewCollectionWithOptions(), but I'm not doing that now.

[ti-mo]

It's probably that the program should be loaded from NewCollectionWithOptions(), but I'm not doing that now.

Exactly. 🙂 This is not an error related to StructOps, it goes for all ELFs. See https://ebpf-go.dev/concepts/loader/#loadandassign if you're only interested in a subset of map or program handles.

[shun159]

Thank you!

I will reconsider the implementation a bit while referring to the documentation. (I didn't know there was such a good resource available 😄)

[shun159]

Thanks. For now, I implemented it according to the document and your advise, confirmed that it works, and submitted a draft PR. #1603

[shun159]

@takehaya could you reopen here? I'd like to continue discussing how to proceed.

[shun159]

@ti-mo
let me explanatory slides. The implementation basically based on the content described in the slides. I hope it will be helpful. As for how to split the PRs, let me discuss while also referring to the draft PR. thanks!

https://speakerdeck.com/shun159/introduction-structops

[shun159]

First, we'll set our current goal as enabling linking of .struct_ops.link.
The requirements for supporting StructOps can be broadly categorized into two:

1. ELF reading process
2. Loading process

For 1, the expected scope of work includes classification of structOpsSection, generation of MapSpec, and association of ProgramSpec and MapSpec. Also, we will consider the points you pointed out below. I'm thinking of placing necessary helper functions and StructOpsSpec in internal/struct_ops.

MapSpec and btf.Spec are important pieces of exported API and the bar is high for extending them. Avoid using them for storing arbitrary state that isn't useful for the caller.
Likewise, avoid trivial, overly-specific helpers on btf.Spec. The ones added here are basically struct ops business logic. Conder making them free functions in a separate package, e.g. internal/structops.

Question:
I'm wondering how to pass the state processed by LoadCollectionSpecFromReader() to LoadAndAssign(). While it might be possible to create some mapping by extending CollectionSpec, what do you think about extending CollectionSpec? For example, like adding a map to CollectionSpec to map StructOpsSpec and MapSpec that hold things like a list of pointers to ProgramSpec and btfID, as shown below:

+type StructOpsSpec struct {
+	ProgramSpecs []*ProgramSpec
+	KernFuncOff  []uint32
+	/* e.g. struct tcp_congestion_ops in bpf_prog's btf format */
+	Data []byte
+	/* e.g. struct bpf_struct_ops_tcp_congestion_ops in
+	 *      btf_vmlinux's format.
+	 * struct bpf_struct_ops_tcp_congestion_ops {
+	 *	[... some other kernel fields ...]
+	 *	struct tcp_congestion_ops data;
+	 * }
+	 * kern_vdata-size == sizeof(struct bpf_struct_ops_tcp_congestion_ops)
+	 * bpf_map__init_kern_struct_ops() will populate the "kern_vdata"
+	 * from "data".
+	 */
+	KernVData []byte
+	TypeId    btf.TypeID
+	Btf       *btf.Spec
+}

// CollectionSpec describes a collection.
type CollectionSpec struct {
	Maps     map[string]*MapSpec
	Programs map[string]*ProgramSpec

	// Variables refer to global variables declared in the ELF. They can be read
	// and modified freely before loading the Collection. Modifying them after
	// loading has no effect on a running eBPF program.
	Variables map[string]*VariableSpec

	// Types holds type information about Maps and Programs.
	// Modifications to Types are currently undefined behaviour.
	Types *btf.Spec

	// ByteOrder specifies whether the ELF was compiled for
	// big-endian or little-endian architectures.
	ByteOrder binary.ByteOrder
	
+	StructOpsSpecs map[string]*StructOpsSpec
}

[shun159]

for 2, basically after the program and map are loaded, we expect to need processing that generates kern_vdata and writes kern_vdata as a value to the map when attaching.
Whether to link using AttachRawLink like in the draft PR, or prepare a dedicated function like AttachStructOps and perform kern_vdata writing and Link at that time, can be decided later.
(It might be better to prepare a StructOps-specific Attach function)

[shun159]

I plan to split the PR into at least 4-5 parts. We will add helpers in each PR as needed. (Please let me submit doc/ and example/ in the final PR)

@ti-mo
I'm looking forward to your comment. have a good weekend 😄

[ti-mo]

Hi @shun159, I haven't forgotten, I'll try to allocate some time next week.