Issues with 32 -> 64 pointer translation for XDP

[daxzel]

Hello team!

I have tried to rbpf for testing an XDP program.

I use EbpfVmMbuff and providing xdp_md to my program using mbuff

The problem is that xdp_md uses u32 memory references , e.g. data and data_end
bpf.h#L6476

struct xdp_md {
	__u32 data;
	__u32 data_end;
	__u32 data_meta;
	/* Below access go through struct xdp_rxq_info */
	__u32 ingress_ifindex; /* rxq->dev->ifindex */
	__u32 rx_queue_index;  /* rxq->queue_index  */

	__u32 egress_ifindex;  /* txq->dev->ifindex */
};

As I understand in the real kernel those references will be translated to 64 bit references by kernel's verifier when we run in x64. It doesn't happen when I run EbpfVmMbuff so the XDP program has wrong references to the memory. I use an interpreter mode.

Is there a feature in rbpf which I wasn't able to find or XDP programs are not supported?

Please ping me if we need a test example I will try to set it up.

Thanks in advance for the help!

[qmonnet]

Hello team!

Hi there! 👋 The whole team salutes you.

As I understand in the real kernel those references will be translated to 64 bit references by kernel's verifier when we run in x64.

This is correct

It doesn't happen when I run EbpfVmMbuff so the XDP program has wrong references to the memory. I use an interpreter mode.

Is there a feature in rbpf which I wasn't able to find or XDP programs are not supported?

I understand you want to run an XDP program that you compiled for the Linux kernel, in rbpf, right? This is not possible straight away, the translation that you report is precisely the missing bit, as far as I remember.

There's this example with tc: https://github.com/qmonnet/rbpf/blob/main/tests/misc.rs#L73 the program runs nearly unchanged from the ELF, apart for the first three instructions that were patched to adjust the references.

So, no support today. If we had some heuristic that identifies and patches these instructions for TC and/or XDP, we could add a dedicated function in rbpf, though.

[daxzel]

Thank you for a quick response!

I understand you want to run an XDP program that you compiled for the Linux kernel, in rbpf, right?
Yes

We will try to patch rbpf in my code.

We want to explore fixing it upstream in rbpf as well, so please suggest how to better fix it.

I feel like any heuristic we create could false trigger on non-XDP/non-TC program. We potentially need a configuration in EbpfVmMbuff to specify the type of program. We can use heuristic as a warning, though, for better user-experience.

[qmonnet]

I feel like any heuristic we create could false trigger on non-XDP/non-TC program.

Yes, I wouldn't add it as part of the VM execution code. I'd probably have a separate function that could be used to pre-process a program, up for the user to call explicitly before passing their program to the VM?

[daxzel]

I have a created a patcher for the XDP program to run it. It updates the upper bits in place using additional eBPF instructions. It unblocked my usage of RBPF for me: https://gist.github.com/daxzel/96fbea3697bcae1f1c8877504c9d973d

To make it properly, we need to update RBPF code of interpreter, though.

[qmonnet]

It looks nice! I haven't looked at all the code in detail yet, but it comes complete with docs and tests, excellent. I'd be happy to take it in rbpf, probably as a dedicated module (preprocessor?).

How much work would it be to reuse your code to add the TC equivalent next to the XDP patching function, do you know?

To make it properly, we need to update RBPF code of interpreter, though.

You mean adding it to the rbpf crate, right? There's no modification required in the interpreter (or JIT) itself to make this work, if I understand correctly?