feat: support JIT in no_std mode

[Godones]

In order to allow the use of jit in no_std mode, it is necessary to let the user manually pass in the executable memory area.

[qmonnet]

Thank you! It's great to gain support for the JIT without std. I'm in favour of taking this work (again, apologies for the delay!), but I've got some concerns with the current version.

• I'm not convinced we need the second commit (helpers batch registration), it exposes the internal structure we use to hold the helper functions, for little gain in my opinion.

• I don't fully understand the motivation for commit 3. What are the ranges trying to solve, here?

  Since some maps may store a lot of data, representing the address with a
  single byte may waste a lot of space. Using range can simplify this
  problem.

  We don't support maps yet. And what do you mean by “a single byte” exactly?

• Any chance you could reorder the commits to avoid breaking the tests between commits 3 and 4? By folding the last commit into the previous one, or swapping them if it makes sense. I don't like broken tests between commits, it's a pain when we need to bisect to find a bug.

• We need to update the tests, too, so we can be sure that the JIT behaves as expected. The good news is, it's easy to remove the feature = "std" in tests/ubpf_jit_x86_64.rs and test/misc.rs. Bad news: we need to add macros and use the alternate version of the jit_compile() function everywhere.

Regarding this last point: Would it make sense to maybe change your API? Instead of having a jit_compile() function with different arguments when std is disabled, how about adding a separate set_jit_exec_memory() to add the memory area as an attribute to the VM, and use it in jit_compile() without having to pass it at compilation time? Makes one more function call when we don't have std, but it avoids making different versions of the function, and look slightly cleaner to me? We'd still have to update all the JIT tests though.

[Godones]

For commit 2, suppose we have an array data type as follows:

struct Array{
    data: [u8;512]
}

The size of this array is 512 bytes. If we follow the original method, we need to store 512 u64 types, which requires 512*8 space. Using Range, only 8*2 space is required.

[Godones]

It is a bit difficult to test the JIT in no_std environment, because executable memory allocation is usually kernel-specific. I don't think the lack of this test will affect the use of the JIT in no_std mode.

[qmonnet]

The size of this array is 512 bytes. If we follow the original method, we need to store 512 u64 types, which requires 5128 space. Using Range, only 82 space is required.

OK I looked into it again, and it makes sense now. The mention of “maps” in the commit description is a bit confusing, but fine.
@wdullaer Is the API change on register_allowed_memory() in this PR an issue for you?

As for the tests:

I don't think the lack of this test will affect the use of the JIT in no_std mode.

I don't think either. But then, the number of times I thought my code would work and it didn't! 🙂 No, if we support the JIT without std, we need to have proper tests to check that it does work, and to catch regressions. We have a nice suite of tests in rbpf, would be a shame not to use them to test the different features.

It is a bit difficult to test the JIT in no_std environment, because executable memory allocation is usually kernel-specific.

What does this mean exactly? Is it too hard to put in a wrapper and make it work with the different CI runners? What do you mean by “kernel-specific”, are there more cases than Linux/macOS and Windows?

[Godones]

Maybe we have different understandings of no_std mode, but it doesn’t matter. Now we allow running jit tests in --no-default-features mode.

[wdullaer]

The API change looks good to me.

[qmonnet]

Maybe we have different understandings of no_std mode, but it doesn’t matter.

What do you mean? (Genuine question, I may be wrong and would like to understand better.)

Now we allow running jit tests in --no-default-features mode.

Thanks a lot for this!

[qmonnet]

Looks all good to me. Thanks for this!

[qmonnet]

I was about to merge but noticed the CI failures, which seem unrelated and should be addressed in #126.

Would you mind rebasing so we can run the full tests with --no-default-features, please?

[Godones]

I was about to merge but noticed the CI failures, which seem unrelated and should be addressed in #126.

Would you mind rebasing so we can run the full tests with --no-default-features, please?

No problem.

[Godones]

Maybe we have different understandings of no_std mode, but it doesn’t matter.

What do you mean? (Genuine question, I may be wrong and would like to understand better.)

Now we allow running jit tests in --no-default-features mode.

Thanks a lot for this!

I think the no_std mode is generally suited for typical kernel scenarios, such as the Linux kernel or other embedded systems. In these cases, no_std means we can't use the standard library or rely on libc support. However, from a literal perspective, no_std simply means that we cannot use the standard library or any libraries that depend on it. So, we can still use libc to complete the test in user mode.

[qmonnet]

However, from a literal perspective, no_std simply means that we cannot use the standard library or any libraries that depend on it. So, we can still use libc to complete the test in user mode.

We can, but if we never check that things work without the libc, how do we know that it's working? I'm confused.

Anyway, thanks a lot for this work! Ready to merge, now.