Proposal: Run integration tests across multiple kernel versions

[nickgarlis]

It would be helpful to have visibility into how the library behaves across different kernel versions. While I haven’t spotted any bugs so far, some issues might only appear on certain LTS releases, which wouldn’t be caught by the GitHub Actions runners since they only use the kernel packaged with the runners.

One way to address this could be running the integration tests on QEMU across a range of LTS kernel versions. I realize this could make the CI pipeline heavier, so I’m opening this issue to gauge interest. If there’s interest, I’d be happy to look into implementing it.

Some features may only be available in newer kernels, so this could get a bit complex, but it might still be worthwhile. A good starting point could be to have a POC running tests on just the latest kernel version.

What do you think?

[stapelberg]

Thanks for the suggestion, but as you said, kernel-specific bugs haven’t been an issue in this project thus far. Hence, I am somewhat in favor of keeping our test setup simple, also because it runs very quickly currently, and I really like that.

If we wanted to improve test infrastructure, I think splitting up the large nftables_test.go into more manageable pieces would be a more pressing effort :)

[nickgarlis]

Thanks for the suggestion, but as you said, kernel-specific bugs haven’t been an issue in this project thus far.

I was wondering whether that might partly be because we haven’t had much visibility into potential issues yet.

I think we’re on the same page regarding the kind of bugs one might expect, but just to clarify: by kernel-specific bugs I think of cases where the client makes assumptions about what a particular kernel version accepts, rather than discovering kernel bugs themselves.

Hence, I am somewhat in favor of keeping our test setup simple, also because it runs very quickly currently, and I really like that.

Fair point, it's quite nice indeed. What I had in mind wasn’t replacing the current setup, but possibly extending it with something optional, like a nightly workflow.

I am not sure whether my proposal is a good idea TBH. I mainly thought a small POC might (or might not) turn up something interesting, and if it did, whether it would even be considered (without any expectation of merging it).

My motivation here is to increase my confidence in reliably running this tool across a variety of systems.

If we wanted to improve test infrastructure, I think splitting up the large nftables_test.go into more manageable pieces would be a more pressing effort :)

I could have a look when I have some time. Did you have any ideas on how this might look ? Were you thinking of something like <entity>_test.go files on the root of the project ?

[stapelberg]

I was wondering whether that might partly be because we haven’t had much visibility into potential issues yet.

I doubt it. Usually, people would show up asking for workarounds.

I think we’re on the same page regarding the kind of bugs one might expect, but just to clarify: by kernel-specific bugs I think of cases where the client makes assumptions about what a particular kernel version accepts, rather than discovering kernel bugs themselves.

Yes, we’re on the same page.

Fair point, it's quite nice indeed. What I had in mind wasn’t replacing the current setup, but possibly extending it with something optional, like a nightly workflow.

Yeah, I think a nightly run is the only way something like this makes sense.

I also wonder if this is something that would be easier to set up outside of the project / outside of GitHub Actions to begin with, and then only later port it over once the value is proven?

If we wanted to improve test infrastructure, I think splitting up the large nftables_test.go into more manageable pieces would be a more pressing effort :)

I could have a look when I have some time. Did you have any ideas on how this might look ? Were you thinking of something like <entity>_test.go files on the root of the project ?

I haven’t tried out any ideas yet. But I would expect that maybe we’ll need a test helper package to reduce duplication. Grouping tests logically (by <entity>_test.go, as you say) sounds good.