feat: kernel|initrd plugins: extend and improve behavior#6102
Open
dilyn-corner wants to merge 100 commits intocanonical:feature/kernel-initrd-pluginfrom
Open
feat: kernel|initrd plugins: extend and improve behavior#6102dilyn-corner wants to merge 100 commits intocanonical:feature/kernel-initrd-pluginfrom
dilyn-corner wants to merge 100 commits intocanonical:feature/kernel-initrd-pluginfrom
Conversation
Signed-off-by: Dilyn Corner <dilyn.corner@canonical.com>
Simplify the syntax a bit to make the tests less fragile Use a simpler defconfig as the base config to improve test speed Signed-off-by: Dilyn Corner <dilyn.corner@canonical.com>
The plugin has been relaxed to no longer explicitly require a specified source for the part. However, we still need to have one. Here, we employ a trick: if there is something which looks like a Linux kernel source tree in CRAFT_PART_SRC, then we behave as a plugin normally would. However, if there is nothing in our CRAFT_PART_SRC, see if the user is building the snap in-tree. If so, treat CRAFT_PROJECT_DIR as CRAFT_PROJECT_SRC. The reason we do this is primarily to save on space and time -- it takes a while even to copy the source from CRAFT_PROJECT_DIR to CRAFT_PART_SRC, and it's even worse when cloning. This isn't an explicitly condoned use-case as it violates standard plugin behavior, but the time savings handling kernel sources in this way offers is worth the behavior-violating trade-off. This is made more palatable by the fact that this behavior isn't necessarily destructive to the host environment anyways (only files in debian*/ are changed), unless the user performs an override-pull or something like it; in these cases, however, we can assume the user knows what they are doing. Signed-off-by: Dilyn Corner <dilyn.corner@canonical.com>
When cross-building, one needs <triplet>-gcc. <triplet>-gcc is available in the archive for all relevant architectures, with the one exception that the x86_64 gcc is not named equivalently to the triplet due to the underscore. Signed-off-by: Dilyn Corner <dilyn.corner@canonical.com>
Make the tests more streamlined and increase speed of kernel compile tests by being more minimal in the config choice. Signed-off-by: Dilyn Corner <dilyn.corner@canonical.com>
In many ways the initrd plugin is more experimental than the kernel plugin, and arguably it should be gated behind the experimental flag for similar reasons as the kernel plugin. Signed-off-by: Dilyn Corner <dilyn.corner@canonical.com>
Signed-off-by: Callahan Kovacs <callahan.kovacs@canonical.com>
…anonical#5978) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This behavior is more properly understood as being part of the pull step, so we mirror what the matter SDK plugin does. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Add support for building Ubuntu kernels by release or by merely consuming the deb. This change additionally adapts the pull command for the plugin to improve its standard behavior to be more in line with standard behavior. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
There are several tools available for the kernel, of which perf is only one. Add support for building additional tools. Currently only bpf, perf, and cpupower are supported. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
dkms functionality makes kernel plugin behavior tricky to handle. dkms isn't intended for this specific use-case anyways. Instead, an alternative path should be documented to assist developers in ensuring their kernel modules are built and packaged correctly.
Users should approach adding ZFS similarly to how any other kernel module package should be added to this snap. There's no reason for the kernel plugin to be opinionated on the provenance or version of the ZFS tooling users select. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
The kernel plugin grew some additional options which create a branch in the ways the snap is built. As a result, much of the code is refactored into separate discrete functions to clarify the process in the binary package versus source package cases. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
THe legacy plugin keeps the prior behavior where the source is fetched during the build step if required, whereas the new plugin performs this fetch during the pull step as expected. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
kernel_ubuntu_release_name must take a value; the plugin passes "None" just in case the user hasn't specified a value, and the build script considers this option when checking the value. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
The base name is of the form coreXX. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
This check isn't valid given the viable option of not specifying a source but instead having CRAFT_PROJECT_DIR be the kernel source provider. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
…r and options Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Coincides with new behavior and option changes. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
This reverts commit 231156b.
Signed-off-by: Callahan Kovacs <callahan.kovacs@canonical.com>
A kernel plugin now exists for legacy core20 bases which covers all current needs addressed by the new plugins. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
pending rationalization Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
The implementation is far easier if we handle it in the build script, and makes slightly more sense (as we'd rather handle it as a stage package, but there's no good way) Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
This implementation makes sanity checking options easier up front. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Specify a base.
If kernel-ubuntu-binary-package is set, KERNEL_SRC never gets used; set a "reasonable" value to placate set -u Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Fix a funny miss Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
No longer needed due to refactor Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Install to binary name, not directory. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
It's slightly different on 22.04, for reasons. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Handled by pull step now. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Some tweaked logic to improve variable usability throughout; may change if we broaden option behavior to work on architectures like riscv. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Building bpf on a core22 base won't necessarily work without some intervention; documentation updated (kind of) to reflect this. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
kdefconfig can be multiple configs. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Tests now reflect new options, mostly build in the expected manner. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Got dropped at some point during a rebase. Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
88eb90f to
699a103
Compare
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
Signed-off-by: Dilyn Corner <dilyn@dilyn.cc>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
These changes update the feature branch kernel and initrd plugins in some key ways.
The initrd plugin changes are largely isolated to migrating some of the root tarball fetch logic into the plugin more properly via a
@get_pull_commands. For the legacy plugin, the prior behavior has been left in tact. The logic in the build script and some general formatting and refactoring have occurred to accommodate these changes. Additionally, a "bug" (behavioral glitch, more like) was resolved by adding an--allow-downgradesflag to anapt installcommand. Otherwise, the behavior is largely the same. The spread tests have been updated to leverage some new kernel plugin options, gaining some tests of new kernel options for free.The kernel plugin changes are a bit more substantive. Firstly, like the initrd plugin, the kernel source fetching logic has moved, in part, to the plugin itself (with analogous updates to the build script logic as a result).
Furthermore, some options from the ubuntu-kernel plugin have been integrated into this plugin. Specifically, the ability to build a snap from a deb, to specify the Ubuntu release of a kernel tree to build from, as well as the ability to build several kernel tools (instead of just perf).
Enabling ZFS support was removed, for what I think are good reasons. Those reasons being...
DKMS support "exists" in the ubuntu-kernel plugin insofar as the plugin builds a deb and leverages the existing kernel build system to build DKMS packages. Attempts were made at emulating this logic within the kernel plugin, but interacting with DKMS felt far too cumbersome (not to mention incredibly fragile) in this setting; DKMS requires a lot of facts to be true of the build environment, and it proved onerous to make the logic work in a consistent fashion.
DKMS is not intended to be used in this fashion anyways; DKMS is intended to be run on runtime targets after a kernel has been updated. If you are building the module at kernel build time (ostensibly what the kernel plugin purports to do), then you can merely build the module in the relevant fashion alongside the kernel in the traditional way; no need to bother with DKMS.
The enable-zfs kernel plugin option demonstrated this fact -- enabling this option effectively ran the compilation commands the DKMS tooling would run, albeit in a far more opinionated fashion than DKMS would.
In lieu of this support, documentation has been created (along with some examples) for how to perform something like an out-of-tree kernel module build within a snapcraft.yaml for kernel snaps. If DKMS is desired, its usage can be spoofed by adding some relevant part and creating the correct file hierarchies within the build environment.
As such, building a kernel snap with ZFS support has been removed; it was too opinionated and prescriptive for what the true intention of the flag was, and to make it more flexible would force the kernel plugin to lose its identity.
The ability to build additional kernel tools (bpftool, perf, cpupower) have been added.
Most critically, the legacy implementations of the plugins have been removed due to the addition of a different kernel plugin (one which many of us are used to) being merged at some point; the implementation in this PR would probably break many existing users of this new plugin, and as such it has been reverted :)
Things to resolve before merging:
kernel-use-binary-packagefor sure) fails in some cases due to RISC-V kernel debs having dumb names. The logic in the script could be improved to resolve this issue, or we can simply make the build fail with an error (checked by the plugin rather than the script).ETA: docs-related tests failing are probably because of something unrelated? The docs tests ran fine (locally) for me previously, I suspect something has happened to e.g. Frame docs because of the RTD move/reorg.
cc @stewarthore @kubiko @LaurenBrock @jicheu
make lint && make test.