EVE Kernel Documentation

This document describes how EVE's Linux kernel is built, configured, and maintained.

TL;DR for Developers

To quickly build a custom kernel for development (example shows disabling MODULE_SIG_FORCE, but you can modify any kernel option):

# Clone the kernel repository
git clone https://github.com/lf-edge/eve-kernel/
cd eve-kernel

# Checkout the branch you want to modify (example for amd64 v6.12.49)
git checkout eve-kernel-amd64-v6.12.49-generic

# Generate base config (see "Kernel Config Flavors" section for other options: rt, hwe)
make eve-core_defconfig

# Modify kernel configuration interactively
make menuconfig
# Example: disable MODULE_SIG_FORCE (or modify any other option you need)
# - Press '/' to search for an option
# - Type the option name (e.g., "MODULE_SIG_FORCE")
# - Press the number shown to jump to the option
# - Toggle to enable/disable it
# - Save and exit

# IMPORTANT: Save as defconfig (do not manually edit .config!)
make savedefconfig
cp defconfig arch/x86/configs/eve-core_defconfig

# Build kernel in container
make -f Makefile.eve kernel-gcc

# Option 1: Build EVE on the same machine using local kernel
cd /path/to/eve
# Note: -s flag is crucial (silent mode) to output only the tag without make noise
make KERNEL_TAG=$(make -C /path/to/eve-kernel -s -f Makefile.eve docker-tag-gcc) pkgs rootfs

# Option 2: Push to DockerHub and build EVE elsewhere
# Push your kernel container to your DockerHub account, then:
cd /path/to/eve
make KERNEL_TAG=<your-dockerhub-tag> pkgs rootfs

Kernel Repository

The EVE kernel is maintained in a separate repository:

Repository: https://github.com/lf-edge/eve-kernel

Historical Note

The kernel was previously built from source within EVE's main repository under pkg/kernel/. As of 2024, the kernel has been moved to its own repository (lf-edge/eve-kernel) and is consumed by EVE as a pre-built container image.

Kernel Versioning and Branches

EVE supports multiple kernel versions across different architectures and platforms. The kernel version selection is handled automatically based on the target architecture (ZARCH) and platform (PLATFORM).

Current Kernel Versions

amd64: v6.12.49 (generic and rt platforms)
arm64:
- v6.1.155 (generic)
- v5.10.192 (nvidia-jp5)
- v5.15.136 (nvidia-jp6)
riscv64: v6.1.112 (generic)

Branch Naming Convention

Kernel branches in the eve-kernel repository follow this naming pattern:

eve-kernel-<ARCH>-<VERSION>-<FLAVOR>

Where <FLAVOR> corresponds to the EVE PLATFORM variable.

Examples:

eve-kernel-amd64-v6.12.49-generic (PLATFORM=generic)
eve-kernel-arm64-v5.10.192-nvidia-jp5 (PLATFORM=nvidia-jp5)
eve-kernel-arm64-v5.15.136-nvidia-jp6 (PLATFORM=nvidia-jp6)
eve-kernel-amd64-next (next LTS development branch)

Important Files

kernel-commits.mk

Location: /kernel-commits.mk (root of EVE repository)

⚠️ Warning: This file is auto-generated by tools/update_kernel_commits.py and should NOT be manually modified. Always use the update script to change kernel commits.

This file contains the specific commit hashes for each kernel branch that EVE uses. Each entry maps a combination of architecture, version, and flavor to a specific commit:

KERNEL_COMMIT_amd64_v6.12.49_generic = ef7ccc4d151c
KERNEL_COMMIT_arm64_v5.10.192_nvidia-jp5 = b4464b03583e
KERNEL_COMMIT_arm64_v6.1.155_generic = ece043d8b403
KERNEL_COMMIT_riscv64_v6.1.112_generic = 18e1d313b90b

Purpose: Pins EVE to specific kernel commits to ensure reproducible builds.

kernel-version.mk

Location: /kernel-version.mk (root of EVE repository)

This file defines the kernel version selection logic based on architecture and platform. It:

Maps platforms to kernel versions
Sets kernel flavors and config flavors
Constructs Docker tags for kernel images
Includes kernel-commits.mk to get commit hashes

Key Variables:

KERNEL_VERSION: The kernel version (e.g., v6.12.49)
KERNEL_FLAVOR: The kernel flavor matching the EVE PLATFORM (e.g., generic, nvidia-jp5, nvidia-jp6)
KERNEL_CONFIG_FLAVOR: Config variant (e.g., core, rt, hwe) - see detailed description below
KERNEL_COMMIT: The specific commit hash from kernel-commits.mk
KERNEL_TAG: The full Docker image tag

tools/update_kernel_commits.py

Location: /tools/update_kernel_commits.py

This Python script automates the process of updating kernel commits in EVE.

Features:

Queries GitHub API to fetch latest commits from eve-kernel branches
Generates updated kernel-commits.mk file
Creates detailed commit messages with changelog
Validates kernel tags against DockerHub
Supports testing mode for dry-run updates

Usage:

# Update all kernel branches
./tools/update_kernel_commits.py

# Update specific branch
./tools/update_kernel_commits.py --branch eve-kernel-amd64-v6.12.49-generic

# Dry run to see what would change
./tools/update_kernel_commits.py --test

# Use custom GitHub token
./tools/update_kernel_commits.py --token <your-github-token>

GitHub Token: The script requires a GitHub personal access token for API access. It will prompt you to create one if not found. The token is stored in ~/.config/eve/github_token.

Branching Strategy

EVE's kernel repository uses a multi-branch strategy:

1. Versioned Branches

These track specific stable kernel versions:

Example: eve-kernel-amd64-v6.12.49-generic
Based on upstream stable kernel releases
Receive backported fixes and EVE-specific patches
Used for production EVE releases

2. Next Branch

The "next" branch is prepared for the next LTS kernel version, not for current development:

Example: eve-kernel-amd64-next
Represents the next planned LTS version (at time of writing, the next LTS version is not yet defined)
All current patches are applied to next as well as stable branches
Used in EVE's evaluation images for continuous testing

Evaluation EVE Strategy: Evaluation EVE has 3 partitions where IMGA and IMGB run the current stable LTS (e.g., v6.12.x) but use different kernel config flavors:

IMGA uses the core flavor
IMGB uses the hwe flavor
The third partition runs the next branch kernel

This setup enables:

Testing of the upcoming LTS kernel alongside stable versions
Continuous validation of patches on the next LTS
Smoke testing of new kernel versions before they become official

LTS Transition Process: When time comes to move to a new LTS:

The floating next branch content is used to create a new versioned branch (e.g., eve-kernel-amd64-v6.13.x-generic)
The old stable version (e.g., v6.12.x) is replaced with this new version
The next branch is rebased on top of the next upcoming LTS
Evaluation EVE continues testing with the new stable + new next

This strategy ensures EVE is always prepared for kernel updates in terms of CI, patch rework, and smoke testing.

3. Branch Selection

The appropriate branch is automatically selected based on:

Architecture (amd64, arm64, riscv64)
Platform (generic, rt, nvidia-jp5, etc.)
Configuration defined in kernel-version.mk

Kernel Config Flavors

⚠️ Important Distinction: KERNEL_FLAVOR and KERNEL_CONFIG_FLAVOR are different concepts and can be confusing:

KERNEL_FLAVOR (also called just FLAVOR in branch names): Corresponds to the EVE PLATFORM variable and determines the kernel branch (e.g., generic, nvidia-jp5, nvidia-jp6). This is part of the branch name like eve-kernel-amd64-v6.12.49-generic.
KERNEL_CONFIG_FLAVOR: Controls which configuration variant is used when building from a specific branch (e.g., core, rt, hwe). This is NOT part of the branch name but determines which defconfig file is used.

The KERNEL_CONFIG_FLAVOR variable controls which kernel configuration variant to use. This allows multiple configurations to be built from the same branch without maintaining separate branches with identical patch sets.

Available Config Flavors

For x86/amd64:

core: Standard EVE kernel configuration with essential drivers
hwe: Hardware Enablement - includes many additional drivers for broader hardware support
rt: Real-time kernel configuration with RT patches for deterministic latency

Usage in EVE:

Evaluation EVE images use three partitions with different kernel configurations:

IMGA: Runs current stable LTS with core flavor (minimal drivers)
IMGB: Runs current stable LTS with hwe flavor (many drivers for broad hardware support)
Third partition: Runs the next branch (preparing for next LTS)

This multi-partition approach allows:

Testing and comparing different kernel configurations on the same hardware
Continuous validation of the next LTS kernel before official adoption
Quick rollback if issues are found with specific flavors

Not all flavors are available on all branches. The flavor determines which defconfig file is used:

arch/x86/configs/eve-core_defconfig
arch/x86/configs/eve-hwe_defconfig
arch/x86/configs/eve-rt_defconfig

Building the Kernel

In eve-kernel Repository

The kernel is built using the Makefile.eve in the eve-kernel repository:

# Build with GCC compiler (standard method)
make -f Makefile.eve kernel-gcc

# Build with specific config flavor
make KERNEL_CONFIG_FLAVOR=rt -f Makefile.eve kernel-gcc
make KERNEL_CONFIG_FLAVOR=hwe -f Makefile.eve kernel-gcc

# Build with LLVM/Clang (experimental, not currently used in production)
make -f Makefile.eve kernel-llvm

# Get the Docker tag that was built
make -f Makefile.eve docker-tag-gcc

Kernel Configuration Files

Kernel configurations are stored in the standard Linux kernel location:

arch/x86/configs/eve-<FLAVOR>_defconfig (for amd64)
arch/arm64/configs/eve-<FLAVOR>_defconfig (for arm64)
arch/riscv/configs/eve-<FLAVOR>_defconfig (for riscv64)

Where <FLAVOR> is determined by the KERNEL_CONFIG_FLAVOR make variable:

eve-core_defconfig: Standard EVE kernel config
eve-rt_defconfig: Real-time kernel config
eve-hwe_defconfig: Hardware enablement config (many drivers enabled)

The asterisk (*) in config paths is controlled by the KERNEL_CONFIG_FLAVOR variable.

Modifying Kernel Configuration

⚠️ Important: Always use the kernel build system to modify configuration. Manual edits to .config can break dependencies.

Step-by-Step Process

Start with base config:

make eve-core_defconfig
# Or for other flavors:
# make eve-rt_defconfig
# make eve-hwe_defconfig

Modify interactively:
```
make menuconfig
```
- Use / to search for options (e.g., search for "MODULE_SIG_FORCE")
- Navigate with arrow keys
- Use space to toggle options
- Save and exit
Example: The TL;DR section shows disabling MODULE_SIG_FORCE as a common development task.
Save as defconfig:
```
make savedefconfig
```
This creates a minimal defconfig file with only the changes from defaults.

Copy to proper location:

# For x86/amd64
cp defconfig arch/x86/configs/eve-core_defconfig

# For arm64
cp defconfig arch/arm64/configs/eve-core_defconfig

# For riscv64
cp defconfig arch/riscv/configs/eve-core_defconfig

Commit and push:

git add arch/*/configs/eve-*_defconfig
git commit -m "kernel: update configuration"
git push

Why Use savedefconfig?

savedefconfig generates a minimal config with only non-default values
Keeps config files small and manageable
Automatically handles option dependencies
Makes config changes easier to review in git diffs
Prevents configuration conflicts

Integrating Kernel Changes into EVE

Option 1: Use Local Build

Build EVE with your locally-built kernel:

cd /path/to/eve
make KERNEL_TAG=$(make -C /path/to/eve-kernel -s -f Makefile.eve docker-tag-gcc) pkgs rootfs

This must be done on the same machine where you built the kernel.

Option 2: Push to Docker Registry

Tag your kernel:

docker tag lfedge/eve-kernel:$(make -f Makefile.eve docker-tag-gcc) \
    yourusername/eve-kernel:custom-tag

Push to DockerHub:

docker push yourusername/eve-kernel:custom-tag

Build EVE with custom kernel:

cd /path/to/eve
make KERNEL_TAG=yourusername/eve-kernel:custom-tag pkgs rootfs

Option 3: Update Official Kernel

Make changes in eve-kernel repository
Push to your fork and create PR

After PR is merged, update EVE:

cd /path/to/eve
./tools/update_kernel_commits.py --branch eve-kernel-amd64-v6.12.49-generic
git add kernel-commits.mk
git commit -F kernel-update-commit-message.txt

Container-Based Build

The kernel is built inside a container to ensure reproducibility and consistency. The build container includes:

Specific toolchain versions (GCC or LLVM)
Required build dependencies
Deterministic build environment

This approach ensures that the same kernel source produces identical binaries regardless of the host system.

Reproducible Builds

EVE kernels are configured for bit-by-bit reproducible builds. This means:

Build timestamps are fixed
Toolchain versions are pinned
Build flags are consistent
Output is deterministic

For more details, see EVE-IMAGE-REPRODUCIBILITY.md.

Kernel Modules and Signing

EVE supports signed kernel modules for enhanced security. By default:

Module signing is enabled in production builds
Modules must be signed with the correct key
MODULE_SIG_FORCE can be disabled for development

To use custom module signing keys in the eve-kernel repository:

Generate your key pair
Place the signing certificate in the appropriate location in the eve-kernel repository
Update the kernel build configuration to use your custom keys

Note: Disabling MODULE_SIG_FORCE (shown in the TL;DR section) is just one example of a common kernel configuration change for development. You may need to modify other options depending on your development needs.

Platform-Specific Kernels

NVIDIA Jetson Platforms

Use vendor-specific kernel versions (JP5: v5.10.192, JP6: v5.15.136)
Include NVIDIA-specific drivers and patches
Branch names include platform identifier (nvidia-jp5, nvidia-jp6)

Real-Time (RT) Kernel

amd64 with PLATFORM=rt uses real-time kernel patches
Uses the KERNEL_CONFIG_FLAVOR=rt configuration
Built from the same branch as generic, but with RT-specific config
Provides deterministic latency for time-critical applications

Generic Platforms

Standard mainline kernel with EVE patches
Used for most hardware platforms
Optimized for virtualization and container workloads

Additional Resources

EVE Kernel Repository: https://github.com/lf-edge/eve-kernel
Linux Kernel Documentation: https://www.kernel.org/doc/
Kernel Newbies: https://kernelnewbies.org/

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EVE Kernel Documentation

TL;DR for Developers

Kernel Repository

Historical Note

Kernel Versioning and Branches

Current Kernel Versions

Branch Naming Convention

Important Files

kernel-commits.mk

kernel-version.mk

tools/update_kernel_commits.py

Branching Strategy

1. Versioned Branches

2. Next Branch

3. Branch Selection

Kernel Config Flavors

Available Config Flavors

Building the Kernel

In eve-kernel Repository

Kernel Configuration Files

Modifying Kernel Configuration

Step-by-Step Process

Why Use savedefconfig?

Integrating Kernel Changes into EVE

Option 1: Use Local Build

Option 2: Push to Docker Registry

Option 3: Update Official Kernel

Container-Based Build

Reproducible Builds

Kernel Modules and Signing

Platform-Specific Kernels

NVIDIA Jetson Platforms

Real-Time (RT) Kernel

Generic Platforms

See Also

Additional Resources

FilesExpand file tree

KERNEL.md

Latest commit

History

KERNEL.md

File metadata and controls

EVE Kernel Documentation

TL;DR for Developers

Kernel Repository

Historical Note

Kernel Versioning and Branches

Current Kernel Versions

Branch Naming Convention

Important Files

kernel-commits.mk

kernel-version.mk

tools/update_kernel_commits.py

Branching Strategy

1. Versioned Branches

2. Next Branch

3. Branch Selection

Kernel Config Flavors

Available Config Flavors

Building the Kernel

In eve-kernel Repository

Kernel Configuration Files

Modifying Kernel Configuration

Step-by-Step Process

Why Use savedefconfig?

Integrating Kernel Changes into EVE

Option 1: Use Local Build

Option 2: Push to Docker Registry

Option 3: Update Official Kernel

Container-Based Build

Reproducible Builds

Kernel Modules and Signing

Platform-Specific Kernels

NVIDIA Jetson Platforms

Real-Time (RT) Kernel

Generic Platforms

See Also

Additional Resources