Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(bpir3): update to kernel 6.9 and add cross compiled configuration #1

Draft
wants to merge 49 commits into
base: main
Choose a base branch
from

Conversation

steveej
Copy link

@steveej steveej commented Jun 2, 2024

No description provided.

@steveej steveej force-pushed the kernel-6.9_and_cross-compile branch from b04407e to a463dfa Compare June 4, 2024 07:20
steveej and others added 26 commits June 5, 2024 13:39
A symlink called result gets dropped into the users current working
directory when certain nix commands are run, for example
"nix build '.#derivation'".

These should never be included, so they should be ignored.

Co-authored-by: Sachi King <[email protected]>
Packages was misspelled, correct that.

How does a misspelling like that happen?  Copy and paste after typing it
once and QWERTY!  Packages is heavily biased to left hand, and contains
a redirect that is easy to get carried away with as it's bouncing back
to the last character hit.
Left hand: A->C->A and you need to hit K between the C and A.

Co-authored-by: Stefan Junker <[email protected]>
Co-authored-by: Sachi King <[email protected]>
When blame is run after a mass reformat everything will show as being
touched by that commit.  This makes finding the commit that changed
something difficult/slow and make it more difficult to understand why
something was added or changed.

This file can be configured in local git and is used by github to avoid
that pitfall making reformats more tenable.
We're a bit of an outlier, building ATF after u-boot, as everything else
builds u-boot after ATF, incorporating ATF blobs into u-boot's output.

This was difficult with MT7986 as all prior art does it the opposite.
The MT7986 does not exist in upstream ATF, so we have to use a vendor
fork, and it's been patched in a way where it wants tools from uboot to
build bl2.bin, so it was easier to just do what it wanted.

This is annoying as working on u-boot is far more likely than hacking on
ATF, but I was too lazy to deal with it then.  With cross-compiling,
dealing with the vendor ATF wanting to call its own in-tree copies of
its tools is a problem.  Rather than hack around that, get our order
right and call fiptool after uboot builds.

Currently this is untested, but fip.bin produced by ATF and u-boot hash
the same, so it should be right.
There's no reason to restrict `nix fmt` to only aarch64 and riscv.
Use every platform in nixpkgs for the formatter closure.
The blacklist method is plural, like the attribute in boot.
Also, why would we want to disable with the enable list, obviously
that's wrong.
If the device being resized is not the first block device, its
partitions may not start with minor 0 for the block device and minor 1
for the first partition.  This results in a bogus partition being
created immedately following the root partition.

`lsblk` has PARTN column, meaning we don't even need to do fragile
parsing of the block device to get this right.  (sda3, vda3, hda3,
nvme0n1p3, mmcblk0p1, etc).
When an extreme number of directories exist in a directory tree being
moved between btrfs subvolumes the metadata usage explodes and we fill
up the extremely limited space we operate in during the first boot,
before resize.

With the release of 24.05, an extreme number of directories exist in the
minimal installation by default as nixpkgs is included in the system
closure.

By creating a snapshot of the top-level subvolume we avoid making a new
directory tree.  Then we remove the files from the top-level and empty
out the subvolume of anything extra, before moving the contents of the
mountpoint up into the new root.

Other methods of attempting to deal with this included copying then
removing each store path one at a time, which failed.  `mv` doesn't
work across filesystem boundaries, which this is.

It was a bit of a hisenbug, because it depends on how large metadata is
during mkfs, and some of my images are 512MB, and thus have more than
enough space, and some are 256MB, which would fail.
The OrangePi5B will require a bit more than setting status = "okay" to get
overlays working, so the simple-enable method isn't going to cut it
anymore.

simple-enable was very strict, and had it's arguments in a hard-coded
order that is difficult to extend without proliferation of tons and tons
of xyz-enable variants.

By working with submoduleWith, an appropriate amount of info can be
passed up into the submodule, and reasonable defaults can be set, letting
the dtOverlay support be fairly flexible.

This is still messy, and there's a lot of duplication between i2c and
rtc.  Additionally UART needs to grow this same functionality.  These
need to be made modular so there's less code duplication so the code can
be more reliably built upon.
Upstream uboot is 2024.4, which doesn't require custom patches anymore,
and by dropping the `DTC=` line cross-compliation works.
v3 referenced v2 incorrectly providing a uboot that loads a more
conservative RAM timing due to v2 hardware issues.
These aren't the hard things to maintain, but it's the low-hanging fruit
and that's a good spot to start I guess.

It might not even stay this way, I'm not sure.  Device is going to get
deduplicated soon, but UART is going to be the device that is going to
challenge all the current work, as it has things that are specific to it
that should actually build on top of everything i2c and RTC has, because
all the enable/disable functionality is missing from UART, but it has a
bunch of its own complexity.
By improving the submodules used to build a device there is no longer a
need for per-type device.nix declaration.

For now this only applies to i2c and rtc as they're the device types
that were the most built out and had the most in common.
It's not boilerplate free, but it is a lot less messy, a lot easier to
follow, and a lot less bug-prone.
With less duplication, brining proper device support to wifi is just par
for the course.

BPiR3 use of WiFi has been updated accordingly, and its assert made more
clear.  Also added a warning to make it really clear that if you don't
have training data it's not going to do what you want.
Upstream kernel, slightly modified OPi5 device-tree for 5b.
No wifi support, as kernel doesn't support it, eMMC works.

No handling of optional hardware pins, yet.
Need a better way to handle the unfree-redistributable so build is not
with --impure to allow unfree.
More features flushed out in the main SBC module, and making an entrance
in the OPi5 config.
OPi5B works.
OPi5 should work, but I don't have access to one of those.
ATFTools in nativeBuildInputs is the last bit required to get
cross-compiling functional.

Co-authored-by: Stefan Junker <[email protected]>
This is going to change, however, this is good enough for now.
I need RTC support to consume this in my main nixos configuration, which
I need to start doing instead of having a completely different and worse
implementation of SBC support in use as opposed to this.
@steveej steveej force-pushed the kernel-6.9_and_cross-compile branch from 033471d to 377c8ff Compare June 23, 2024 16:13
nakato added 2 commits June 24, 2024 14:18
If an NVMe drive is in use, which is one of the key features of this
board, it needs to be usable during stage-1 to be able to be reliably
mounted.
If an installation has been converted to use a tmpfs root, we spend a
ton of seconds waiting for "none" to appear for moutning.  Once
waitDevice times out, we continue and do nothing.

If `rootDevice` is "none", then skip adding the RAMIFY script to initrd.

Devices should support this layout on initial-boot, but currently that
is not the case, so this does not remove functionality at this time.
This should however be corrected.
nakato and others added 10 commits July 5, 2024 14:10
Nixpkgs version is now at 2024.04, which includes the btrfs patch that
was in use here.
It's very hard to maintain these options if they're everywhere.  Use a
generic way to get the repeatable stuff done.
See patch:
https://lore.kernel.org/u-boot/[email protected]/

The issue presents as such during boot:
> BUG at fs/btrfs/extent-io.c:629/free_extent_buffer_internal()!
> BUG!
> resetting ...

Using the reproduction method outlined here, I used a copy of my
filesystem to check this patch against it, and it results in a bootable
system.
https://lore.kernel.org/u-boot/3281192.oiGErgHkdL@youmu/

I had one system do this 5 months ago, and another one do it yesterday.
The first one I wasn't sure if it wasn't a one-off bug.  The timing of
this patch is very fortunate as otherwise I'd be making ext4 the
default.
Unessessary clutter in the top-level makes it hard to see what the flake
gives you, so clean that out before it become unmanageable.  More will
be added to support CI builds and cache pushing.
Getting extra attributes into meta is a pain otherwise, which I may end
up doing depending on the CI build/caching path that is chosen.
Adopt this now rather than fight conflits as I clean the CI work up.
Tihs is a lot smaller after merging 99f773c
@steveej steveej force-pushed the kernel-6.9_and_cross-compile branch from 377c8ff to 9d9ca99 Compare August 8, 2024 08:55
nakato and others added 11 commits August 11, 2024 11:05
The current kernel used by the BPiR3 is basically a "allmodconfig"
kernel, which means we're building way way too much stuff.  Additionally
there's a lot of other configuration we can get away with turning off.

This takes the build-time down from ~8hr on a OrangePi 5B to ~1hr.  On
the actual BananaPi R3, the build time for the former would probably
stretch into a full day.

My end goal here is going to be to do away with the non-minimal kernel
and make this one the default, however both need to exist for a period
to allow a transition.  It should compile in a reasonable amount even
on the R3, but the goal is to do this with CI and push it into a cache.

If a feature is missing or has been explicitly disabled that you want,
feel free to request it is added, the goal is a reasonable kernel build
not an absolute minimum.
This makes a list of filtered derivations to build in CI.  I want it to
fail if build fails, which will signal there's a problem, so I can't let
things fail for being unfree and just keep going as it would not be very
noticeable.  I also have limited compute that's hanging off a slow
internet pipe, so it needs to kick big things, autoModules kernels, out.

This is a monstrosity and was painful and time consuming to write, I
would very much appreciate ways to make this less horrible.
raspberryPiFirmware is a fat filesystem that is treated as a binary blob
that contains raspberrypifw and uboot.
The use of mkForce should not be required to set the kernel outside of
this repo.  It requiring mkForce does make a pause before pointing it to
a broken kernel, but still, it shouldn't be that way.
First pass with primary focus on the BPiR3.

Goal is to document status, such as kernels and patches and what works
and doesn't.

In the long run I want to standarise some testing of various features,
such as what is actually on the expansion bus, do those things work with
upstream, and would they ever work.  I have a board, which is not in
this repository, that looks good on paper until you realise that to
enable X feature on the expansion bus you need to turn off the I2C
interface that's connected to the PMIC, so you effectivly cannot ever
use those pins on the expansion bus or X as it is only available on
those pins.
This adds initial support for the BPiR4.  It uses Frank-W's kernel as
large parts of mt7988a support has not made its way upstream yet.
Frank-W kernel's are very clean, so they're one of the fiew non-upstream
kernels this repo has the appitite to consume.

While the BananaPiR4's intended WiFi addon card will likely load up
without any extra work, it likely needs aditional work to not misbehave.

It is reported that FS resizing is problematic in a manner not caused by
the userspace resize code and that mtk_soc doesn't always load requiring
manual intervention with `rmmod mtk_eth && modprobe mtk_eth`.  As such
support of this SBC should be considered preliminary.

Comimt message edited by:
Sachi King <[email protected]>
The device is very new and the wording should reflect that. Work is
occuring to improve support not only for the chipset but also the
device, and I think this makes that more obvious.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants