Firmware change initial_turbo=60 causes kernel hangs when booting from USB

[agners]

Describe the bug
Running firmwares from 4f43ba2 or newer cause Home Assistant OS 15.1.dev20250326 (using Raspberry Pi Kernel 6.6.74) to hang on I/O with hung tasks errors (presumably USB bus stops working).

Note: I am one of the developer of Home Assistant OS. I report this here as the error has been pinpointed to the firmware update, and bisected to the offending firmware commit: The firmware before 4f43ba2 works reliable, the firmware at 4f43ba2 doesn't work.

I've also tried to revert the change by setting initial_turbo=0 (presumably the previous default) in config.txt manually. However, this does not seem to resolve the issue.

Are there other changes in that particular firmware update which might be responsible for this behavior change?

Sidenote: Besides the above issue we have also noticed that some SD card stopped working in U-Boot (timeouts communicating with the SD card, see home-assistant/operating-system#3965). That seemed to be resolved by initial_turbo=0. Now this might be a bug in U-Boot, and not related what I am reporting here, just thought I mention that this firmware update seems to have side effects.

To reproduce
Install Home Assistant OS 15.1 on a USB flash drive (by default shipped with firmware binaries from 1.20250305 tag).

Expected behaviour
No hangs.

Actual behaviour
Kernel hang on I/O.

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

• Which model of Raspberry Pi? Pi3B+
• Which OS and version (cat /etc/rpi-issue)? Home Assistant OS 15.1
• Which firmware version (vcgencmd version)? See above
• Which kernel version (uname -a)? Linux ha-shelf1-rpi3plus-usb 6.6.74-haos-raspi #1 SMP PREEMPT Wed Mar 26 18:49:29 UTC 2025 aarch64 HAOS

Logs
If applicable, add the relevant output from dmesg or similar.

Additional context
Add any other relevant context for the problem.

[timg236]

Does 'force_turbo=1' work ?

Please could you try this on RPi OS to help us determine whether it's just software or software + hardware?

[popcornmix]

I've also tried to revert the change by setting initial_turbo=0 (presumably the previous default) in config.txt manually. However, this does not seem to resolve the issue.

Checking the code, initial_turbo=0 should completely undo the change, your result is surprising.
And an additional check, can you go to the latest working firmware (just before the initial_turbo=60 commit) and confirm if adding initial_turbo=60 breaks it.

[agners]

FWIW, this is the kernel boot log firmware information:

Non-working, from 4f43ba2.

# dmesg | grep firmware
[    0.052936] raspberrypi-firmware soc:firmware: Attached to firmware from 2024-11-11T15:50:32, variant start
[    0.053947] raspberrypi-firmware soc:firmware: Firmware hash is 903570ba72a9e117f92e5499de439f59dd96e417

Working, from 3b48b3b

# dmesg | grep firmware
[    0.026331] raspberrypi-firmware soc:firmware: Attached to firmware from 2024-10-17T11:35:59, variant start
[    0.027334] raspberrypi-firmware soc:firmware: Firmware hash is b580e2acde306434ab07a913745a21451643ff55

Testing now with initial_turbo=60 on the latter firmware.

[agners]

So with the latter firmware (from 3b48b3b) and initial_turbo=60 the hang happens too, so it seems to be related to initial_turbo then.

Does 'force_turbo=1' work ?

I've tried that on the non-working firmware (from 4f43ba2), and it seems to hang after a short while as well.

Please could you try this on RPi OS to help us determine whether it's just software or software + hardware?

That setup worked for about 2 years, in my closet as a test setup, and it definitely is related to the mentioned firmware update. But if necessary, I can try to reproduce it on RPi OS as well.

[agners]

Just some thoughts: Since config.txt is stored on the USB flash drive, there must some thing happen with the new default (with turbo enabled). Presumably, USB initialization is now run with a higher clock by default. Could it be that this influences USB host controller setup such that it has a lasting negative effect? 🤔

I guess that wouldn't explain why old firmware + initial_turbo=60 also makes things fail 🤷

[agners]

I was able to reproduce the problem on the same device with a different USB SSD, in this case a Samsung Portable SSD T1. Same thing with Home Assistant, shortly after boot with the new firmware or the old firmware plug initial_turbo the device locks up.

I then moved on to Raspberry Pi OS (64-bit), and it didn't manage to complete booting. The system did hang during later bootup stage, similar to Home Assistant OS.

I then replaced the firmware files with the older ones, and bootup completed successful. So on my Raspberry Pi 3+, this is definitely also reproducible with Raspberry Pi OS.

[agners]

I then moved on to Raspberry Pi OS (64-bit), and it didn't manage to complete booting. The system did hang during later bootup stage, similar to Home Assistant OS.

I've looked into this again today, it seems that Raspberry Pi OS initially comes with an older version, specifically:

[    0.072482] raspberrypi-firmware soc:firmware: Attached to firmware from 2024-08-30T19:19:11, variant start
[    0.076475] raspberrypi-firmware soc:firmware: Firmware hash is 2808975b80149bbfe86844655fe45c7de66fc078

Today however, that firmware with Raspberry Pi OS behaved just fine 🤔 🤷

I then updated to the problematic firmware:

[    0.052936] raspberrypi-firmware soc:firmware: Attached to firmware from 2024-11-11T15:50:32, variant start
[    0.053947] raspberrypi-firmware soc:firmware: Firmware hash is 903570ba72a9e117f92e5499de439f59dd96e417

First things appeared to be stable for a while, but by stressing I/O a bit (downloading a Docker container) the Raspberry Pi OS froze twice here.

Does 'force_turbo=1' work ?

With that, HAOS this seems to freeze already in U-Boot.

FWIW, I also switched the power supply, but that didn't seem to make things more stable.