New TF-A causes PANIC booting OP-TEE on Zynq Ultrascale+ ZCU104 board

[frank-kue]

Bug Description

In March 2024 I successfully built and run OP-TEE on an Xilinx Zynq Ultrascale+ ZCU104 board. Building a current version (4.2.0 or 4.3.0 or master) causes the boot to fail with a PANIC:

[...]
D/TC:0 0 call_initcalls:40 level 7 gic_set_primary_done()
I/TC: Primary CPU switching to normal world boot
PANIC at PC : 0x0000000000004188
[...]

This happens after initializing OP-TEE during boot and before running U-boot. During a successful boot (see bug workaround below) the console output at this place looks like:

[...]
D/TC:0 0 call_initcalls:40 level 7 gic_set_primary_done()
I/TC: Primary CPU switching to normal world boot
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x8000000
INFO:    SPSR = 0x3c9


U-Boot 2021.01 (Aug 09 2024 - 11:25:41 +0000)

Model: ZynqMP ZCU104 RevC
Board: Xilinx ZynqMP
[...]

Bug Workaround

The bug workaround is to set the git commit of arm-trusted-firmware to 04013814718e. (I did not try other git commits. This one works for me.)

How to Reproduce

The Dockerfile in the attachment builds two files: BOOT.bin and zynqmp-zcu104.ub which successfully boot from SD-card on the ZCU104 board. For build instructions see the comments in the Dockerfile which follow closely the ones of https://optee.readthedocs.io/en/latest/building/devices/zynqmp.html .

Docker.zip

To reproduce the bug (i.e. build a none-booting BOOT.bin) remove the line

git -C arm-trusted-firmware checkout 04013814718e870261f27256216cd7da3eda6a5d

from the Dockerfile.

[jforissier]

Hi @frank-kue, perhaps your detailed description will ring a bell for someone, but if it doesn't you could always git bisect TF-A to identify the commit that causes the problem.

[jenswi-linaro]

The PANIC is from TF-A. The address looks suspiciously low. Does TF-A panic even if OP-TEE isn't loaded?

[frank-kue]

@jenswi-linaro I am willing to try. Do you know how I can change the OP-TEE build process so that it builds TF-A without loading OP-TEE?

In a different context, I once used bl31.elf from Peta Linux and booted successfully U-Boot and buildroot Linux without OP-TEE. Then I ended up in a Linux without OP-TEE available. But that bl31.elf was most likely not a current TF-A.

[jenswi-linaro]

Removing "NEED_BL32=yes SPD=opteed" might do it.

[frank-kue]

@jenswi-linaro To answer your question: TF-A does not panic if OP-TEE is not loaded.

The boot process loads U-boot and then boots Buildroot Linux. Calling optee_example_hello_world shows that OP-TEE is not there as expected.

The TF-A commit is:

root@container:/optee/build# git -C ../arm-trusted-firmware rev-parse HEAD
e4a37b000fb9a708112da1e06da0e8fad939dc86

The output from the boot process till U-boot startup is:

Zynq MP First Stage Boot Loader 
Release 2024.1   May 19 2024  -  02:40:12
INFO:    BL31: Early console setup
NOTICE:  BL31: Secure code at 0x60000000
NOTICE:  BL31: Non secure code at 0x8000000
NOTICE:  BL31: v2.10.0	(release):xilinx-v2024.1
NOTICE:  BL31: Built : 12:38:58, Aug 16 2024
INFO:    ARM GICv2 driver initialized
INFO:    BL31: Initializing runtime services
INFO:    BL31: PM Service Init Complete: API v1.1
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x8000000
INFO:    SPSR = 0x3c9


U-Boot 2021.01 (Aug 16 2024 - 13:36:59 +0000)

Model: ZynqMP ZCU104 RevC
Board: Xilinx ZynqMP
DRAM:  2 GiB
PMUFW:	v1.1
EL Level:	EL2
Chip ID:	zu7e
NAND:  0 MiB
MMC:   mmc@ff170000: 0
Loading Environment from FAT... *** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Bootmode: LVL_SHFT_SD_MODE1
Reset reason:	EXTERNAL 
Net:   
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr 12, interface rgmii-id
eth0: ethernet@ff0e0000
Hit any key to stop autoboot:  0 
ZynqMP>

[SilviaS]

Following @frank-kue's bug workaround I also managed to get a system firmware image that can boot Linux and OP-TEE for the Xilinx Zynq UltraScale+ ZCU104 board, building version 4.3.0 of OP-TEE.
I noticed that the DS33 LED (PS_LEAD_STATUS) is red instead of green after booting is complete, @frank-kue does this happen on your system too? How can I fix this?

[frank-kue]

@SilviaS Openly, I did no care so much about the LEDs. Unfortunately, I cannot check with the board right now. But yes, I remember some red light but I believe also in other contexts than OP TEE.

[SilviaS]

@SilviaS Openly, I did no care so much about the LEDs. Unfortunately, I cannot check with the board right now. But yes, I remember some red light but I believe also in other contexts than OP TEE.

Many thanks @frank-kue for your reply!