Severe rendering degradation (LVGL v8)

[JontyLi0312]

Summary
Severe rendering degradation (LVGL v8) after migrating from Keil AC6 to ATfE

To Reproduce

1. Prepare a project running the official LVGL v8 music demo.
2. Baseline Test: Compile and flash the project using Keil MDK with the AC6 compiler.
   Observe smooth performance.
3. Issue Test: Migrate the build system to CMake and compile the same LVGL source code using the ATfE toolchain linked with picolibc.
4. Flash and run on the target hardware.
5. Compare the FPS and CPU load.

Expected behavior

1. FPS: Approximately 33 ~ 55 FPS.
2. CPU Usage: Approximately 3% ~ 60%.
3. Display: Smooth rendering with no visual artifacts.

Actual behavior

1. FPS: Dropped to 5 ~ 12 FPS.
2. CPU Usage: Increased to 87% ~ 92%.
3. Display: Visible screen flickering and tearing.

Environment

1. ATfE 21.1.1
2. windows 11 pro for workstation 25H2 x86

[Additional context]

1. The project source code logic remains unchanged between the two builds; the primary variables are the compiler and the C library.
2. I suspect the issue might be related to specific compiler optimizations in ATfE or the integration efficiency of picolibc with LVGL's rendering loop.

[voltur01]

Hi, thank you for reporting this!

I assume you refer to this project https://github.com/lvgl/lvgl

To make this issue a bit more concrete and actionable, would you be able to provide command lines used for AC6 and ATfE? For example, target CPU, optimization level, etc.

I believe Keil MDK provides a profiler as well, would it be possible to check the execution profile of the ATfE generated image to at least see if the performance problem is in the compiler generated code (so maybe optimizations) or the picolibc library (a less performant library function called in a loop)?

[JontyLi0312]

MCU: STM32H743IIT6
Both compilers are configured with Optimization Level -O3. Below are the specific flags used for comparison:
ATfE:

set(TARGET_FLAGS "--target=arm-none-eabi -mcpu=cortex-m7  -mfpu=fpv5-d16 -mfloat-abi=hard")

AC6:

-xc -std=c99 --target=arm-arm-none-eabi -mcpu=cortex-m7 -mfpu=fpv5-d16 -mfloat-abi=hard -c -fno-rtti -funsigned-char -fshort-enums -fshort-wchar

Regarding performance analysis: Unfortunately, I am using a CMSIS-DAP debugger, which does not support the Performance Analyzer feature in Keil MDK. Do you have any alternative methods recommended? Or should I perhaps wait until I purchase a supported debugger before seeking further assistance?

[voltur01]

Thank you for the details, let me check internally what we can do about this.

[JontyLi0312]

Looking forward to your further help. Here is the content of my ATfE.cmake file with the detailed build parameters. Hope this is useful.

set(CMAKE_SYSTEM_NAME 		Generic)
set(CMAKE_SYSTEM_PROCESSOR 	arm)

set(CMAKE_C_COMPILER_ID 		Clang)
set(CMAKE_CXX_COMPILER_ID 		Clang)

# ATfE path must be set in here.
set(TOOLCHAIN_PREFIX	"D:/Toolchain/ATfE-21.1.1/bin/")

set(CMAKE_C_COMPILER 	${TOOLCHAIN_PREFIX}/clang.exe)
set(CMAKE_ASM_COMPILER 	${CMAKE_C_COMPILER})
set(CMAKE_CXX_COMPILER 	${TOOLCHAIN_PREFIX}/clang++.exe)
set(CMAKE_LINKER 		${TOOLCHAIN_PREFIX}/clang.exe)
set(CMAKE_OBJCOPY 		${TOOLCHAIN_PREFIX}/llvm-objcopy.exe)
set(CMAKE_SIZE 			${TOOLCHAIN_PREFIX}/llvm-size.exe)

set(CMAKE_EXECUTABLE_SUFFIX_ASM 	".elf")
set(CMAKE_EXECUTABLE_SUFFIX_C 		".elf")
set(CMAKE_EXECUTABLE_SUFFIX_CXX 	".elf")

set(CMAKE_TRY_COMPILE_TARGET_TYPE 	STATIC_LIBRARY)

# MCU specific flags
set(TARGET_FLAGS 	"--target=arm-none-eabi -mcpu=cortex-m7  -mfpu=fpv5-d16 -mfloat-abi=hard")

set(CMAKE_C_FLAGS 	"${CMAKE_C_FLAGS} ${TARGET_FLAGS}")
set(CMAKE_ASM_FLAGS "${CMAKE_C_FLAGS} -x assembler-with-cpp -MP")
set(CMAKE_C_FLAGS	"${CMAKE_C_FLAGS} -Wall -fdata-sections -ffunction-sections")

set(CMAKE_C_FLAGS_DEBUG 	"-O0 -g3")
set(CMAKE_C_FLAGS_RELEASE 	"-O3 -g0")
set(CMAKE_CXX_FLAGS_DEBUG 	"-O0 -g3")
set(CMAKE_CXX_FLAGS_RELEASE "-O3 -g0")

set(CMAKE_CXX_FLAGS			"${CMAKE_C_FLAGS} -fno-rtti -fno-exceptions -fno-threadsafe-statics")

set(CMAKE_CXX_FLAGS "${TARGET_FLAGS}")

if(ATFE_CMAKE_INCLUDED)
  return()
endif()

set(CMAKE_EXE_LINKER_FLAGS 	"${CMAKE_EXE_LINKER_FLAGS} -lcrt0-hosted")
set(CMAKE_EXE_LINKER_FLAGS 	"${CMAKE_EXE_LINKER_FLAGS} -T \"${CMAKE_SOURCE_DIR}/STM32H743XX_FLASH.ld\"")
set(CMAKE_EXE_LINKER_FLAGS 	"${CMAKE_EXE_LINKER_FLAGS} -Wl,-Map=${CMAKE_PROJECT_NAME}.map -Wl,--gc-sections")
set(CMAKE_EXE_LINKER_FLAGS 	"${CMAKE_EXE_LINKER_FLAGS} -z noexecstack")
set(CMAKE_EXE_LINKER_FLAGS 	"${CMAKE_EXE_LINKER_FLAGS} -Wl,--print-memory-usage ")

set(ATFE_CMAKE_INCLUDED TRUE)

[smithp35]

Some initial thoughts. We have limited access to M-profile hardware, and none that has a display, so it may take some time to find out what the root causes are.

There is a config file https://github.com/arm/arm-toolchain/blob/arm-software/arm-software/embedded/Omax.cfg that can be selected with --config=Omax that uses some internal tuning options to more closely match -O3 in Arm Compiler 6. That could be something low cost that is easy to try.

My suspicion, not based on any build or test, is that there will be some library routine on the critical path that is key for performance of this workload. As well as the AC6 libraries being different source code they are also compiled differently. In particular the v7-m libraries are optimised for size [0] rather than performance in ATfE https://github.com/arm/arm-toolchain/blob/arm-software/arm-software/embedded/arm-multilib/json/variants/armv7m_hard_fpv4_sp_d16_unaligned_size.json (see minsizerel). The math libraries used by AC6 are (by default) a bit looser with respect to IEEE 754 conformance than ATfE's math library.

In the future we'd like to factor out the library routines that are most important for performance can be optimised for speed, while maintaining size optimisation for the bulk of the libraries.

It is possible to build your own toolchain with performance optimised libraries by modifying the json files but that is normally a step too far for most.

[0] The performance optimised libraries are too large for other M-profile customers that prefer small code-size.
[1] AC6 uses a proprietary -ffp-mode=std option by default https://developer.arm.com/documentation/101754/0624/armclang-Reference/armclang-Command-line-Options/-ffp-mode?lang=en . There are separate math library variants that are selected when more compliance is needed.
[2] ATfE uses clang's default -ffp-model=precise https://developer.arm.com/documentation/107976/21-1-1/Clang-reference/clang-command-line-options/-ffp-model?lang=en

[JontyLi0312]

When I attempt to add 0max, I encounter the following error:

D:\Toolchain\ATfE-21.1.1\bin\clang.exe -DSTM32H743xx -DUSE_HAL_DRIVER -ID:/STM32/CMake_Project/STM32H743_template_ATfE/drivers/STM32H7xx_HAL_Driver/Inc -ID:/STM32/CMake_Project/STM32H743_template_ATfE/drivers/STM32H7xx_HAL_Driver/Inc/Legacy -ID:/STM32/CMake_Project/STM32H743_template_ATfE/drivers/CMSIS/Device/ST/STM32H7xx/Include -ID:/STM32/CMake_Project/STM32H743_template_ATfE/drivers/CMSIS/Include -ID:/STM32/CMake_Project/STM32H743_template_ATfE/core/Inc --config=Omax --target=arm-none-eabi -mcpu=cortex-m7  -mfpu=fpv5-d16 -mfloat-abi=hard -Wall -fdata-sections -ffunction-sections -O3 -g0 -std=gnu11 -MD -MT drivers/CMakeFiles/drivers.dir/STM32H7xx_HAL_Driver/Src/stm32h7xx_ll_fmc.c.obj -MF drivers\CMakeFiles\drivers.dir\STM32H7xx_HAL_Driver\Src\stm32h7xx_ll_fmc.c.obj.d -o drivers/CMakeFiles/drivers.dir/STM32H7xx_HAL_Driver/Src/stm32h7xx_ll_fmc.c.obj -c D:/STM32/CMake_Project/STM32H743_template_ATfE/drivers/STM32H7xx_HAL_Driver/Src/stm32h7xx_ll_fmc.c
clang: error: configuration file 'Omax' cannot be found
clang: note: was searched for in the directory: D:\Toolchain\ATfE-21.1.1\bin   
ninja: build stopped: subcommand failed.

However, I can confirm that the file exists at the following path:

Additionally, I would like to confirm: is there currently no solution for this issue? Would it be helpful to the team if I provided a performance analysis of my current code?

[voltur01]

Please specify the file extension: --config=Omax.cfg (add .cfg).

Do you mean solution to the overall performance issue? It is too early to say before we know the cause. If you have any kind of comparison of the differences and/or performance profiles of AC6 and ATfE generated code, then yes, it may help to identify the cause.

[JontyLi0312]

After adding ‘--config=Omac.cfg’

set(TARGET_FLAGS "--config=Omax.cfg --target=arm-none-eabi -mcpu=cortex-m7 -mfpu=fpv5-d16 -mfloat-abi=hard"),

there was no change in LVGL's rendering performance. I plan to perform a comparative performance analysis (profiling) on both the current code and the Keil project to help pinpoint the issue.

[smithp35]

I've just had a thought about what could cause such a large difference in performance. I've just remembered that -mfpu=fpv5-d16 has double precision support, whereas the current libraries we ship for M-profile are compiled for Cortex-M4's fpv4-sp-d16 which only has single precision support.

If the example makes substantial use of library functions that use double precision, such as sin (rather than sinf) then the performance of the library functions will be substantially reduced.

What I suggest is to disassemble the ATfE produced binary and look for calls to the software floating point helper functions such as __aeabi_dadd (see https://github.com/ARM-software/abi-aa/blob/main/rtabi32/rtabi32.rst#512the-floating-point-helper-functions). If you see a lot of these, then it is a sign that there is software floating point emulation being used for double precision.

[JontyLi0312]

I disassembled the .elf file using llvm-objdump:

llvm-objdump -d -S .\build\Release\STM32H743IIT6.elf > disassembly.txt

I will attach disassembly.txt at the end. However, upon a quick search, I only found __aeabi_memset and did not find __aeabi_dadd.

disassembly.txt

[smithp35]

Thank you for the disassembly, a couple of observations:

• There is no floating point usage at all that I can see (use of s or d registers). From a brief look at the source it seems like there is an "integer only" mode (CONFIG_LV_USE_FLOAT and LV_USE_FLOAT), so we can rule out the float vs double precision library.
• There is a very small number of C-library functions used, list below:

08000298 <__aeabi_uldivmod>:
080002ac :
080002c0 :
080002d0 :
0804d39c <__udivmoddi4>:
0804d55a <__aeabi_memset8>:
0804d564 :
0804d56c :
0804d57e :
0804d58c <__libc_init_array>:

The majority of the calls to the <string.h> functions go to hand-written lv_ prefixed equivalents so it is looking like the libraries influence on performance will be small. The size optimised library routines above will be much slower than the AC6 equivalents, but, at least statically, it doesn't seem like any of them are on the critical path.

Unless there is some obvious difference in the disassembly such as AC6 and ATfE are somehow using different configurations of the library, it is looking like we'll need to figure out what parts of the program are most performance critical to see what the differences in code-generation are.

[JontyLi0312]

Thank you for the analysis. I plan to perform a performance profiling using a J-Link tomorrow.

Is there anything else I can check in the meantime? Or is it possible that the issue lies not with the compiler or C library, but rather with an oversight during my porting process?

[smithp35]

If you do have the disassembly of the AC6 compiled example I can take a look at that to see if I can spot anything obvious. The outputs of the linker map files (armlink --map --list=armlink_mapfile.txt for AC6 and lld -Map=lld_mapfile.txt) would also be helpful to see which parts of lvl are being used.

Assuming they aren't too big, the executables themselves would also be useful.

Looking at LVGL itself it seems like there is a lv_conf file that is used which sets up a bunch of macros which decide how to build the library. It will be worth checking that has exactly the same options in both builds. I'm guessing some of those values will need to be set to customise the lvl for

[JontyLi0312]

Thank you. I have attached the following files linke(URL: https://github.com/JontyLi0312/Debug) below for your reference:

1. FK743M1.map (generated by AC6)

2. STM32H743IIT6.map (generated by ATfE)

3. My lv_conf.h configuration file