[Bug] Hardfault on zenoh with FMU-v6x

[MDEAGEWT]

Describe the bug

I am encountering a Hardfault immediately when executing zenoh start on the PX4 shell. This occurs specifically when a Zenoh router (rmw_zenohd) is running on the companion computer (Jetson Orin).

I have previously submitted PR #26213 to address what I believed to be the cause of this issue. However, I am opening this issue to provide detailed logs (hardfault log) and to ask if there are alternative solutions or a recommended approach to resolve this, other than my proposed PR.

To Reproduce

Environment:

Flight Controller: FMU-v6x
Companion Computer: Jetson Orin
PX4 Firmware: main branch (Commit: 435040f)
Build Command: make px4_fmu-v6x_zenoh

Companion Computer Setup:
Run the Zenoh router: ros2 run rmw_zenoh_cpp rmw_zenohd

PX4 Console:
Execute: zenoh start

Result:
The system crashes immediately with a Hardfault.

fault_2000_01_01_00_13_08.log

Flight Log / Additional Information

Version:

main branch (Commit: 435040f)

Hardware:

Jetson Orin Nano, fmu-v6x

[PetervdPerk-NXP]

@MDEAGEWT Could you include the elf file, so we can decode the fault log.
Also on the companion computer which version of ROS2 you're running?
Most of the testing has been done on the latest rmw_zenohd for Jazzy (note that earlier versions could cause crashes).
Also before running zenoh start, could you share the memory status on the V6X by sharing free output?

The need of CONFIG_PTHREAD_MUTEX_TYPES=y is correct, this was changed with the zenoh-pico library but I've missed upgrading the V6X target.

[MDEAGEWT]

@PetervdPerk-NXP Thank you for looking into this.

I have tested this on both ROS 2 Humble and ROS 2 Kilted, but the issue persists in both environments.

Here is the free, top command output and elf file:

px4_fmu-v6x_zenoh.zip

 PID COMMAND                   CPU(ms) CPU(%)  USED/STACK PRIO(BASE) STATE FD
   0 Idle Task                   25072 65.541   180/  768   0 (  0)  READY  3
   1 hpwork                          0  0.000   292/ 1224 249 (249)  w:sem  3
   2 lpwork                          0  0.000   292/ 1576  50 ( 50)  w:sem  3
   3 nsh_main                        0  0.000  2252/ 3144 100 (100)  w:sem  4
   4 wq:manager                      0  0.000   604/ 1232 255 (255)  w:sem  3
   5 wq:lp_default                   0  0.173  1340/ 3472 205 (205)  w:sem  3
   6 Telnet daemon                   0  0.000   532/ 1984 100 (100)  w:sem  1
   7 netinit                         0  0.000   580/ 2024  49 ( 49)  w:sem  4
1497 log_writer_file                 0  0.000   372/ 1144  60 ( 60)  w:sem  3
  56 wq:hp_default                   1  0.823  1132/ 2776 237 (237)  READY  3
 158 dataman                         0  0.005  1068/ 1376  90 ( 90)  w:sem  5
 236 wq:I2C0                         0  0.000   660/ 2312 247 (247)  w:sem  3
 250 wq:SPI3                         6  3.224  1720/ 2368 251 (251)  w:sem  3
 253 wq:SPI2                         5  2.776  1720/ 2368 252 (252)  w:sem  3
 265 wq:SPI1                         5  2.896  1720/ 2368 253 (253)  w:sem  3
 283 wq:I2C4                         0  0.245   932/ 2312 243 (243)  w:sem  3
 305 wq:I2C2                         0  0.161   752/ 2312 245 (245)  w:sem  3
 466 wq:nav_and_controllers          9  4.805  1244/ 2216 242 (242)  w:sem  3
 467 wq:rate_ctrl                    8  4.018  2252/ 3120 255 (255)  w:sem  3
 468 wq:INS0                        13  6.484  3860/ 5976 241 (241)  w:sem  3
 469 wq:INS1                         2  1.151  1268/ 5976 240 (240)  w:sem  3
 470 wq:INS2                         2  1.119  1268/ 5976 239 (239)  w:sem  3
 494 wq:ttyS5                        0  0.204  1044/ 1704 229 (229)  w:sem  3
 525 commander                       1  0.505  1536/ 3192 140 (140)  READY  5
 810 gps                             0  0.029  1004/ 1936 205 (205)  w:sem  4
1023 mavlink_if0                     2  1.067  1668/ 3048 100 (100)  w:sig  5
1032 mavlink_rcv_if0                 0  0.197  2336/ 6136 175 (175)  w:sem  5
1163 mavlink_if1                     5  2.960  1652/ 3048 100 (100)  w:sig  5
1167 mavlink_rcv_if1                 0  0.154  2452/ 6136 175 (175)  w:sem  5
1207 navigator                       0  0.095  1568/ 2168 105 (105)  w:sem 11
1456 logger                          0  0.185  3028/ 3608 230 (230)  READY  3
1570 top                             0  0.025  2108/ 4056 237 (237)  RUN    3

Processes: 32 total, 5 running, 27 sleeping
CPU usage: 33.31% tasks, 1.15% sched, 65.54% idle
DMA Memory: 5120 total, 1024 used 1536 peak
Uptime: 37.324s total, 25.072s idle

                   total       used       free    largest  nused  nfree
        Umem:     907136     260560     646576     416064   1109     42
        Prog:    2097184    1760896     336288     250304  55028  10509

[MDEAGEWT]

As the issue is now resolved, I will close this issue given the inactivity over the past few days. Please don't hesitate to ask if you require any more information.