microdds_client not working with real hardware

[Ecuashungo]

Describe the bug

I use a raspberry pi 4 companion computer with ros2 humble and micro_ros_agent installed. When I try to connect it to a pixhawk 4 mini (setup for HITL) through serial (TELEM 1 port on pixhawk, /dev/ttyAMA1 on the raspberry pi) they start the communication, but very quickly abort.
On the companion computer the following message is shown:

pi@cm17:~/ros_ws $ ros2 run micro_ros_agent micro_ros_agent serial --dev /dev/ttyAMA1 -b 921600 -v
[1668776671.744447] info     | TermiosAgentLinux.cpp | init                     | running...             | fd: 3
[1668776671.745096] info     | Root.cpp           | set_verbose_level        | logger setup           | verbose_level: 4
[1668776672.420570] info     | Root.cpp           | create_client            | create                 | client_key: 0xAAAABBBB, session_id: 0x81
[1668776672.420935] info     | SessionManager.hpp | establish_session        | session established    | client_key: 0xAAAABBBB, address: 1
[ros2run]: Segmentation fault

On the px4-side I get the following error after executing microdds_client start -t serial -d /dev/ttyS1 -b 921600 in the nuttshell:

nsh> microdds_client start -t serial -d /dev/ttyS1 -b 921600
nsh> ERROR [microdds_client] create entities failed: participant: 255

To Reproduce

0. Prepare the flight controller for HITL mode as explained here and install micro-ros-agent on the companion computer.
1. Setup the cabling between the companion computer and the pixhawk 4 mini
2. Start HITL environment (pixhawk 4 mini connected to the development computer through USB) as explained here.
3. On the companion computer run ros2 run micro_ros_agent micro_ros_agent serial --dev /dev/ttyAMA1 -b 921600 -v
4. In QGC open the mavlink shell of the pixhawk 4 mini
5. in the mavlink shell run microdds_client start -t serial -d /dev/ttyS1 -b 921600

Expected behavior

When I run this on my development computer in a SITL environment the agent and the client communicate through udp without a problem. In the SITL example I used a modified c++ node (see my pull request) inspired by px4-offboard.

I expect the agent and the client to communicate as they do in SITL.

Log Files and Screenshots

None

Drone (please complete the following information):

I use a Pixhawk 4 mini in HITL mode

Additional context

I am sure that the serial connection between the companion computer works, because I have tried to access the mavlink shell with the mavlink_shell.py script from the companion computer. After that I have disabled Mavlink on TELEM 1 (MAV_0_CONFIG = 0), set the serial baudrate to 921600 (SER_TEL1_BAUD = 921600) and enabled (XRCE_DDS_0_CFG = TELEM1).

[dagar]

I'm not sure if there's any useful information, but you could try launching the agent with more verbosity to see if it has anything helpful (-v4 I think).

Have you tried pixhawk -> PC through serial for comparison? I'll give it a try myself later, but I did verify it during development earlier this year.

[Ecuashungo]

I tried to launch it with -v 6 which is the maximum verbosity and I still get the same error. There are some extra messages, that I don't understand right now:

ros2 run micro_ros_agent micro_ros_agent serial --dev /dev/ttyAMA1 -b 921600 -v 6

[1669023041.369509] info     | TermiosAgentLinux.cpp | init                     | running...             | fd: 3
[1669023041.370166] info     | Root.cpp           | set_verbose_level        | logger setup           | verbose_level: 6
[1669023112.161773] info     | Root.cpp           | create_client            | create                 | client_key: 0xAAAABBBB, session_id: 0x81
[1669023112.161988] info     | SessionManager.hpp | establish_session        | session established    | client_key: 0xAAAABBBB, address: 1
[1669023112.162331] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0xAAAABBBB, len: 19, data: 
0000: 81 00 00 00 04 01 0B 00 00 00 58 52 43 45 01 00 01 0F 00
[1669023112.165235] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0xAAAABBBB, len: 108, data: 
0000: 81 80 00 00 01 05 62 00 00 0A 00 11 01 02 00 00 53 00 00 00 3C 64 64 73 3E 3C 70 61 72 74 69 63
0020: 69 70 61 6E 74 3E 3C 72 74 70 73 3E 3C 6E 61 6D 65 3E 70 78 34 5F 6D 69 63 72 6F 5F 78 72 63 65
0040: 5F 64 64 73 3C 2F 6E 61 6D 65 3E 3C 2F 72 74 70 73 3E 3C 2F 70 61 72 74 69 63 69 70 61 6E 74 3E
0060: 3C 2F 64 64 73 3E 00 00 00 00 00 00
[ros2run]: Segmentation fault

Does the data mean anything to you?

In the meantime I have tried the same setup with a pixhawk 4 on the TELEM 2 port. The same problem persists. Is it correct to set MAV_1_CONFIG to Disabled and the XRCE_DDS_0_CFG to TELEM2?

I will try to do the same with my development computer and post updates here.

[Ecuashungo]

@dagar I have tried to do the same setup with my development computer running ubuntu 20.04 and ROS2 Galactic and I could make it work just fine. Therefore, the error must be somewhere on my companion computer setup.

I have tried to reinstall the micro_ros_agent and during the re-installation there was a warning:

/usr/bin/ld: warning: libfastrtps.so.2.6, needed by /home/pi/ros2_humble/install/rmw_fastrtps_shared_cpp/lib/librmw_fastrtps_shared_cpp.so, 
may conflict with libfastrtps.so.2.8.1
---
Finished <<< micro_ros_agent [5min 50s]

Could this be part of my problem?

[Ecuashungo]

After raising the issue on the agent side, I have managed to make the microdds_client to agent bridge communicate just fine using the Micro XRCE-DDS Agent instead of the micro-ros-agent.

I was able to run an offboard example using HITL and commanding trajectory setpoints from the raspberry pi companion computer. One problem that remains is that I get a warning in the nuttShell saying:
WARN [timesync] RTT too high for timesync: 106 ms

Does this have to do anything with the microdds_client?

[dirksavage88]

@Ecuashungo The microxrcedds agent you installed uses a superbuild cmake to fetch dependencies, it probably would have resolved the libfastrtps dynamic library issue. It looks like you had a newer version of libfastrtps installed.

[ViniciusAbrao]

@Ecuashungo please, how do you start the microdds_client ? is it via mavlink? I am trying to do it but if I work with mavlink I can not simultaneously work with the xrce-dds birdge !!

I tried to open the mavlink console in the QGroundControl, start the client, so I close the QGround and in the sequence I start the Agent. Following it I just have the following:

[1676258724.774920] info | TermiosAgentLinux.cpp | init | running... | fd: 3
[1676258724.775022] info | Root.cpp | set_verbose_level | logger setup | verbose_level: 6

Looks like there is not any ROS topic being published. How do you manage it to see the topics?

[ozitoune]

Hello , i 'm on the same situation. Impossible to see any topics. I'm on ubuntu 22.4 with humble.
I have the xrce client started on pixhawk (i see it on top) and i can run Xrce agent or microros agent on my raspberry pi but i can't see any topics.
Is there any solution to make this work ?
Why they replace the micro rtps in the main branch if the new micro dds is not working with harware ???
I can downgrade px4 firmware because i use a pixhawk 6X only compatible with the 1.13.1 firmware.
pixhawk is connected to my raspberry by usb, can this work or should i connect by telemetrie 2?
Thank for any help

[ViniciusAbrao]

@ozitoune I managed to fix it as following (reference - #21120 (comment)):

if you want to use TELEM1, you have to stop mavlink and then activate the microdds_client. You can do that changing the parameters:
MAV_0_CONFIG = 0
to disable mavlink on TELEM1,
XRCE_DDS_0_CFG = TELEM1
to start microdds_client on TELEM1 and SER_TEL1_BAUD to set the baudrate.

to use TELEM2 (ttyS3 of pixhawk):
XRCE_DDS_0_CFG = TELEM2
SER_TEL2_BAUD = 921600
To check if it is already running, in the QGroundControl's MavlinkConsole: microdds_client status
if not running yet, start the client in the MavlinkConsole: microdds_client start -t serial -d /dev/ttyS3 -b 921600
In raspberrypi: sudo MicroXRCEAgent serial --dev /dev/serial0 -b 921600

make sure your raspberrypi is correct connected to your pixhawk:
connect pixhawk TELEM's tx/rx/ground on raspberrypi correspondent pins
connect pixhawk on 5v (pixhawk's usb on smartphone charger for example)
remember that you also need to configure the raspberry's serial port /dev/ttyAMA0.

It is not possible to configure XRCE_DDS_0_CFG = USB (/dev/ttyACM0 port of laptop) only for the pixhawk's TELEM, so it would not work on laptop via USB.

[ozitoune]

Thank you, you're an angel !!
There is one mouth i try to make this work ... But impossible via usb , i have read, long time ago , usb is allow to development only, so during the developpement phase it would be possible to use ACM0, but apperently no !

I'm a little confused , AMA0 is when we use a fdti/usb adaptator? so if connect telem2 on a fdti adaptator then on a raspberry usb port it could be work ? in a raspbery is /dev/serial0 or /dev/ttyAMA0 ???

The best is to connect directly telem2 to raspbery tx and rx ?

I have a pixhawk 6x so i would like to connect via ethernet but , i dont see where to tell the ip adress on Xrce client / agent, there is -p for the port but how to specify the host ?
Thanks a lot for your work and your help ! For now i can't see the topics, but i think tomorow i will connect telem2 to raspberry and it would better !!

Another question Mavros is still usable ? There is a MAVROS / ROS2 page so perhaps it could be usable if i can't make it work with xrce.

Thanks again !!

[ViniciusAbrao]

Thank you, you're an angel !! There is one mouth i try to make this work ... But impossible via usb , i have read, long time ago , usb is allow to development only, so during the developpement phase it would be possible to use ACM0, but apperently no !

I'm a little confused , AMA0 is when we use a fdti/usb adaptator? so if connect telem2 on a fdti adaptator then on a raspberry usb port it could be work ? in a raspbery is /dev/serial0 or /dev/ttyAMA0 ???

The best is to connect directly telem2 to raspbery tx and rx ?

I have a pixhawk 6x so i would like to connect via ethernet but , i dont see where to tell the ip adress on Xrce client / agent, there is -p for the port but how to specify the host ? Thanks a lot for your work and your help ! For now i can't see the topics, but i think tomorow i will connect telem2 to raspberry and it would better !!

Another question Mavros is still usable ? There is a MAVROS / ROS2 page so perhaps it could be usable if i can't make it work with xrce.

Thanks again !!

@ozitoune you do not need a fdti/usb adapter in order to connect the pixhawk with raspberrypi. You can connect it in the raspberrypi IO pins directly. When you do it, you are working with the AMA0 in the raspberrypi (same as serial0).

I think that mavlink is still working correctly, since you need it for communication in QGroundControl, so should not have problems with mavros.