`communication_constraints_violation` with Ubuntu PREEMPT_RT kernel

[christian-rauch]

I am using libfranka 0.9.2 on Ubuntu 24.04 with kernel 6.8.1-1015-realtime, installed via sudo apt install ubuntu-realtime. The robot is connected via a dedicated ethernet card.

Still, I am frequently getting the communication_constraints_violation error: libfranka: Move command aborted: motion aborted by reflex! ["communication_constraints_violation"].

I previously used the XanMod kernel 6.12.49-rt-x64v3-xanmod1, installed via sudo apt install linux-xanmod-rt-x64v3. The XanMod kernel runs much smother with communication_constraints_violation occurring very rare. However, for some reason, the /sys/kernel/realtime file does not exist, so I am getting the warning [FrankaHardwareInterface]: You are not using a real-time kernel. Using a real-time kernel is strongly recommended! with the ROS node.

Does someone have experienced similar issues with libfranka and the Ubuntu realtime kernel and know how to avoid this?

[mikelasa]

I'm facing the same exact issue with the robot, I took some measures, executing cartesian impedance example:
In my case is a 6.8.1-1031-realtime in ubuntu 24.04

====== update() timing after 10.0007 seconds ======
sendRobotCommand: median=1 us, min=0 us, max=11 us, stddev=0.481053 us
receiveRobotState: median=994 us, min=1 us, max=4937 us, stddev=192.446 us
convertRobotState: median=0 us, min=0 us, max=7 us, stddev=0.090047 us
logger_.log: median=0 us, min=0 us, max=2 us, stddev=0.0300168 us
TOTAL update(): median=998 us, min=4 us, max=4940 us, stddev=192.425 us

====== update() timing after 10.001 seconds ======
sendRobotCommand: median=1 us, min=1 us, max=8 us, stddev=0.40089 us
receiveRobotState: median=994 us, min=1 us, max=4993 us, stddev=211.089 us
convertRobotState: median=0 us, min=0 us, max=3 us, stddev=0.0458883 us
logger_.log: median=0 us, min=0 us, max=1 us, stddev=0.0141641 us
TOTAL update(): median=998 us, min=4 us, max=4997 us, stddev=211.071 us 

@christian-rauch have you found the issue¿

[hoo223]

In my case, I checked the ping test with Franka Control IP (e.g., 172.16.0.2) and it showed more than 1ms. Finally, I found that the issue was related to my PC’s LAN port. After changing the connection from the LAN port to a USB-C port (with an Ethernet-to-USB-C adapter), the average ping dropped to less than 1ms and the problem was solved.

[christian-rauch]

I am using a Aquantia Corp. AQtion AQC113 NBase-T/IEEE 802.3an Ethernet Controller [Antigua 10G] (rev 03), a dedicated 10 Gbit/s Ethernet card in a PCIe slot, to connect to the internet, and a Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05), an integrated 2.5GbE, to connect to the robot:

05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
	Subsystem: Gigabyte Technology Co., Ltd RTL8125 2.5GbE Controller
	Flags: bus master, fast devsel, latency 0, IRQ 18, IOMMU group 17
	I/O ports at 3000 [size=256]
	Memory at 43400000 (64-bit, non-prefetchable) [size=64K]
	Memory at 43410000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: r8169
	Kernel modules: r8169

06:00.0 Ethernet controller: Aquantia Corp. AQtion AQC113 NBase-T/IEEE 802.3an Ethernet Controller [Antigua 10G] (rev 03)
	Subsystem: Aquantia Corp. Device 0001
	Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 18
	Memory at 42800000 (64-bit, non-prefetchable) [size=512K]
	Memory at 428a0000 (64-bit, non-prefetchable) [size=4K]
	Memory at 42400000 (64-bit, non-prefetchable) [size=4M]
	Expansion ROM at 42880000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: atlantic
	Kernel modules: atlantic

And indeed, sudo dmesg shows me a lot of errors related to PCIe and the atlantic kernel module:

[  172.937567] pcieport 0000:00:1c.4: AER: Correctable error message received from 0000:05:00.0
[  172.937572] atlantic 0000:05:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)
[  172.937573] atlantic 0000:05:00.0:   device [1d6a:04c0] error status/mask=00000001/0000e000
[  172.937575] atlantic 0000:05:00.0:    [ 0] RxErr                  (First)

and I sometimes see kernel stack traces.

We are using the same Ethernet card in other setups (Ubuntu 24.04 with 6.14.0-36-generic) that do not show these pcieport and atlantic issues.

I don't know what is causing this, but I guess this is the cause of the communication interruption, and it is related to the realtime kernel on Ubuntu.

Which Ethernet cards are you using and are they dedicated PCIe cards with similar issues reported in dmesg?