rx_st40p: out-of-order packets detected with redundant stream

[DianaEs2]

Test setup: PacketStorm tool was used for this test with Hitless Merge / Protection test pattern. The differential latency was set to zero (~5us) and the ancillary format was 1080i59.

Bug description: During the 10 second period of synchronized alternative burst loss, the following error was observed in rx_st40p session in MTL.
_04-07 17:44:58.792 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: PTP(0): time 1775583935792289284, 2026-04-07 17:45:35
04-07 17:44:58.792 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: RX_ANC_SESSION(0:rx_st40p_1): fps 57.799567 frames 578 pkts 3746 (redundant 3068)
04-07 17:44:58.792 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: RX_ANC_SESSION(0): out of order pkts 4 (74:77)
04-07 17:44:58.792 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: RX_ANC_SESSION(0): interlace first field 3668 second field 3146
04-07 17:44:58.792 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: RX_st40p(0,rx_st40p_1), p(6:receiving) c(6:receiving)
04-07 17:44:58.792 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: RX_st40p(0), frame get try 577 succ 577, put 577
04-07 17:44:58.792 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: * * E N D S T A T E * *

RTP packet marker was also lost as well sequence number discontinuity was detected (this is an application log, based on MTL info).
_04-07 17:44:58.893 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: recv_buffer, anc marker not detected
04-07 17:44:58.893 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: rx_ancillary_session_handle_pkt(0,1), detected interlaced stream (F=0x2)
04-07 17:44:58.893 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: recv_buffer, anc sequence number dscontinuity

Here is pcap of the ancillary hitless_merge_anc.zip. The packet sequence number is actually is continuous but it received on the redundant stream.

Expected result: rx_st40p should recover stream and not report any errors.

[DianaEs2]

I replayed the attached pcap, but I have not been able to reproduce the issue. It is not reliably reproducible; it fails only occasionally. I will post another pcap for reproduction once I get it.

For your reference, here are the steps to replay pcap.

1. Download and compile this project https://github.com/appneta/tcpreplay/releases/download/v4.4.4/tcpreplay-4.4.4.tar.xz
2. Prepare cache file by filtering pcap by source ip
   tcpprep --cidr=10.12.144.124/32 --pcap=./hitless_merge_anc.pcap --cachefile=./hitless_merge_anc.cache
3. Replay the pcap file on two interfaces
   sudo tcpreplay --cachefile=./hitless_merge_anc.cache -i ens3f0 -I ens3f1 ./hitless_merge_anc.pcap
4. After it's done, observe the result
   Actual: 7544 packets (741986 bytes) sent in 10.01 secondsRated: 74114.6 Bps, 0.592 Mbps, 753.54 pps
   Flows: 13 flows, 1.29 fps, 7544 unique flow packets, 0 unique non-flow packets
   Statistics for network device: ens3f0
   Successful packets: 3321
   Failed packets: 0
   Truncated packets: 0
   Retried packets (ENOBUFS): 0
   Retried packets (EAGAIN): 0
   Flows total: 1
   Flows unique: 0
   Flows expired: 0
   Flow packets: 3321
   Non-flow packets: 0
   Invalid flow packets: 0
   Statistics for network device: ens3f1
   Successful packets: 4223
   Failed packets: 0
   Truncated packets: 0
   Retried packets (ENOBUFS): 0
   Retried packets (EAGAIN): 0
   Flows total: 12
   Flows unique: 0
   Flows expired: 0
   Flow packets: 4223
   Non-flow packets: 0
   Invalid flow packets: 0

[DianaEs2]

I was able to capture the state of the bug in this pcap hitless_merge_anc_v2.zip
When pcap is replayed, MTL detects this error:
Warn: RX_ANC_SESSION(0): out of order pkts 1 (9:8)

Here is the spot where packets are out-of-order

[DianaEs2]

This is a slightly different case, which is not currently supported in rx_st40p hitless_merge_anc_v3.zip

st40_frame_info::rtp_marker is missing, or st40_frame_info::seq_discont is set for the following frames
recv_buffer, anc marker not detected
recv 35: time 2208549877 status 2
recv_buffer, anc sequence number dscontinuity
recv 36: time 2208564892 status 2
recv_buffer, anc marker not detected
recv 65: time 2208608436 status 2
recv_buffer, anc sequence number dscontinuity
recv 66: time 2208623451 status 2
recv_buffer, anc sequence number dscontinuity
recv 96: time 2208680508 status 2