Packet loss when streaming adjacent slots

[jlashner]

Hi all, I'm having a packet drop issue at site and thought I'd post here to try to get more eyes on it. Our smurf is configured with slots 2, 3, and 5. Communication between the smurf and the slot seems good, but I'm seeing high data drop rates when trying to run all slots at once. In particular, I'm seeing:
• No packets dropped when running slots 2 +5 together
• No packets dropped when running slots 3+5 together
• Lots of packets dropped when running slots 2+3 together

In particular:

Slots [2, 3], 4 kHz, 20 sec
Slot 2:
  Dropped frac: 0.0003179828663349669
  Dropped samples: 34
Slot 3:
  Dropped frac: 0.00030895880881429583
  Dropped samples: 31

Slots [2, 5], 4 kHz, 20 sec
Slot 2:
  Dropped frac: 0.0
  Dropped samples: 0
Slot 5:
  Dropped frac: 0.0
  Dropped samples: 0

Slots[3, 5], 4 kHz, 20 sec
Slot 3:
  Dropped frac: 0.0
  Dropped samples: 0
Slot 5:
  Dropped frac: 0.0
  Dropped samples: 0

It also seems like the fraction of dropped packets is independent of flux-ramp rate, which to me rules out software, and points to something wrong with the fiber connections. I've tried:

• replacing the entire fiber between the smurf and the server
• Replacing the QSFP module
• Cleaning the QSFP, mtp, all LCs, and SFP modules.

I'm starting to run out of ideas so wanted to know if anyone (@yuhanwyhan, @swh76, @msilvafe) has seen anything like this, or has any more ideas of things to check.

[msilvafe]

Is it just 2,3 or do 3,4 and 4,5 show the same behavior? Are you sure that the SFPs are installed in the correct SFP cages on the vadatech switch and that they're all installed properly. I would suspect the interface on that side given that you cleaned and swapped out all of this different hardware.

[jlashner]

So I've tried in 3 different slot configurations...

When running with slots [2, 3, 4] I was seeing this kind of interference between slots 2+3, but none between 3+4
When running with slots [2, 4, 5] I saw interference between slots 4+5, and 2 was the good slot
And then with [2, 3, 5] interference between slots 2+3....

I believe all of the SFPs are correct.... but one of them is definitely stuck (slot 3), and I'm having trouble removing it.

[msilvafe]

Can you try swapping the switch used for the timing system with the switch used in sat1-crate2 to see if the issue follows the switch?

[jlashner]

I'll try this

[jlashner]

Swapped the switches and used all new SFPs, and getting same issue:

Slots [2, 3], 4 kHz, 20 sec
dropped frac: 0.00011003117549972492, 0.00012157192499012228

Slots [3, 5], 4 kHz, 20 sec
dropped frac: 0.0, 0.0

Slots [2, 5], 4 kHz, 20 sec
dropped frac: 0.0, 0.0

[jlashner]

Ok, I swapped to a brand new QSFP, and am still seeing the same issues, but somehow not as symmetric as before? This might just be randomness in the packet drops, but before 2 and 3 were dropping about the same number of packets, and now 2 is dropping much more than 3. I think it might be an issue with the PCIe port...

Slots [2, 3], 4 kHz, 20 sec
dropped frac: 0.00041269963144963145, 7.70594128072744e-05
dropped samps: 43, 8

Slots [2, 5], 4 kHz, 20 sec
dropped frac: 8.575079105104745e-06, 0.0
dropped samps: 1, 0

Slots [3, 5], 4 kHz, 20 sec
dropped frac: 0.0, 9.368734658696996e-06
dropped samps: 0, 1

For now, I'm going to run with slots [2, 5, 7], and see if that avoids interference. Hopefully this is just an issue with the PCIe and we don't see the same type of thing with smurf-srv20

[jlashner]

Alright, I didn't know what else I could swap out so I'm thinking this might be an issue with the PCIe port. Running with slots 2, 5, and 7 and we are under spec for the packet drop rates:

Slots [2, 5, 7], 10 kHz, 60 sec
dropped frac: 0.0, 1.3479795808053099e-06, 4.286112281854744e-06
dropped samps: 0, 1, 3

[swh76]

Working with Larry and Ryan (SLAC engineers) we think we have found and solved this problem. It was due to a bug in the PCIe card firmware that fortunately has been fixed since we froze the SMuRF PCIe firmware version. Larry was able to build us a new version, https://github.com/slaclab/smurf-pcie/releases/tag/v3.0.1, with the fix, and at least on SAT1-Crate1 with four carriers, we are seeing no drops streaming all four slots simultaneously, 2000 channels at 10 kHz. This is a big fix!
Larry reports that in his test bench at SLAC he was able to stream at 100kHz with no drops. Here's instructions for how to upgrade the firmware (we are still using the same kernel driver version, v5.7.0):

1. Ssh to the SMuRF server you want to upgrade. Be aware the upgrade process will reboot the machine. If you're running any dockers or SMuRF software, take it all down.

2. Download the firmware binary mcs files from https://github.com/slaclab/smurf-pcie/releases/tag/v3.0.1 to
   SmurfKcu1500RssiOffload10GbE-0x03000000-20230830025732-ruckman-d89c87b_secondary.mcs.gz
   the SMuRF server you want to upgrade. I'll upload them here too in case you don't have access to that repo: SmurfKcu1500RssiOffload10GbE-0x03000000-20230830025732-ruckman-d89c87b_primary.mcs.gz and
   SmurfKcu1500RssiOffload10GbE-0x03000000-20230830025732-ruckman-d89c87b_secondary.mcs.gz

3. If you don't already have it, check out the SMuRF PCIe docker:

# Upgrade the SMuRF docker release script
release-docker.sh -u

# Checkout pcie docker version v2.1.1
release-docker.sh -t pcie -v v2.1.1

# On the SMuRF server, assuming you copied the mcs files to /home/cryo
mv ~/SmurfKcu1500RssiOffload10GbE-0x03000000-20230830025732-ruckman-d89c87b_* ~/docker/pcie/v2.1.1/shared/
cd ~/docker/pcie/v2.1.1/
./run.sh
# in docker
load_firmware.sh

When prompted, enter the number corresponding to the d89c87b mcs file. The script will then program the PCIe card FPGA and reboot the server. You should see this:

cryo@smurf-srv-SO6:/usr/local/src/smurf-pcie$ load_firmware.sh 
Rogue/pyrogue version v4.6.3. https://github.com/slaclab/rogue

==========================================================
 Passing startup args in start() method is now depcreated.
 Startup args should now be passed to the root creator.
    Example: pyrogue.Root(timeout=1.0, pollEn=True
==========================================================
#########################################
Current Firmware Loaded on the PCIe card:
#########################################
Path         = PcieTop.AxiPcieCore.AxiVersion
FwVersion    = 0x2030000
UpTime       = 3:00:20
GitHash      = 0x7aa0ea1a0ae6890e9e594a3d8a37476183075ed5
XilinxDnaId  = 0x400200010117f1801c8102c5
FwTarget     = SmurfKcu1500RssiOffload10GbE
BuildEnv     = Vivado v2019.2
BuildServer  = rdsrv302 (x86_64)
BuildDate    = Tue 12 May 2020 01:44:23 PM PDT
Builder      = ruckman
#########################################
0 : /shared/SmurfKcu1500RssiOffload10GbE-0x03000000-20230830025732-ruckman-d89c87b
Enter image to program into the PCIe card's PROM: 0
PcieTop.AxiPcieCore.AxiMicronN25Q[0].LoadMcsFile: /shared/SmurfKcu1500RssiOffload10GbE-0x03000000-20230830025732-ruckman-d89c87b_primary.mcs.gz
PROM Manufacturer ID Code  = 0x20
PROM Manufacturer Type     = 0xbb
PROM Manufacturer Capacity = 0x20
PROM Status Register       = 0x2
PROM Volatile Config Reg   = 0xfb
Reading .MCS:    [####################################]  100%          
Erasing PROM:    [####################################]  100%          
Writing PROM:    [####################################]  100%          
Verifying PROM:  [####################################]  100%          
LoadMcsFile() took 0:02:12 to program the PROM


            ***************************************************
            ***************************************************
            The MCS data has been written into the PROM.       
            To reprogram the FPGA with the new PROM data,      
            a IPROG CMD or power cycle is be required.
            ***************************************************
            ***************************************************


PcieTop.AxiPcieCore.AxiMicronN25Q[1].LoadMcsFile: /shared/SmurfKcu1500RssiOffload10GbE-0x03000000-20230830025732-ruckman-d89c87b_secondary.mcs.gz
PROM Manufacturer ID Code  = 0x20
PROM Manufacturer Type     = 0xbb
PROM Manufacturer Capacity = 0x20
PROM Status Register       = 0x2
PROM Volatile Config Reg   = 0xfb
Reading .MCS:    [####################################]  100%          
Erasing PROM:    [####################################]  100%          
Writing PROM:    [####################################]  100%          
Verifying PROM:  [####################################]  100%          
LoadMcsFile() took 0:02:13 to program the PROM


            ***************************************************
            ***************************************************
            The MCS data has been written into the PROM.       
            To reprogram the FPGA with the new PROM data,      
            a IPROG CMD or power cycle is be required.
            ***************************************************
            ***************************************************



Reloading FPGA firmware from PROM ....

Please reboot the computer

It says "please reboot the computer" but will actually reboot the computer for you. Once the server is back up, ssh back in to confirm the firmware has been properly loaded like this:

cryo@smurf-srv-SO6:~$ cat /proc/datadev_0
-------------- Axi Version ----------------
     Firmware Version : 0x3000000
           ScratchPad : 0x0
        Up Time Count : 132
            Device ID : 0x0
             Git Hash : d89c87b6b29ddc5dab8a3be374e749725d598399
            DNA Value : 0x00000000400200010117f1801c8102c5
         Build String : SmurfKcu1500RssiOffload10GbE: Vivado v2023.1, rdsrv407 (Ubuntu 20.04.6 LTS), Built Wed 30 Aug 2023 02:57:32 AM PDT by ruckman

-------------- General HW -----------------
          Int Req Count : 0
        Hw Dma Wr Index : 0
        Sw Dma Wr Index : 0
        Hw Dma Rd Index : 0
        Sw Dma Rd Index : 0
     Missed Wr Requests : 0
       Missed IRQ Count : 0
         Continue Count : 0
          Address Count : 4096
    Hw Write Buff Count : 1024
     Hw Read Buff Count : 0
           Cache Config : 0x0
            Desc 128 En : 1
            Envable Var : 0x4010101
      Driver Load Count : 1

-------------- General --------------------
          Dma Version : 0x5
          Git Version : v5.7.0

-------------- Read Buffers ---------------
         Buffer Count : 1024
          Buffer Size : 131072
          Buffer Mode : 1
      Buffers In User : 0
        Buffers In Hw : 1024
  Buffers In Pre-Hw Q : 0
  Buffers In Rx Queue : 0
      Missing Buffers : 0
       Min Buffer Use : 0
       Max Buffer Use : 0
       Avg Buffer Use : 0
       Tot Buffer Use : 0

-------------- Write Buffers ---------------
         Buffer Count : 1024
          Buffer Size : 131072
          Buffer Mode : 1
      Buffers In User : 0
        Buffers In Hw : 0
  Buffers In Pre-Hw Q : 0
  Buffers In Sw Queue : 1024
      Missing Buffers : 0
       Min Buffer Use : 0
       Max Buffer Use : 0
       Avg Buffer Use : 0
       Tot Buffer Use : 0

You should confirm that the driver version is v5.7.0 and the git hash is d89c87b6b29ddc5dab8a3be374e749725d598399 (=the new firmware version).

I've already upgraded the 2x SAT1 servers (smrfsrv20-satp1 and smrfso3-satp1) and the LAT server (smrfso6-lat). Caveat emptor : we should watch for any issues with this new version - so far we have only checked whether or not it drops frames.

[swh76]

Another new feature of the SMuRF software stack : while tracking down this issue, SLAC added a new ability to poll / inspect registers in the PCIe card firmware space. This capability is present starting with the new v2.1.1 version of the smurf-pcie-docker. Here's how to pull that down to your SMuRF server:

# Upgrade the SMuRF docker release script
release-docker.sh -u

# Checkout pcie docker version v2.1.1
release-docker.sh -t pcie -v v2.1.1

then you can start an EPICs server to serve up registers in the PCIe like this:

cd ~/docker/pcie/v2.1.1/
./run.sh 
start_epics.sh

Leave the docker running, then you will be able to poll PCIe registers anywhere you can access EPICs (e.g. in the SMuRF client dockers, utils docker, etc.):

cd docker/utils/
./run.sh 
caget pcie_test0:pcie:Core:EthPhyGrp:EthPhy[2]:rxCrcError
pcie_test0:pcie:Core:EthPhyGrp:EthPhy[2]:rxCrcError 0

this particular register, for instance, records a certain kind of packet corruption that was useful to monitor to track down the issue in this ticket.