Comm link too unstable for RTK use.

[tonycanike]

My LoRaSerials are unusable for RTK work. The latency (age) of my RTK solution, as report by the Facets, is very unstable when using the LoRaSerial radios. It will work for a few minutes, then the comm link appears to go down, the latency climbs, I lose RTK fix. Then the comm link returns, I get an RTK fixed solution, things are good for a while, and then it happens all over again.

I do not have this problem with my Holybros or my RFD900x radios. They work perfectly with my two Facets in base rover RTK mode.

Here is my full problem description:
https://forum.sparkfun.com/viewtopic.php?f=116&t=60898

Multiple other users seem to be reporting this issue also.

viewtopic.php?f=116&t=60609
viewtopic.php?f=117&t=60671
viewtopic.php?f=117&t=60673
viewtopic.php?f=117&t=60872

[nseidle]

Tony's configuration:

Base/Transmitter/Server
ATR
AT-AirSpeed=0
AT-AutoTune=0
AT-Bandwidth=500.00
AT-ClientFindPartnerRetryInterval=3
AT-CodingRate=7
AT-DataScrambling=0
AT-EnableCRC16=1
AT-EncryptData=1
AT-EncryptionKey=xxx
AT-FramesToYield=3
AT-FrequencyHop=1
AT-FrequencyMax=928.000
AT-FrequencyMin=902.000
AT-HeartBeatTimeout=5000
AT-MaxDwellTime=400
AT-MaxResends=0
AT-NetID=192
AT-NumberOfChannels=50
AT-OperatingMode=0
AT-OverHeadtime=10
AT-PreambleLength=8
AT-SelectLedUse=4
AT-Server=1
AT-SpreadFactor=8
AT-SyncWord=18
AT-TrainingKey=xxx
AT-TrainingTimeout=1
AT-TxPower=30
AT-TxToRxUsec=280
AT-VerifyRxNetID=1

ATS
AT-CopySerial=0
AT-Echo=0
AT-FlowControl=0
AT-InvertCts=0
AT-InvertRts=0
AT-RTSOffBytes=32
AT-RTSOnBytes=256
AT-SerialDelay=50
AT-SerialSpeed=57600
AT-UsbSerialWait=0
OK

Receiver/Rover
ATR
AT-AirSpeed=0
AT-AutoTune=0
AT-Bandwidth=500.00
AT-ClientFindPartnerRetryInterval=3
AT-CodingRate=7
AT-DataScrambling=0
AT-EnableCRC16=1
AT-EncryptData=1
AT-EncryptionKey=xxxx
AT-FramesToYield=3
AT-FrequencyHop=1
AT-FrequencyMax=928.000
AT-FrequencyMin=902.000
AT-HeartBeatTimeout=5000
AT-MaxDwellTime=400
AT-MaxResends=0
AT-NetID=192
AT-NumberOfChannels=50
AT-OperatingMode=0
AT-OverHeadtime=10
AT-PreambleLength=8
AT-SelectLedUse=4
AT-Server=0
AT-SpreadFactor=8
AT-SyncWord=18
AT-TrainingKey=xxxx
AT-TrainingTimeout=1
AT-TxPower=30
AT-TxToRxUsec=280
AT-VerifyRxNetID=1
OK

ats
ATS
AT-CopySerial=0
AT-Echo=0
AT-FlowControl=0
AT-InvertCts=0
AT-InvertRts=0
AT-RTSOffBytes=32
AT-RTSOnBytes=256
AT-SerialDelay=50
AT-SerialSpeed=57600
AT-UsbSerialWait=0

My radios were in multipoint mode (this makes the most sense for the continuous stream of updated RTK data).

[nseidle]

I had SerialTest send a 100-character string every 500ms to my "server" LoRaSerial radio, and I monitored the output of the other radio with either Teraterm or another instance of SerialTest. The 4 green signal strength LEDs were illuminated on the receiving radio, I think 3 or 4 on the server/transmitting radio.

The yellow LED on the server radio would flash with every string transmitted (every 500ms), and the blue LED on the other radios would flash with every string received (every 500ms). Sometimes, often after 4 or so minutes, the reception would stop, the data wasn't displayed in TeraTerm, the blue LED stopped flashing. Nothing would happen for about 30 seconds. Then the green LEDs would blink on the receiving radio, and everything would start working fine again. Sometimes this would happen again every 3-4 minutes, and sometimes it would not.

It's like there's some bug and the receiving radio loses sync with the transmitting radio, and it has to restart/resync. The transmitting radio shows no apparent anomalous behavior - that yellow LED just keeps blinking every 500ms.

[nseidle]

Thanks for reporting!

after 4 or so minutes, the reception would stop,

It sounds as if the units are getting out of sync. In MP mode, the server is transmitting a clock sync but if the client misses the clock sync multiple times, it will eventually get off frequency and the link will go down. We have the time delay of the client realizing it's truly out of sync, and the time delay where the server has to come back around through the hop table. If the client misses the clock sync, it has another wait for the server to come around again. We have a few mitigations in place to reduce this time: the client will enter a discover_scan mode where it actively pings the hop table but this can come back negative if the server is actively transmitting when the client is pinging.

P2P doesn't have this time delay. Because they are expecting to regularly hear from each other, the desync and sync times are much shorter.

Point-To-Point and Multipoint are very different beasts. Are you seeing similar issues with Point-To-Point?

[cturvey]

Point-To-Point and Multipoint are very different beasts.

Indeed, skimming thru this there doesn't look to be a method where one unit establishes itself as a master station, so they all potentially throw DATAGRAM_SYNC_CLOCKS packets at each other, or back-n-forth

[cturvey]

In the One-to-Many situation, really need to establish one as the primary station, driving the hop time and pattern. The primary should be only one broadcasting DATAGRAM_SYNC_CLOCKS, and ideally this should communicate the time to next hop, and where it's going to go. The rest of the stations need to synchronize to this, and not be sending their own DATAGRAM_SYNC_CLOCKS as this will just result in chaos as there's no indication of who's in charge, some of the stations will not be in range of each other, and apt to be synchronized by the periodicity of the messaging from the GPS receivers.

Perhaps some way to identify who is sending sync messages, and some level of precedence. Data from GPS/GNSS should allow for time domain synchronization around multiple units

[tonycanike]

@nseidle Nathan, thanks for jumping on this.

I don't have a solid answer on your P2P vs. MP question. I'm geographically distant from my equipment right now and hope to work on this more mid-February.

I tried Point-to-Point once when I first got the LoRaSerial radios, but I've been focusing on MP as I believe it makes better sense for the RTK use case. If data is lost forgetaboutit, as updated data will be sent in the next second.

I experienced the MP issues with the two radios within 5 feet of each other on my workbench.

@cturvey I do configure the one radio at my RTK base to be the "server", and the docs say only the server is transmitting the sync heartbeats. I haven't looked at the code though.
https://docs.sparkfun.com/SparkFun_LoRaSerial/operating_modes/

Tony.

[cturvey]

Mostly just skimmed the source doing a quick static-analysis as to what looks to be going on, and where the sync packets are transmitted and where received. Unpacked the hopping somewhat, but suppose if it misses a hop it's going to have to wait until it cycles around. I'd need to get some units to do dynamic analysis and review debug output side-by-side, or back-port onto the DISCO / Murata platform

[nseidle]

Hi @cturvey - I welcome the analysis and help. I can send you hardware if desired, just say the word and I'll PM you for an address.

[cturvey]

TBH the logic as I'm unpacking it suggests that both ends schedule transmission, the server doesn't act on reception

SparkFun_LoRaSerial/Firmware/LoRaSerial/States.ino

Line 175 in 62aa9cc

getTxTime(xmitDatagramP2PSyncClocks, &txSyncClocksUsec, "SYNC_CLOCKS");

[j-w-bullfrog]

Yes, Point -to Point has the same issues. I havn't seen any performance difference between to two modes, I also tested with the radio's between 1000ft, 10 & 3 ft apart so I could watch the led's.

[cturvey]

@nseidle Nathan, thanks for the units arrived today, battled the IDE and have it building, needed to regress from RadioLib 6.4.2 back to 5.1.2, but do have closure now. Need to find the LoRaSerial driver now.
Will dig in.

Made a .INF to pull in USBSER.SYS
https://github.com/cturvey/RandomNinjaChef/blob/main/sparkfun_loraserial.inf

[cturvey]

Not sure how you'd like to do support on this.
I've got the build process working on two boxes, one with IDE 1.8.19 and the other with IDE 2.2.1
Everything builds and downloads using the GitHub code (LoRaSerial v2.0 ?)
When I go into the AT-Server=1, AT-OperatingMode=0
It seems to trap out very quickly, either a power-on-reset or perhaps a watchdog.
Do I need to power these externally? Currently just running off USB on an older laptop / powered hub
The initial firmware they shipped with the unit didn't reset (USB ding-dong) like this.

[cturvey]

Reverting to the original image shared in the repo. So something in the build/library. Least confident in WDTZero
Noting method to push in original using Arduino 2.2.1 IDE / Arduino SAMD Board Package

"C:\Users\xx\AppData\Local\Arduino15\packages\arduino\tools\bossac\1.7.0-arduino3/bossac.exe" -i -d --port=COM8 -U true -i -e -w -v "C:\SparkFun\LoRaSerial\SparkFun_LoRaSerial_v2_0.bin" -R

[cturvey]

Turning off watchdog, last message, then it hangs.
"State: MP: Waiting for TX done"
Using RadioLib 5.1.2
Using SAMD_TimerInterrupt 1.10.1, will try 1.9.0
...
Ok 1.9.0 is happier, so making some progress

[cturvey]

@nseidle Likely not the source of the issue, but the logic here is broken allowing corrupt packets to be processed down-stream. Should be || (OR) not && (AND)

https://github.com/sparkfun/SparkFun_LoRaSerial/blob/main/Firmware/LoRaSerial/Radio.ino#L1902

    if ((incomingBuffer[rxDataBytes - 2] != (crc >> 8))
        && (incomingBuffer[rxDataBytes - 1] != (crc & 0xff)))
    {

I can compile and run code, so walking, adding/enabling instrumentation, and doing some dynamic analysis.
Right now not seeing the link dying, functional 10 Hours to this point. Occasional loss/recovery, with the 10 seconds of Blue LED fast flashing.

HEARTBEAT sent on a consistent/repetitive basis (with server ms time-stamping), SYNC_CLOCKS (with channel, in multi-point) on demand, reporting ACK-1 in response to FIND_PARTNER.

Unpacking the to-and-fro of the protocol in my head.