rfm_rh_asyncio_listen example stops receiving after a few hours

[WhatIsThisAllAboutSilly]

I've used this example:

https://github.com/adafruit/Adafruit_CircuitPython_RFM/blob/main/examples/rfm_rh_asyncio_listen.py

As a base for my code receiving data from two Feather M0 before passing to influx. This works fine for a while.

However after usually a day I stop getting messages, even though the M0's are still TX'ing - which I've confirmed by restarting the python script on my RPI4 4G and immediately start getting messages again.

It seems I can trigger the same behaviour by commanding a reset, i.e. rfm.reset() followed with or without rfm.listen() , after the reset is issued - no more messages are RX.

What is going on?

[jerryneedell]

I'm not sure why it fails after running for hours. I'll see if I can reproduce that.

rfm.reset will definitely not work. It does not re-execute the radio initialization so it will be reset to the radio chip defaults.
You need to restart the code to reinitialize.

[WhatIsThisAllAboutSilly]

Is there a way of working out what state the radio is in?

Reason being I could periodically poll the state of the radio and if it is wrong, reinitialise it.

I'm currently using the original RFM Adafruit (non asyncio) library for a few days to see if that works long-term (albeit with the single core pegged at 100%...!)

[jerryneedell]

I did a test last night and my receiver (rfm_rh_asyncio.py) did stop receiving, but The issue was with my transmitter. When I restarted the transmitter, it started receiving again. The receiver was still running fine after > 9 hours. I'll try more tests today but then I will be away from my hardware for a few days, so I will likely not get back to this until next weekend.

If the receiving program has hung, can you terminate it with a Control-C to see where where it is hanging?

I'm not sure what you mean my "state of the radio" You can read most of the radio settings but I'm not sure what setting would tell you why it is not receiving. I'll need to do some head-scratching if I can reproduce the issue.

[WhatIsThisAllAboutSilly]

What I mean by "state of the radio" is its not obvious when it has stopped receiving, eventually I'll have 10+ sensors transmitting once an hour (each containing an hours worth of data) so I could assume that if no packets are received within an hour do action X, but that still risks missing data.

Hence we need:

A) To detect when this happens
B) Automatically action something within the script itself.

A brute force option could be to end the script every hour and restart it via cron, but rather work out why it is dying and fix it.

Thanks :)

[jerryneedell]

I've had mine running again for > 5 hours ....
It may help if you can post your code for comparison with the example code.
I can try to run your code here and see if I can reproduce this. It's hard to help if I can't ;-)

Can you provide more information about the hardware , and post code, for both your receiver and transmitter?
As I noted, I won't be able to do much more on this before the weekend.

[WhatIsThisAllAboutSilly]

Sorry if it wasn't clear - I've had the issue just by using your example code on its own, i.e. Only change is the frequency - to 868.0.

Even your example, after a while it just stops saying its received any packets. Restart the script, it's back. I'm talking 24-48 hours.

Is there some debugging code we can add that I can try?

[jerryneedell]

Sorry if it wasn't clear - I've had the issue just by using your example code on its own, i.e. Only change is the frequency - to 868.0.

Even your example, after a while it just stops saying its received any packets. Restart the script, it's back. I'm talking 24-48 hours.

Is there some debugging code we can add that I can try?

If you can do a "Control-C" when it is hung to get a traceback of the code so we can see where it it hanging, that may help.
Otherwise, I'd suggest sprinkling some print statements in the code to see if you can identify where it is hanging.

[WhatIsThisAllAboutSilly]

Control-C doesn't show any errors, just quits like it would normally.

I'll add something to print "I'm alive" every 10 minutes and see what happens...!

[jerryneedell]

I left my transmitter and receiver running for the 3 days I was away. The receiver ran the entire time but unfortunately, the transmitter stopped aver about a day, but when I restarted it, the receiver was still able to receive OK. I've set up a new transmitter that I think will be more reliable and will let it run again.

However, if you can provide any more information about where your receiver is hanging, that would help since I have not been able to reproduce the issue here.

How fast are you transmitting data to it? How large are the packets?

[WhatIsThisAllAboutSilly]

I modified your example with the following, and has also been running for days without issue to.

Which is baffling because before it stopped multiple times, so going to try again with my code with a tick added like below and see if it stops again.

Bizarre.

# setup interrupt callback function
async def wait_for_packets(packet_status):

    last_alive = 0

    while True:

        current_time = time.time()
        if current_time - last_alive >= 600:  # 600 seconds = 10 minutes
            timestamp = time.strftime("%H:%M:%S")
            print(f"{timestamp} - Still alive...")
            last_alive = current_time

        if rfm.payload_ready():
            packet = await rfm.asyncio_receive(with_header=True, timeout=None)
            if packet is not None:
                packet_status.received = True
                # Received a packet!
                # Print out the raw bytes of the packet:
                print(f"{timestamp} - Received (raw bytes): {packet}")
                #print([hex(x) for x in packet])
                #print(f"RSSI: {rfm.last_rssi}")
        await asyncio.sleep(0.001)

[jerryneedell]

Looks like you fixed it ;-) well done!
That is strange and frustrating, but so far, I have not been able to reproduce the issue.
I'm curious if you are exiting the program from the Pi Desktop or vi an ssh connection. I have had issues with keeping my transmit program running on a "headless" pi zero. My receiver is running from the Desktop of a Pi 5 and so far has not had any issues.

[jerryneedell]

I modified your example with the following, and has also been running for days without issue to.

Which is baffling because before it stopped multiple times, so going to try again with my code with a tick added like below and see if it stops again.

Bizarre.

# setup interrupt callback function
async def wait_for_packets(packet_status):

    last_alive = 0

    while True:

        current_time = time.time()
        if current_time - last_alive >= 600:  # 600 seconds = 10 minutes
            timestamp = time.strftime("%H:%M:%S")
            print(f"{timestamp} - Still alive...")
            last_alive = current_time

        if rfm.payload_ready():
            packet = await rfm.asyncio_receive(with_header=True, timeout=None)
            if packet is not None:
                packet_status.received = True
                # Received a packet!
                # Print out the raw bytes of the packet:
                print(f"{timestamp} - Received (raw bytes): {packet}")
                #print([hex(x) for x in packet])
                #print(f"RSSI: {rfm.last_rssi}")
        await asyncio.sleep(0.001)

Have you been able to reproduce the issue?

[WhatIsThisAllAboutSilly]

Yes - it has stopped - something defiantly buggy with it. This time it was well over a week before it stopped.

The 123 and 243 length packets are mine, the others are presumably other things on the band that aren't mine(?)

2025-02-03 16:08:52,347 - INFO - Packet received, length: 123
2025-02-03 16:15:30,048 - INFO - Packet received, length: 243
2025-02-03 16:16:35,260 - INFO - Still Alive...
2025-02-03 16:17:54,511 - INFO - Packet received, length: 214
2025-02-03 16:26:35,261 - INFO - Still Alive...
2025-02-03 16:36:35,262 - INFO - Still Alive...
2025-02-03 16:46:35,262 - INFO - Still Alive...
2025-02-03 16:56:35,263 - INFO - Still Alive...
2025-02-03 17:06:35,264 - INFO - Still Alive...
2025-02-03 17:07:52,853 - INFO - Packet received, length: 123
2025-02-03 17:14:34,381 - INFO - Packet received, length: 243
2025-02-03 17:16:35,264 - INFO - Still Alive...
2025-02-03 17:26:35,265 - INFO - Still Alive...
2025-02-03 17:36:35,266 - INFO - Still Alive...
2025-02-03 17:46:35,266 - INFO - Still Alive...
2025-02-03 17:56:35,267 - INFO - Still Alive...
2025-02-03 18:06:35,268 - INFO - Still Alive...
2025-02-03 18:06:53,358 - INFO - Packet received, length: 123
2025-02-03 18:13:38,715 - INFO - Packet received, length: 243
2025-02-03 18:16:35,268 - INFO - Still Alive...
2025-02-03 18:19:18,545 - INFO - Packet received, length: 171
2025-02-03 18:26:35,269 - INFO - Still Alive...
2025-02-03 18:28:30,627 - INFO - Packet received, length: 90
2025-02-03 18:36:35,270 - INFO - Still Alive...
2025-02-03 18:46:35,270 - INFO - Still Alive...
2025-02-03 18:56:35,271 - INFO - Still Alive...
2025-02-03 19:06:35,271 - INFO - Still Alive...
2025-02-03 19:16:35,272 - INFO - Still Alive...
2025-02-03 19:26:35,273 - INFO - Still Alive...
2025-02-03 19:36:35,274 - INFO - Still Alive...
2025-02-03 19:46:35,274 - INFO - Still Alive...
2025-02-03 19:56:35,275 - INFO - Still Alive...
2025-02-03 20:06:35,276 - INFO - Still Alive...
2025-02-03 20:16:35,277 - INFO - Still Alive...
2025-02-03 20:26:35,278 - INFO - Still Alive...
2025-02-03 20:36:35,278 - INFO - Still Alive...
2025-02-03 20:46:35,278 - INFO - Still Alive...
2025-02-03 20:56:35,279 - INFO - Still Alive...
2025-02-03 21:06:35,280 - INFO - Still Alive...
2025-02-03 21:16:35,280 - INFO - Still Alive...
2025-02-03 21:26:35,280 - INFO - Still Alive...
2025-02-03 21:36:35,281 - INFO - Still Alive...
2025-02-03 21:46:35,281 - INFO - Still Alive...
2025-02-03 21:56:35,282 - INFO - Still Alive...
2025-02-03 22:06:35,282 - INFO - Still Alive...
2025-02-03 22:16:35,282 - INFO - Still Alive...
2025-02-03 22:26:35,283 - INFO - Still Alive...
2025-02-03 22:36:35,284 - INFO - Still Alive...
2025-02-03 22:46:35,284 - INFO - Still Alive...
2025-02-03 22:56:35,285 - INFO - Still Alive...

[WhatIsThisAllAboutSilly]

Going to run my version of the non asycio to see if its hardware related or not.

[jerryneedell]

Just to clarify, you were expecting it to receive packets approximately once per hour(both the 123 and 243 size) correct? Are they from the same source? Even so, I assume you are sure they are still transmitting ....just thinking out loud....
And it ran OK for a week, before it stopped receiving?