The Outer Limits: A Magical Time Period on Random Hours?

[JohnJ9ml]

I'm presenting this almost purely for amusement, but there's something interesting going on here. Perhaps this is only good for a laugh:

I admit I'm using the pypowerwall library as a vehicle to pull my PW3 data into another set of analytics I've been doing for years. As such, while I know next to nothing about Python, I've managed to shim together something that hangs out for an hour or two at a time, runs from cron, uses a lockfile to avoid running multiple instances, and pulls various bits of information via the TEDAPI at regular intervals, writes them to files, and something else scoops the data up, parses it, stores it, and moves on. Rube Goldberg-esque, indeed.

When 25.2.2 came out, everything seemed to truly settle into place for me. The strange flapping that had been going on between the gateway and the grid dramatically minimized, and now I had a properly running system. Hand in hand, my monitoring really stabilized as well. Bliss ensued.

Bliss was shortlived: with the release of 25.2.4, and continuing on to 25.2.6, the strangest behavior started occurring: for some reason totally not understood to me, and totally unrelated in time to anything else I have running on my own systems monitoring things, the Powerwall() connection to the primary PW3 inverter just hangs. What's truly strange about it is that it rarely happens, sometimes goes days without occurring, but sometimes only hours and hangs multiple times - but when it does hang, it hangs at 23 minutes 50 seconds past the hour. Go figure. For example, it didn't hang at all on the 29th, but on the 30th thus far it hung at 03:23:50 and again at 10:23:50.

It's a very strange anti-pattern to me. It's one for which I can locate no correlation in any of my own systems. I had the problem on 0.12.8, upgraded to 0.12.9, and nothing changed. To continue my Rube Goldberg device, I finally rigged up something using my own polling and MQTT to restart the Python process if this is detected, and if I have to just restart the entire RPi to get a clean slate. When it restarts, all is happy and well - possibly until the next "witching moment".

Like I said, maybe this is purely posted for amusement, but sometimes you see a pattern and recognize it, even though you have absolutely no explanation for it. I'm trying to build up a bit more diagnostic logic so I can maybe better see what might be taking place, but right now this is a real head scratcher.

You are clear to laugh at me and my predicament.

[jasonacox]

Not laughing.. just wish we could find out what may be causing this. Have you tried turning on debug to see if that could help?

import pypowerwall
pypowerwall.set_debug(True)

[Nexarian]

I think another important piece would be for us to see the code you've written for your custom monitoring system, including the crontab file. Multi-threading is really hard, especially deadlocks and race conditions. You mention uses a lockfile to avoid running multiple instances but there could actually be a bug in that.

When debugging this kind of thing you can't leave any stone unturned and you have to question every single assumption, including the ones you don't realize you're making. How are you setting up the static route? What type of router does your home LAN use? Are there any VLANs involved? Is the PW3 connected over wifi or ethernet? What type of RPi are you using, what are its specs? Have you tried using the latest version of Python? Etc, etc.

[JohnJ9ml]

Yes, there is still a lot of diagnostic to be done here. I have found a way out of it for the short term (which forced me to play around with MQTT, something that's been on the mental list for some time), but I know I have a long period of picking at this over time.

What caught my eye was just the consistency of the stop time. You see it twice in one night and you think, "wow, what a strange coincidence!" Then you see it six or eight times where it's always 23 minutes 50 seconds after the hour that it last worked, and at some point you stop believing in this as a mere coincidence. That's the only bit that caused me to write this up right now. It's never the same hour it happens; Mickey's big hand is always in the same place when it apparently locks up.

"The Twelve Dwarves were down to eight before they kicked out Hungry."