PI & IndiAllSky Service Unavailable After 3 Days Operations

[ldb20151]

I've noticed several occurances where the camera goes completly offline. It won't respond via SSH or the web interface. This is new since my update of Indi-AllSky a week or so ago. The pi5 refuses to connect -- same behavior as if the unit was powered down. The web interface shows:

When I phyically inspect the pi, I see its powered powered but no output. Heaters and fans are off but probably should be on. When I rebooted the pi, it showed last login was today even though it was inaccessible. So, perhaps it was still operating 'normally' except for user interfaces.

I reverted to a last known good config in hopes I accidentally changed something in the update process, but I don't know of any changes other than the system updates.

Does anyone have a suggestion to help run down what's causing this?

[aaronwmorris]

The web application log is located at /var/log/indi-allsky/webapp-indi-allsky.log That may provide insight into what is happening.

[ldb20151]

The system routinely stays online anywhere from 1 day to 3 days before it becomes unresponsive and web interface gives Service Unavailable message. I checked the log you mentioned but it appears empty.

sudo ls -lh /var/log/indi-allsky/webapp-indi-allsky.log
-rw-r--r-- 1 root adm 0 Oct 17 00:00 /var/log/indi-allsky/webapp-indi-allsky.log

Perhaps this is due to the fact that I needed to reboot the system to regain access -- does it empty the log? Once it becomes unresponsive, I'm locked out of the web interface as well as SSH. Power cycle is required to restore control.

As mentioned, this began after my upgrade to latest OS and Indi-allsky code almost 3 weeks ago. FYSA, I'm uploading current support info to another discussion topic #2469. I believe the support info, if helpful, could be relevant to both issues.

v/r,
-Larry

[aaronwmorris]

The log is rotated daily. There should be archived logs from previous days in the same folder.

[ldb20151]

OK, I think I found them. Attached are the last 2 files. They're very small for the few logs I have experience with. I tried opening them with notepad on my PC but just got random characters. I can only read them in the mobaxterm built in text editor -- maybe that's normal? Anyway, I do see multiple gunicorn errors but I don't know what they mean, or why they would be happening. How do I interpret these? I'm still a newby. I'm sending one of these files via cut and paste in case I'm not copying them correctly. I looked at several more files and they all look very similar to me. Sorry if this is all duplicative.

-Larry

2025-10-15T12:33:52.880895-04:00 ldb20151rpi5 gunicorn.indi_allsky.wsgi.error: [1169] Starting gunicorn 23.0.0
2025-10-15T12:33:52.881136-04:00 ldb20151rpi5 gunicorn.indi_allsky.wsgi.error: [1169] Listening at: unix:/var/lib/indi-allsky/gunicorn-indi-allsky.sock (1169)
2025-10-15T12:33:52.881192-04:00 ldb20151rpi5 gunicorn.indi_allsky.wsgi.error: [1169] Using worker: gthread
2025-10-15T12:33:52.883711-04:00 ldb20151rpi5 gunicorn.indi_allsky.wsgi.error: [1171] Booting worker with pid: 1171
2025-10-15T12:56:43.319438-04:00 ldb20151rpi5 [INFO] MainProcess-1171 forms.init() [6935]: Dates processed in 0.0018 s
2025-10-15T12:56:43.406289-04:00 ldb20151rpi5 [INFO] MainProcess-1171 forms.getVideos() [6621]: Found 19 timelapses
2025-10-15T12:58:02.243428-04:00 ldb20151rpi5 [INFO] MainProcess-1171 forms.init() [6935]: Dates processed in 0.0005 s
2025-10-15T12:58:02.378536-04:00 ldb20151rpi5 [INFO] MainProcess-1171 forms.getVideos() [6621]: Found 19 timelapses
2025-10-15T13:02:43.491334-04:00 ldb20151rpi5 gunicorn.indi_allsky.wsgi.error: [1173] Starting gunicorn 23.0.0
2025-10-15T13:02:43.491509-04:00 ldb20151rpi5 gunicorn.indi_allsky.wsgi.error: [1173] Listening at: unix:/var/lib/indi-allsky/gunicorn-indi-allsky.sock (1173)
2025-10-15T13:02:43.491552-04:00 ldb20151rpi5 gunicorn.indi_allsky.wsgi.error: [1173] Using worker: gthread
2025-10-15T13:02:43.494285-04:00 ldb20151rpi5 gunicorn.indi_allsky.wsgi.error: [1175] Booting worker with pid: 1175
2025-10-15T13:06:42.372790-04:00 ldb20151rpi5 [INFO] MainProcess-1175 views.dispatch_request() [3617]: Saved new config
2025-10-15T17:50:16.804685-04:00 ldb20151rpi5 gunicorn.indi_allsky.wsgi.error: [1138] Starting gunicorn 23.0.0
2025-10-15T17:50:16.804963-04:00 ldb20151rpi5 gunicorn.indi_allsky.wsgi.error: [1138] Listening at: unix:/var/lib/indi-allsky/gunicorn-indi-allsky.sock (1138)
2025-10-15T17:50:16.805024-04:00 ldb20151rpi5 gunicorn.indi_allsky.wsgi.error: [1138] Using worker: gthread
2025-10-15T17:50:16.807664-04:00 ldb20151rpi5 gunicorn.indi_allsky.wsgi.error: [1159] Booting worker with pid: 1159
2025-10-15T19:02:56.298198-04:00 ldb20151rpi5 [INFO] MainProcess-1159 views.dispatch_request() [3617]: Saved new config
2025-10-15T19:08:00.890234-04:00 ldb20151rpi5 [INFO] MainProcess-1159 views.dispatch_request() [3617]: Saved new config
2025-10-15T22:44:04.375148-04:00 ldb20151rpi5 [INFO] MainProcess-1159 views.getChartData() [1731]: Calculating histogram from RoI
2025-10-15T22:44:04.379719-04:00 ldb20151rpi5 [WARNING] MainProcess-1159 base_views._load_detection_mask() [816]: No detection mask defined
2025-10-15T22:45:58.228601-04:00 ldb20151rpi5 [INFO] MainProcess-1159 views.getChartData() [1731]: Calculating histogram from RoI
2025-10-15T22:45:58.233239-04:00 ldb20151rpi5 [WARNING] MainProcess-1159 base_views._load_detection_mask() [816]: No detection mask defined

webapp-indi-allsky.log-20251017.gz
webapp-indi-allsky.log-20251016.gz

[aaronwmorris]

Nothing there appears abnormal. You may need to check the system logs to make sure the drive is not failing.

sudo dmesg

And check out the various logs at /var/log/kern.log.*

[ldb20151]

UPDATE: I'm still trying to track down this issue. So far, I found that one of the OS updates reset the pi's power management configuration.
Command line 'iwconfig wlan0' showed Power Management below was 'on'. Once I reconfigured this to be 'off' again, the pi data rate greatly improved and freeze-ups have decreased -- but not eliminated.
wlan0 IEEE 802.11 ESSID:"+++REDACTED+++"
Mode:Managed Frequency:5.32 GHz Access Point: +++REDACTED+++
Bit Rate=180 Mb/s Tx-Power=31 dBm
Retry short limit:7 RTS thr:off Fragment thr:off
Power Management:on (I changed this to off)
Link Quality=53/70 Signal level=-57 dBm
Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0
Tx excessive retries:0 Invalid misc:0 Missed beacon:0
At present, I'm running the system on my workbench in a warm-box (my 3D filament dryer); I'm slowly increasing the ambient temp to see if I can expose a flaw in the hardware; especially the SSD. This will take another week to complete.
Larry

[aaronwmorris]

Interesting.

Raspberry Pi OS normally uses network manager for managing the network connections. You can set the powersave flag on the connection using nmcli.

nmcli con mod "Connection Name" 802-11-wireless.powersave 2

[ldb20151]

UPDATE #2: I completed temperature burn in testing. I verified the pi with all hats and nvme card worked fine up to around 55C -- which is warmer than my hotest day this summer. That said, it was running inside on my workbench so no real stars, no GPS, stronger wifi and I probably didn't access it much except to see if it was responding. I have also tested the SSD card at room temp, separately from the pi and no errors.

I have resolved the power config issue (disabled) so I'm putting the system back outside to see if it performs better.

As suggested I looked through the logs via sudo dmesg. Its a very long file. I'm not very experienced so not sure what to look for. I don't see any obvious errors.
-Larry