Camera periodically hangs after Indi AllSky Upgrade this week #2126
-
|
The following was mentioned in #2121 but that issue was largely driven by a different problem, so I'm opening a separate topic. I apologize if this is redundant. I'm still learning the Git Hub etiquet. In addition to characters corrupted in the config file templates (noted in #2121) I immediately started noticing the camera would periodically hang up. It was recommended in #2121 to reseat the camera cable. I since totally replaced the cable. The problem continues to occur. I have no data to say this is connected with the recent upgrade, but it definitely feels that way. Prior to the upgrade the system had been running on my workbench or outside almost continuously since Nov 24. I had never seen this error until the upgrade a few days ago, and even after replacing and reseating the camera connector multiple times, the camera continues to lock up with the error below. If I reboot the system, it'll often operate a few hours before repeating, but that's the only way I've found to get it going again. I'm at a loss to know what to do to fix this. For kicks, I installed my backup SSD and ran the system a few hours on the old AllSky version. No problems. When I switch back to the version upgraded this week, the issue returns. It doesn't seem hardware related. That said, there are some config file changes since the Nov 24 version. Perhaps I have something configured wrong for the upgraded system? Could my exposure settings cause this issue? Any suggestion would be welcome. I did the upgrade because I was trying to figure out the dew heater and fan control settings -- discussions indicated there were improvements, so I decided it was time for the upgrade. Otherwise I would have stayed with what worked. v/r,
|
Beta Was this translation helpful? Give feedback.
Replies: 9 comments 12 replies
-
|
Have you compared the versions of libcamera between the working system and the system that hangs? There is nothing in indi-allsky that would cause the rpicam-still program to hang. A Support Info for the old and new systems would show this detail. |
Beta Was this translation helpful? Give feedback.
-
|
No I haven't. That will take some time -- the only way I know to get that is to swap the SSD again, which I will do. |
Beta Was this translation helpful? Give feedback.
-
|
The rpicam-utils project manages the commands. There are updates to these utilities about every 6 months. The libcamera team manages the interface and the kernel has the overlay modules for the camera hardware. This receives updates periodically, as well. Taking images with libcamera is just a matter of running the rpicam-still command. When the command exits, it is finished taking the exposure. If that command hangs, something else has gone wrong. You might check |
Beta Was this translation helpful? Give feedback.
-
|
This exact thing started happening for me as well a few months ago I switched from an RPi 4 with an USB connected SSD to an RPi 5 with an NVME on an NVME hat pretty recently. The RPi 4 ran superbly well, no issues there. I think that the RPi 5 ran just fine in the beginning. When it started happening I for some reason I began thinking that there might be a correlation between the issues starting and the temperature of the NVME rising close to 70 degrees Celcius as the weather got warmer during the spring, sun going higher in the sky, etc. Would that even be an issue that could possibly arise from the NVME overheating? I have a different camera than the thread starter, my camera is an SVBony SV705C. |
Beta Was this translation helpful? Give feedback.
-
|
For what its worth, I do also have the Geekom, bottom hat for single NVMe drive. However, I have not used mine outside much -- only a month or two total. Most of the time, I've been runing it on my workbench at room temp while I work out the bugs, including addition of heaters and humidity sensor.The hangups appear more correlated to my exposure settings than temp right now. Also, I notice the pi5 does not completely hang and is not restarted by the watchdog, so its something related to Indi AllSky and/or camera hardware chain causing it -- my humble opinion. When the camera hangs I can always reboot the pi via web interface and it temporarily seems to fix the issue. Anyway, as suggested above, I completed a comparison of the support info pre and post Indi AllSky update -- which is when the camera freezes began for me. I was able to do this because I made an SSD copy a few days before I did the upgrade. There's a lot in the support info I don't know how to interpret. ChatGPT offers the following assessment, but I don't know if any of this is really relevant.
I'm attaching the full pre and post upgrade support info. Suggestions on how to resolve this will be much appreciated. Regarding hardware, I tried swapping the camera cable multiple times -- I had a spare -- but it makes no difference. I have a spare camera coming (pi HD) and I'll swap that next to see if that helps. I don't have high hopes that this is just a camera issue. I've also ried multiple pi power supplies and no difference. |
Beta Was this translation helpful? Give feedback.
-
|
All, Disclaimer: I'm a newby at linux and the pi so I apologize in advance if I'm making dumb mistakes. I'm continuing to track down the reason for my HD camera hanging up. Its somewhat random but is very repeatable. It won't run more than a few minutes without a hangup. This all started with an Indi-AllSky and os update in May. If I return to the Dec 24 version (saved on an extra SSD) the hangups do not happen. When I return to the May or newer code saved on my operational SSD, the hangups immediately return. I've swapped hardware -- camera and cables -- but no help at all. It does seem somehow tied to exposure settings. Maybe I have something wrong in the AllSky configuration Shorter exposure better, longer settings at night worse, meaning it hangs up sooner/more frequently. However I cannot find a setting that eliminates hangups. I enabled logging and it appears to me there are some coding errors somewhere. These may be caused by me misconfiguring something but I can't figure out what/where. To summarize, I see the follwing 2 repeating error instances:
System reboot seems required to solve this issue. The watchdog does not currently restart the process but even if it did I think this is happening so often, the pi would reboot several times per minute and not be useful. What I'm Trying:
I'm attaching a better log of the errors encountered. Again, any ideas/suggestions would be very helpful. I'm so close to completing this project if I could just figure out this issue. v/r, |
Beta Was this translation helpful? Give feedback.
-
|
I would certainly be nice to know if others are actually reporting similar issues. People using the camera module for all sky systems would likely be the only people using long exposures like this. I have not noticed any reports on the [TJ] Allsky issues. My guess is that this is somehow related to the libcamera upgrade to 0.4 or 0.5. I am not sure back leveling only libcamera is enough to fully downgrade. The kernel also has overlays for the camera module which could affect stability as well. |
Beta Was this translation helpful? Give feedback.
-
|
After reviewing the last update, I think part of the problem might be due to you using the libcamera auto-white balance AWB. libcamera AWB has a pretty nasty side effect when using an all sky camera at night. Effectively, how libcamera performs the white balance is by taking several exposures to determine the best white balance point. If you are taking 100 second exposures, this means libcamera will take several 100 second exposures, likely taking more than 300 seconds (5 minutes) to complete a single 100 second exposure. indi-allsky has a hard coded limit of 300 seconds, so if the exposure takes longer than 300 second, it thinks the camera has hung and tries to abort the exposure. I can work on making the limit dynamic where it can be changed in the configuration. |
Beta Was this translation helpful? Give feedback.
-
|
Merged #2187 to make the timeout configurable. The timeout is near the other exposure settings on the Camera tab. You may want to set a timeout of 8-10 (480-600s) with a long exposure like 100s. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you. Your hypothesis does make sense. I'll check it out. Do most people run without AWB? If I make that change do I need to make other changes to maintain approximately the correct white balance in long exposures or is that just not possible? As mentioned, I put this issue on the libcamera page in git hub -- no response as yet but I'll post here if I get feedback. Regarding the Type Error, all I have is the log and I provided most of it in this message. Is there a way I can track down what's generating the error? Perhaps a way to make the log more verbose? I may have entered an incorrect value someplace but I have no idea how to find it. Does "must be real number, not NoneType" mean I have a blank field where there should be a real number? If that's the way to interpret it, then I can look for empty fields everywhere I can enter data -- at least a starting point. I have not noticed problems other than the camera hang, but I've been working on that issue for several weeks. If I can solve it, I'll likely find the next issue. One layer of the onion at a time! |
Beta Was this translation helpful? Give feedback.
-
|
As for AWB... I have a feeling many people do not realize the behavior is even happening. Disabling the AWB does create some challenges. I am afraid I have not been able to fully replicate the colors of the libcamera AWB, but I can replicate decent white balance. I need to write a wiki page on tuning the image. |
Beta Was this translation helpful? Give feedback.
-
|
The explanations are helpful. As a next step I turned off AWB, set day&night max exposure to 22 (no great logic to 22 - just picked a lower number that could be sufficient in bench testing) and relaunched allsky. So far, I see little difference in the daytime image but its only been running about 10-min. If its going to hang it should do so within the hour. If it does hang again I'll try to capture more log information. I'm hoping I can reduce exposure and increase the timeout durection via the new parameter you added to solve it. We'll see. I have not yet upgraded to get the new parameter. That'll be next. |
Beta Was this translation helpful? Give feedback.
-
|
As mentioned, I posted on the libcamera Github asking for help on this issue to see if similar hangups had been reported elsewhere. Today I received the following response from the libcamera dev team: Thank you for the report. This may or may not be related to another issue we are investigating: raspberrypi/rpicam-apps#799. Are you able to provide logs wihtout the Allsky application running to check this? I will attempt to work through their suggestions and if I can nail it down to a libcamera issue I'll report back here. I may be over my head troubleshooting this :) |
Beta Was this translation helpful? Give feedback.
-
|
I do not believe that issue is related. |
Beta Was this translation helpful? Give feedback.
-
|
I upgraded AllSky today and adjusted the new exposure timeout parameter. Since then, I have been to run almost 12hrs without a hang up -- this is good because in the past I would hang up (with or without error messages) with an hour. Progress I think! However... Although the system doesn't l hang, I am getting the below message periodically. I aknowledge it and the system seems to churn ahead OK. I see no issues with the live image, but I don't know if any of the image products are impacted -- I would expect some issues or artifacts in timelapse at least but I don't know for certain. I have not been able to check these is the short time the system has been up. Plus I'm still testing on my workbench. I don't know what's causing it but I suspect I still don't understand the exposure controls and have something worng in config/camera. I'm including a snap of my exposure settings. I'd like to take advantage of the high exposure capability if the system can handle it. When it works, it provides impressive dark sky images for a cheap camera. If 100s is just too much, I could reduce it and enable stacking to help compensate, but I understand that can introduce registration challenges. The longer exposures, if they work, seem a more straight forward way to pull out those dim objects. I know the wiki is being updated, but do my exposure settings make sense, or could these cause the error message here? I picked the high value for exposure timeout just to eliminate that issue as a cause. Is there a problem if I set this too high, other than missed imaging opportunity? Can the hardware be damaged?
As always, assistance greatly appreciated. Attaching support info in case its useful. |
Beta Was this translation helpful? Give feedback.
-
|
The Exposures returning too quickly happen from time to time, I am not sure there is much you can do about it. I would just ignore it unless it becomes a larger problem. I do not see anything particularly wrong with your settings. It appears everything is generally working like you want it to. |
Beta Was this translation helpful? Give feedback.
-
|
That's good news. Yes, this may have fixed the issue. The system has run about 24hrs without a hangup. In the past it would have stopped at least once an hour, so definitely progress. I don't understand why it wasn't hanging on the earlier ~Nov/Dec versions of Indi-AllSky with libcamera 0.3y. The config parameters are the same and that earlier version appears rock solid. But, if the latest version remains solid now I'll take the victory. |
Beta Was this translation helpful? Give feedback.
-
|
I successfully ran Indi-AllSky for over 24 hrs since the update. I had 1 allsky hangup (no message) with exposure at 120-sec. Out of caution, I lowered the exposure to 100 sec and rebooted. No hangups since. Don't know if exposure had anything to do with any of it. I also ran a test loop to repeatedly take 100s exposures over and over for 1 hr. I ran that loop 8x with no hangups. I'm still getting occassional exposure finished early notices but they're not causing much problem. No impact to performance as far as I can tell. Other than that one instance, I'm pretty happy with the system. Progress has definitely been made. I'll probably continue to monitor -- see how it goes. Not sure what else to do anyway. v/r, |
Beta Was this translation helpful? Give feedback.
-
|
I am glad to hear it! |
Beta Was this translation helpful? Give feedback.
-
|
Update: I’m told a bug causing hangups in some instances was tracked down and resolved in the most recent version of libcamera. Since updating this past August I’ve had zero hangups. In my case hangs had become infrequent, maybe once a week. Still, it’s been about 5 weeks with no issues so I’m optimistic the issue is resolved. |
Beta Was this translation helpful? Give feedback.
-
|
Great news! |
Beta Was this translation helpful? Give feedback.
Merged #2187 to make the timeout configurable. The timeout is near the other exposure settings on the Camera tab. You may want to set a timeout of 8-10 (480-600s) with a long exposure like 100s.