Autoexposure on RasPi Camera

[ArtiGiny]

On last version and on Night the autoexposure doesn't works on RasPi Cameras. The exposure is less than 1s.
If config like AutoGain and AutoExposure only change the Gain and exposuretime is always 0,925s and the pictures are too much noise.
In the version of 2022 works perfectly the AutoExposure and Autogain and the result was a well definition pictures.
Do you have any idea to resolve it?
I try it a lot of configs and the results always are a poor images.
There are a way to install a old version fo the allsky program?

[EricClaeys]

@ArtiGiny, would you please attach your ~/allsky/config/settings.json file? I suspect your nighttime Max Auto Exposure is causing the problem.

[ArtiGiny]

Sorry for my delay answer.
I attach my settings.json file.
settings.json

[EricClaeys]

@ArtiGiny, the Version 1 camera has a max exposure of 0.97 seconds. If you hover your mouse over the Exposure field in the WebUI it should say that.
That camera isn't very good for an allsky camera.

Eric

[ArtiGiny]

@EricClaeys, the last stable version of allsky I had (the one I installed around April 2023) worked perfectly on this same hardware I'm using. Before I updated the allsky I was doing about 30 second exposures and Gain 1.
Maybe the camera isn't the best, but it used to work and now it doesn't. The problem comes from another side.
There must be some restriction that wasn't there before and is there now.

[EricClaeys]

@ArtiGiny, interesting. Support for that camera was added in v2023.05.01_03 from Aug 30 with the same max exposure. Do you still have the release where it worked?
Are you using the same Pi operating system? e.g., Bullseye?

[ArtiGiny]

@EricClaeys, I think I'm using the same OS . I actually did the install on a RasPi Zero, and once I saw that it worked, I put the SD to a RasPi 3B+ (and works perfectly). But I did use the "Little OS" version and now I use the "with desktop OS" version and do it directly to RasPi 3b+ platform.

Unfortunately I don't have the SD cloned (which I always do, but this time I thought everything would work well), so I don't have the working version that was work.

One of my intentions was to try to roll back the release with a git checkout (version 2023.05.01 or before 2022.03.01), but I haven't managed that work from this point (in this case give some error after run the script ./install.sh).

[EricClaeys]

With Allsky stopped, run

time libcamera-still --shutter 60000000 --metadata x.json

and look at the 'real" time to see how long the command ran. If 60 seconds, look in x.json to see how long the shutter was. If 60000000 or so, try a 200000000 ((200 second) image.

I am trying to determine if the 9 second advertised max exposure is accurate.

[ArtiGiny]

when you talk about x.json where is it located?
Or it will give me the final value of the "time" function. This value is the value that you need?
I'll try this tonight if I can, I don't have access to the allskycamera right now

[EricClaeys]

The time command runs the command after it, in your case libcamera-still, and tells you how long it took to run. Basically a stopwatch.

x.json will be in whatever directory you are in when you ran the command. If you don't know how to interpret it then attach it to your post and I will look.

[ArtiGiny]

@EricClaeys I'm sorry but i can't find the file x.json.
But I attach the screenshot for 60000000 and for 200000000. (Image of 60000000 the results of time function is the time appear over the function. I did twice for each time-function and take the screenshot with the result and for the header of the function)
I hope that can use this information.

[EricClaeys]

@ArtiGiny, the screenshots didn't get attached. GitHub doesn't accept attachments via email so you'll need to log into GitHub to attach them. If the screenshots are just what time returned you can just tell me what the real time was. The output from time looks like this:

real	0m0.004s
user	0m0.001s
sys	0m0.003s

Note that libcamera-still produces a bunch of useless output.

Since x.json didn't get created, please try the following:

export LIBCAMERA_LOG_LEVELS=ERROR,FATAL; time libcamera-still --nopreview --timeout 1 --shutter 60000000 -v

and copy/paste the output to your reply.

[ArtiGiny]

This are the picture. Sorry.

And the new one:

allsky@allsky:~ $ export LIBCAMERA_LOG_LEVELS=ERROR,FATAL; time libcamera-still --nopreview --timeout 1 --shutter 60000000 -v
Options:
verbose: 2
info_text:
timeout: 1
width: 0
height: 0
output:
post_process_file:
rawfull: 0
preview: none
qt-preview: 0
transform: identity
roi: all
shutter: 6e+07
metering: centre
exposure: normal
ev: 0
awb: auto
flush: false
wrap: 0
brightness: 0
contrast: 1
saturation: 1
sharpness: 1
framerate: 30
denoise: auto
viewfinder-width: 0
viewfinder-height: 0
tuning-file: (libcamera)
lores-width: 0
lores-height: 0
autofocus-range: normal
autofocus-speed: normal
autofocus-window: all
mode: unspecified
viewfinder-mode: unspecified
metadata:
metadata-format: json
encoding: jpg
quality: 93
raw: 0
restart: 0
timelapse: 0
framestart: 0
datetime: 0
timestamp: 0
keypress: 0
signal: 0
thumbnail width: 320
thumbnail height: 240
thumbnail quality: 70
latest:
immediate 0
AF on capture: 0
Running without preview window
Opening camera...
Acquired camera /base/soc/i2c0mux/i2c@1/ov5647@36
Configuring viewfinder...
Viewfinder size chosen is 1296x972
Camera streams configured
Available controls:
NoiseReductionMode : [0..4]
ScalerCrop : [(0, 0)/128x128..(0, 0)/2592x1944]
Sharpness : [0.000000..16.000000]
AwbEnable : [false..true]
AeEnable : [false..true]
ExposureTime : [92..0]
AeConstraintMode : [0..3]
Brightness : [-1.000000..1.000000]
ColourGains : [0.000000..32.000000]
AnalogueGain : [1.000000..63.937500]
Contrast : [0.000000..32.000000]
AeMeteringMode : [0..3]
Saturation : [0.000000..32.000000]
AeExposureMode : [0..3]
AwbMode : [0..7]
ExposureValue : [-8.000000..8.000000]
FrameDurationLimits : [23123..760729]
Buffers allocated and mapped
Viewfinder setup complete
Requests created
Camera started!
Viewfinder frame 0
Closing Libcamera application(frames displayed 0, dropped 0)
Camera stopped!
Tearing down requests, buffers and configuration
Camera closed

real 0m1,959s
user 0m1,263s
sys 0m0,363s
allsky@allsky:~ $

Thanks.

[EricClaeys]

@ArtiGiny, thanks for the info.

libcamera-still is limiting the exposure to the camera's documented limit of less than a second. Notice that you asked for a 60-second exposure and it finished in less than 2 seconds. There is some overhead in running the command and saving the file so in total it took more than a second.
If you were on Bullseye and are now on Bookworm it's possible that Bullseye didn't check for the exposure limit and Bookworm does.
Please run cat /etc/os-release.
It's also possible an update to libcamera started enforcing the exposure limit.

If you want to find out when the change occurred try posting a question on the rpicam team GitHub page.

[ArtiGiny]

I'll check it and tell you de OS release (I use a bulleyes)
Thanks for your time.

[ArtiGiny]

Dear @EricClaeys .
Just today is my first day after Easter days. Have a nice Easter Time.

There any way to install a old version of the allsky software? Is to say, I would like to intall the version of 2022-03-01 or the 2023-05-01? For to check if the problem comes from allsky or from the OS.

Could you tell me the steps for to install one of the OLD-version of allsky?

[EricClaeys]

@ArtiGiny, Pep,
You can try installing an Older Allsky version but may have problems unless you first reimage the Pi, since newer versions of Allsky install newer packages that may not work with older Allsky versions.

To install an older Allsky, click on the + 9 releases link on the right side of the main Allsky GitHub page. Scroll down to the release you want and click ion the zip link under Assets and copy to the Pi then install as usual.

[EricClaeys]

@ArtiGiny, Allsky uses libcamera-still to take the pictures, so if it doesn't work, no version of Allsky will either.

[ArtiGiny]

@EricClaeys I tried to install version 2022.03,01 and the version 2023.05.01 following the process that you told me. But in both cases gave the next errors (please see the pictures attached).
[the IMG_20240403.... is when i tried to install v2022.03.01. IMG_20240404... when I tried to install v2023.05.01]
I tried to install over a RaspBian OS bulleyes on old image that I conserve (May of 2022).
Do you know how I cross error?

[EricClaeys]

I gave you the wrong instructions - sorry. This is a rare thing to do so I don’t have the instructions memorized.
Run the following, replacing XXXX with the release you want, like v2022.03.01:

cd
mv allsky  allsky-BAD
git clone --recursive --depth=1 --branch XXXX https://github.com/AllskyTeam/allsky.git
cd allsky
./install.sh

[ArtiGiny]

Dear @EricClaeys .
I tried the v2022.03.01 but I don't know how to follow for ending to setting the version (I never config this older version)
For other hand I tried to install the v2023.05.01 version, and appear this problem:

This is the problem of the new lib?
There are any way to create the cc.json file to continue the installation?
When I installed the last version (v2023.05.01_04) not appear this issue.

[EricClaeys]

Support for the ov5647 sensor was added after v2023.05.01 - I don't remember which version, but support is in v2023.05.01_04 which is why you didn't get a warning in that release.
Version v2022.03.01 only supported the RPi HQ camera so it didn't check what camera you had. It may work with the ov5647 camera.

To update settings for all releases in the past several years you use the WebUI - either the "Camera Settings" page (also called "Allsky Settings" in some releases), or the "Editor" page to update config.sh and ftp-settings.sh. The Documentation explains what all the settings are.

If you are using the same operating system as when the camera had a short exposure limit, the version of Allsky won't change that.

[MichelMoriniaux]

maybe I've found the solution for this: was having the same issue on the imx477 on a new build:
it seems that in the latest version of rpicam-apps the --timeout 1 flag forces the exposure to 1us. If you replace that flag by --immediate the image is properly exposed
I have not looked at the theory in more detail but that is the only library difference between my working skycam and this new cam :
working rpicam-apps 1.4.3-1 not working 1.4.4-1

[ArtiGiny]

Dear @MichelMoriniaux ,
How and where I can change this flag?
This weekend I had been upgrade it the OS and I want to try your comment.
Thanks,

[ArtiGiny]

@MichelMoriniaux
I attached my "settings.json" file.
settings.json

[MichelMoriniaux]

See issue #3657 it's not a configuration option, you need to modify capture_RPi.cpp ,recompile and copy the new executable to the bin directory

[ArtiGiny]

@MichelMoriniaux
I follow and change like propose on issue #3657 but give me this issue when I refresh on WebUI:

I replaced the timeout command in capture_RPi.cpp by --immediate you could show it on my cature_RPi.cpp file (attached)
capture_RPi.cpp.txt {I put txt at the end of the file for to share with you}

Or do I need to do other steps?
Thanks

[MichelMoriniaux]

If you have that error on the webui it's that you modified functions.sh and you should not have. Only modify capture_RPi.cpp and run make then copy the new executable to allsky/bin

…

On Mon, May 27, 2024, 10:46 Pep Cirera ***@***.***> wrote: @MichelMoriniaux <https://github.com/MichelMoriniaux> I follow and change like propose on issue #3657 <#3657> but give me this issue when I refresh on WebUI: imatge.png (view on web) <https://github.com/AllskyTeam/allsky/assets/162619268/54c13770-b3bd-4e27-9a07-0ce5c3620535> I replaced the *timeout* command in *capture_RPi.cpp* by *--immediate* you could show it on my cature_RPi.cpp file (attached) capture_RPi.cpp.txt <https://github.com/AllskyTeam/allsky/files/15458645/capture_RPi.cpp.txt> *{I put txt at the end of the file for to share with you}* Or do I need to do other steps? Thanks — Reply to this email directly, view it on GitHub <#3511 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADBS7WTBENPEDWPOLZKOFPTZENWQNAVCNFSM6AAAAABEZXTKLCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKNZSG43TA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[MichelMoriniaux]

Also --immediate does not take an argument so you need to remove the + ss.str() you left in there Set debug to 2, the logs will tell you what is wrong most of the time

…

On Mon, May 27, 2024, 10:46 Pep Cirera ***@***.***> wrote: @MichelMoriniaux <https://github.com/MichelMoriniaux> I follow and change like propose on issue #3657 <#3657> but give me this issue when I refresh on WebUI: imatge.png (view on web) <https://github.com/AllskyTeam/allsky/assets/162619268/54c13770-b3bd-4e27-9a07-0ce5c3620535> I replaced the *timeout* command in *capture_RPi.cpp* by *--immediate* you could show it on my cature_RPi.cpp file (attached) capture_RPi.cpp.txt <https://github.com/AllskyTeam/allsky/files/15458645/capture_RPi.cpp.txt> *{I put txt at the end of the file for to share with you}* Or do I need to do other steps? Thanks — Reply to this email directly, view it on GitHub <#3511 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADBS7WTBENPEDWPOLZKOFPTZENWQNAVCNFSM6AAAAABEZXTKLCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKNZSG43TA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ArtiGiny]

I do thew changes that you propose, remove the +ss.str() and set Debug (on WebUI) on 2 and gives me the next screenshot:

Then I do first the debug change and save. And then the refresh on camera type and give the next error:

[MichelMoriniaux]

Logs do not appear in the web UI but in the allsky log file. /var/log/allsky.log At this point you should revert all your changes and try the commands manually and verify they work for your setup. It will prevent further breakage.

…

On Mon, May 27, 2024, 11:28 Pep Cirera ***@***.***> wrote: I do thew changes that you propose, remove the +ss.str() and set Debug (on WebUI) on 2 and gives me the next screenshot: imatge.png (view on web) <https://github.com/AllskyTeam/allsky/assets/162619268/13f63491-7603-46ca-95be-cd6dcdc8ea7e> Then I do first the debug change and save. And then the refresh on camera type and give the next error: imatge.png (view on web) <https://github.com/AllskyTeam/allsky/assets/162619268/ae7dbd2e-e9bf-4ed7-acb6-c0b2dcf89dd9> — Reply to this email directly, view it on GitHub <#3511 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADBS7WRV6IKVYSDUGORY37TZEN3L7AVCNFSM6AAAAABEZXTKLCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKNZTGAYTQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ArtiGiny]

I revert the original capture_RPi.cpp file and make again and copy the new executable on /allsky/bin directory.
Then I check manually and the results are the next screenshot:

[EricClaeys]

@ArtiGiny, any update? Did you figure out why it said "no cameras available"?

[ArtiGiny]

@EricClaeys I find somewher that you suggest to change a file xxx.cpp and make it. And now detect.
but at the end I changed the camera to an IMX219 sensor and yes, it does exposures of 11.2s but it is not enough and playing with the Gain. The recommended IMX7-- are too expensive for me.
In one point of discussion I saw that you considered being able to do multi-exposures to get more detail as it was done in the past (Alignment should be by frame fixed, since it does not rotate, moving stars, but this will always happen) . I think it would be a very good choice.
This will be an option that should be made clear if you want to have more star identification will be lost in bolide detection, and vice-versa if you want to detect bolides you will not have as much exposure.
From inside the city, before, I could see and recognize in the pictures I made (or time-lapses) (on OM47-- sensors) the constellations that were passing by. Currently I can only detect high magnitude stars like Arcturus or Altair.

[EricClaeys]

@ArtiGiny, we are unlikely to implement image stacking, at least in the next couple years. It would be a lot of effort and we believe few people would use it.

Have you tried stretching your images? You will need to play with the settings to find something that works. Look at the Documentation page related to stretching for examples.

[ArtiGiny]

@EricClaeys , thanks a lot,
I'll try playing with the stretch values.
I keep thinking it's curious that before with the OV5647 it worked perfectly and when I updated to the latest allsky version it stopped doing exposures of 30s to do them, now, of 1s maximum.

[EricClaeys]

The libcamera software that Allsky uses probably changed and now enforces the maximum exposure length. It does seem strange that it worked at the higher-than-rated speed though.

[EricClaeys]

@MichelMoriniaux, did you try adding --immediate to the Extra Parameters setting rather than changing the code?

[stevedee]

I have the same situation with my new system fitted with a V1 PiCam on a Pi 4.

As I'm new to this, I'm desperately trying to get familiar with this system before the big-meteor-show next weekend.

And as I don't know if the Module 3 PiCam on order will be delivered in time, can you explain how I set --immediate in Extra Parameters as mentioned? (i.e. give me the full string, and tell me where to stick it!)

I'd like to give it a try, but don't want to modify code, which I think was the other option.

Many Thanks!

[EricClaeys]

@stevedee, in the "Allsky Settings" page in the WebUI, look for the Extra Parameters setting and set it to --immediate, then press the "Save changes" button.

Several problems have been discussed in this thread. Exactly which "same situation" are you seeing?

[stevedee]

Sorry, reading right through this thread once again I don't think this will help me.

I got the impression that with earlier versions of code/drivers, the PiCam V1 limit was ignored and somehow the camera could be forced to take long exposures. But it seems that this situation is not realistic, it could never have been the case.

So I'm stuck with a max exposure time of 969.249ms until my new camera arrives!

Thanks once again. Great project by the way.

[EricClaeys]

@stevedee,
I don't have one of the "short exposure" cameras so could never test, but I gather that prior versions of the software may have allowed longer exposures, i.e., they ignored the specified maximum.
But Allsky and the current libcamera software enforce the specified maximum exposure.

Enjoy the new camera once you get it!

[MichelMoriniaux]

Good call, I will try that and report back. Might be a few days as I need to drive up to the observatory to get it.

…

On Mon, May 27, 2024, 13:44 Eric Claeys ***@***.***> wrote: @MichelMoriniaux <https://github.com/MichelMoriniaux>, did you try adding --immediate to the Extra Parameters setting rather than changing the code? — Reply to this email directly, view it on GitHub <#3511 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADBS7WSO64JX3ALGNU5JYOLZEOLJBAVCNFSM6AAAAABEZXTKLCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKNZTGY4TQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ArtiGiny]

In the end I changed the camera to an IMX219 sensor and yes, it does exposures of 11.2s but it is not enough and playing with the Gain. The recommended IMX7-- are too expensive for me.
In one point of discussion I saw that you considered being able to do multi-exposures to get more detail as it was done in the past (Alignment should be by frame fixed, since it does not rotate, moving stars, but this will always happen) . I think it would be a very good choice.
This will be an option that should be made clear if you want to have more star identification will be lost in ball detection, and vice-versa if you want to detect bolides you will not have as much exposure.
From inside the city, before, I could see and recognize in the pictures I made (or time-lapses) (on OM47-- sensors) the constellations that were passing by. Currently I can only detect high magnitude stars like Arcturus or Altair.