Startrails - improve use and debugging

[EricClaeys]

Startrails work for most people, but some people struggle getting trailed stars. The startrails documentation along with the allsky-config get_brightness_info command help users determine what Brightness Threshold to use, but some people still struggle.

This Issue is a catch-all for ways to improve startrails generation and make it even easier to find the right settings to use.
See this Discussion for many more posts on this topic.

• 1. Add an option in the WebUI to only look at nighttime images. This would use the capability in # 2 below to create a list of nighttime-only images. The default should probably be "Yes", although in general I don't like adding things that "break" existing functionality, which this could do if a user's startrails used some daytime images due to their Angle, Latitude, and Longitude. I think this is not very likely though. A correctly-set Brightness Threshold should eliminate daytime image, but only looking at night images will make processing go faster (possibly twice as fast). Accepting a list of images would also make it easy to implement things like a user-specified start and stop time for image to process.

• 2. Add an option to pass a list of images to process to the startrails command, rather than process all images for a 24-hour period. This would also allow using images from multiple nights or directories. The images in the list would default to a specified directory unless the image name began with / in which case it would signify a full pathname to the image.

• 3. Add allsky-config compare_startrails to create a list of 50 or so nighttime images, then call the startrails command with a bunch of different Brightness Thresholds using that image list, so a bunch of different startrails are created for the user to look at and compare. It could even let the user know which setting resulted in the most images being used. This would automate what the startrails documentation tells the user to do manually.
  A WebUI "helper" page would have a form where the user would enter the desired test settings. Submitting the form would create the startrails and display a link to view them.

• 4. When created at end of night, if no images were used, i.e., none were below the threshold, the startrails command should exit with code EXIT_PARTIAL_OK which results in a System Message in the WebUI suggesting the user run allsky-config compare_startrails (or go to the web page described in # 3 above). This would make it easy for users to know what happened, and if we're lucky, reduce the number of "Startrails don't work" posts here and in Facebook (which almost all have the same, canned answer...).

• 5. UPDATE: this probably isn't needed if the "mean brightness" generated while capturing the image is passed to the startrails command, since the brightness determined by the startrails command when using a mask should be the same as the "mean brightness".
  Add an option to the WebUI for the user to specify an optional mask the startrails command would use when determining the average brightness of an image, and hence whether or not to include the image in the startrails. Currently the whole image is used in calculating the average brightness, which means the brightness of buildings, trees, black borders, and other non-sky objects are included. This in itself probably isn't that bad, but it means the average brightness of an image can't be compared to the Target Mean setting, which uses only the central portion of an image (which is likely to be only sky).
  A future Allsky release will allow users to specify a mask that the auto-exposure algorithm would use when determining mean brightness, and that same mask could be used by the startrails command, thereby allowing the Brightness Threshold to be compared to the Target Mean.

• 6. For each startrails created at the end of night, save the data it produces (Minimum, Maximum, number of images used, etc.). This would allow looking for trends (what impact does the Moon have on the numbers?) and would allow creating a video of all startrails using daily startrails images that have trailed stars. The allsky-config get_startrails_info command could use that list to determine better average numbers - it currently only looks in /var/log/allsky.log` which may only contain a few entries (and for the first day after an Allsky upgrade it won't contain any).

• 7. UPDATE: this isn't needed when # 8 below is implemented, since the database entries will have the fields listed in this item.
  When called from endOfNight.sh, have the startrails command save information about every image it looks at, including the mean brightness and brightness threshold. It would save the info to a file in the same directory as the startrails file, i.e., ~/allsky/image/DATA/startrails, rather than in the database to keep it from getting too big. This would be handled in the generateForDay.sh script.

• 8. Information on every image taken should be saved in the new database. This includes exposure time, DAY / NIGHT flag, mean brightness, and other info. Every module like Star Count that creates per-image data should update the image's database entry. When determining which images to include in a startrails, the database would be queried for all images that meet certain criteria. That list would then be given to the startrails command per number 2 above.

[ova103]

Hello Eric
I like the first one (why would you use day time images for startrails ...). Your concern you have with this one is solved if you allow a user to keep the day images for startails. I thinks limiting to night images only will solve most of the problems.

I also like the last one, but I would add a file with the list of all images of the day/night, indicate if they have been used or not, and add the average brightness (and treshold brightness)

Another idea is to add on the overlay a field that mentions if the image is used for the overlay and a field for the average brightness of the image (and the treshold brightness) . So you can scroll through your images and learn from it.

Hope this helps.

[EricClaeys]

@ova103, thanks for the comments.

why would you use day time images for startrails

In Allsky, "daytime" and "nighttime" are defined by the Longitude and Angle and define when the "Daytime" and "Nighttime" settings should be use, and are only loosly the same as what we generally think of as "daytime" and "nighttime".
I'm helping a user with an unrelated problem and noticed his Angle is -12 which means it's what most people would call "dark" at the beginning and end of his "daytime" for around half an hour, and images in those periods probably contain valid stars for a startrails. Using only "nighttime" image means he'd probably get an hour's worth less stars in his startrails.
It's cases like this that make me hesitant to have the default be "Use only nighttime images".

To implement the first change I'll probably have the saveImage.sh script, which does all the work after a raw image is saved like cropping, adding the overlay, and uploading the file, save the name of the image in a "DAY" or "NIGHT" file. If only nighttime files are included in a startrails, then the "startrails" command would be given the "NIGHT" list to process.

---

For the last one, I already changed the "startrails" command in the next release to include the Brightness Threshold in it's output along with the Minimum, maximum, etc. When you said:

indicate if they have been used or not,

do you mean "indicate if nighttime only, or all images" are used, or indicate for each image the "startrails" command looks at, whether or not it is included in the startrails, i.e., if it's above or below the threshold? I considered the later, but if a user has 3000 images that's a lot of data to keep. For debugging I can see where it would be useful to have an option to the "startrails" command that says "for each file, indicate if it was used or not".

[EricClaeys]

@ova103,
Thanks again for the input. I was hoping for some back-and-forth like this - it helps improve the final result.

add on the overlay a field that mentions if the image is used for the overlay and a field for the average brightness of the image (and the treshold brightness)

Should "if the image is used for the overlay" have been "if the image is used for the startrails" ?

I spent about 20 minutes writing a reply to this, figuring out how it could be done, then realized adding this info to the overly can't be done. The overlay is created immediately after an image is taken, but whether or not that image was used in a startrails isn't calculated until the end of night - too late for the overlay.
However, in order to help "learn from it", I just added change # 7 in my original post to have the startrails command save information about every image. We could then have a command (possibly an option to allsky-config get_brightness_info) that displays that information.

What are your thoughts on HOW to display the information so it would be useful to a user?

The ${MEAN} variable can be added to the overlay now - it's the average brightness (aka "mean") of the image as measured by the auto-exposure algorithm. The startrails command currently calculates the mean brightness of an image differently so its mean and the ${MEAN} of the image can't be compared. Change # 5 in my original post deals with having the startrails command compute the brightness the same way as the auto-exposure algorithm so the numbers are the same.

Allsky doesn't currently save information on each image, but if it did and it saved the MEAN value, then the startrails command could look at the save information and compare each image's saved MEAN value to the Brightness Threshold, rather than recompute the mean. If it did this, then the overlay COULD indicate if an image is going to be included in a startrails - it would simply look at the MEAN value and compare to the Brightness Threshold.

[ova103]

@ova103, thanks for the comments.

why would you use day time images for startrails

In Allsky, "daytime" and "nighttime" are defined by the Longitude and Angle and define when the "Daytime" and "Nighttime" settings should be use, and are only loosly the same as what we generally think of as "daytime" and "nighttime". I'm helping a user with an unrelated problem and noticed his Angle is -12 which means it's what most people would call "dark" at the beginning and end of his "daytime" for around half an hour, and images in those periods probably contain valid stars for a startrails. Using only "nighttime" image means he'd probably get an hour's worth less stars in his startrails. It's cases like this that make me hesitant to have the default be "Use only nighttime images".

Maybe another idea: instead of saying you want to use Night images only, you could say if you want the use the images of astronomical twilight only, or also the ones from nautical, or also the one from civil, or also the one of the day. So it means you specify a 2nd angle for the usage of the images for the strartrail.

For the last one, I already changed the "startrails" command in the next release to include the Brightness Threshold in it's output along with the Minimum, maximum, etc. When you said:

indicate if they have been used or not,

do you mean "indicate if nighttime only, or all images" are used, or indicate for each image the "startrails" command looks at, whether or not it is included in the startrails, i.e., if it's above or below the threshold? I considered the later, but if a user has 3000 images that's a lot of data to keep. For debugging I can see where it would be useful to have an option to the "startrails" command that says "for each file, indicate if it was used or not".

I mean indicate for each image the "startrails" command looks at whether or not it is included in the startrails. To reduce the amount of data it could for example list only the images that have been used to make a startrail.

[ova103]

@ova103, Thanks again for the input. I was hoping for some back-and-forth like this - it helps improve the final result.

add on the overlay a field that mentions if the image is used for the overlay and a field for the average brightness of the image (and the treshold brightness)

Should "if the image is used for the overlay" have been "if the image is used for the startrails" ?

Yes indeed, it should be if the image is used for the startrails"

I spent about 20 minutes writing a reply to this, figuring out how it could be done, then realized adding this info to the overly can't be done. The overlay is created immediately after an image is taken, but whether or not that image was used in a startrails isn't calculated until the end of night - too late for the overlay. However, in order to help "learn from it", I just added change # 7 in my original post to have the startrails command save information about every image. We could then have a command (possibly an option to allsky-config get_brightness_info) that displays that information.

What are your thoughts on HOW to display the information so it would be useful to a user?

The ${MEAN} variable can be added to the overlay now - it's the average brightness (aka "mean") of the image as measured by the auto-exposure algorithm. The startrails command currently calculates the mean brightness of an image differently so its mean and the ${MEAN} of the image can't be compared.

I did not realize $(Mean) is not same as the Mean value used by stratrail command. Would be helpful to specify this in the documentation

Change # 5 in my original post deals with having the startrails command compute the brightness the same way as the auto-exposure algorithm so the numbers are the same.

That makes a lot of sense.

Allsky doesn't currently save information on each image, but if it did and it saved the MEAN value, then the startrails command could look at the save information and compare each image's saved MEAN value to the Brightness Threshold, rather than recompute the mean. If it did this, then the overlay COULD indicate if an image is going to be included in a startrails - it would simply look at the MEAN value and compare to the Brightness Threshold.

Yes that looks very useful. Additional info to save could be if it's DAY or NIGHT time, and the angle of the sun.

[EricClaeys]

@ova103,

instead of saying you want to use Night images only, you could say if you want the use the images of astronomical twilight only

On our to-do list is allowing a user to specify a start and stop time for images to include in startrails, keogram, and timelapse.
The "times" could be an angle, an actual time, like 6:21 pm, or a time relative to sunrise or sunset, and maybe astronomical, civil, or nautical. The last 3 are already relative to sunrise and sunset so probably aren't needed, but might be easier for a user to understand, especially if the do astrophotography since some software uses those terms.
We'll need to think about how to present this in the WebUI so it's not too confusing and doesn't add a ton of new settings.
The current Angle setting is really just an offset from sunrise and sunset expressed in degrees, so in theory it could be replaced by the more general "times" above.

Having said all that, maybe the best thing for the next release is to simply add a "Use Nighttime Images Only" flag for startrails, and get user feedback on whether or not more granularity is needed.

[NiteRide]

perhaps it may be neat to be able to exclude images that have airplane light trails?

Don't need to solution it here, but maybe could be implemented by detecting lines across an image? not sure it a negative or not, but could potentially end up also excluding satellite or meteors as well, but since you are collecting ideas...

[brianboru82]

It would be nice to be able to use the -b brightness flag with the generateForDay.sh command for startrails testing, so one could easily try different settings for a given day. Maybe have it add the brightness level as a suffix for that startrail image? like startrails_20250729_0p15.jpg for a brightness level of 0.15? The bare startrails command is a bit cumbersome on its own.

I wonder if there's a way you could leverage the clear skies module to only use images with a certain percentage of the sky being clear, so it doesn't include images where clouds are passing through. (The brightness threshold can kind of work for this, but when there's only a few clouds it doesn't correspond quite as well).

[EricClaeys]

@NiteRide, @brianboru82,
Thanks for the ideas. I was hoping people would chime in, and they are. Multiple heads are better than one.

Having other criteria (besides just the Brightness Threshold) for filtering out images, like clouds or airplanes in the image I think would benefit from number 2 in my original post. All the intelligence would be in the module(s) that create and edit the list of images to use. The startrails command in theory would just take all the images it was given and create a startrails.

The next release of Allsky will have a database and @Alex-developer and I have talked about saving information on every image, like exposure time, mean brightness, DAYTIME or NIGHTTIME, and more. It would be easy for other modules to add data, like if any meteors were detected, number of stars found, cloud coverage, whether plane trails exist, etc., etc.
Part of the startrails creation process could look at the list and apply certain criteria, and write the image names that pass the all the criteria into a file, then pass that file to the startrails command. Users could pick the criteria they want to use by what modules they run. If none are used then just the brightness would be used as is done now.

It would be nice if someone volunteered to design the solution - no programming needed at this point, just someone that can think through problems.

@brianboru82,
Funny you should mention passing the brightness threshold to generateForDay.sh - I have been wanting to allow a way to pass ANY argument for startrails, keograms, and timelapse, I just haven't yet figured out the user interface. My first thought is by adding something like --startrails-args "-b 0.8" --timelapse-args "--fps 25 --width 960".
I think for more in-depth testing number 3 in my original list would be more helpful because it would produce a series of images you could compare. I've asked Alex about giving a script the ability to have a special overlay added to an image, so the startrails produced by number 3 could have the Brightness Threshold added to the image to make comparing them easier.

[jcauthen78]

As one of the people who have had troubles with Startrails after the new exposure algorithm, I've mused about it for a while and tried different methods of sorting what may be cloudy and clear images. I'm a non-typical user, as some know. I capture the daytime images into images/YYYYMMDD/d so I can review them for interesting weather - have caught a few interesting events using this method. I don't store the daytime images in the core date folder because it'd be over 4,500 images most day + nights. and it makes it easy to delete the daytime only images if they're not of use.

I've thought about adding day or night to the image names, but 4,500+ images in a single folder is quite troublesome. So brute-forcing the /d folder in the saveImage.sh for the daytime images works for my needs/wants.

I thought about methods of star detection, sky clarity, or other methods and wondered if the Starcount module could help. Since it has been working successfully on my system for quite a while, I figured it may be a metric that may work for 'is this image something that is good for startrails'.

I spun up a rough metadata module, writing to a JSON file in the previously mentioned ${ALLSKY_IMAGES}/DATA folder.

Will let it run over night and see what I can glean from the data-grab I have it doing - hopefully the weather is semi-clear to at least grab a few frames with fair starcounts... Angle, Starcount, Mean Brightnes, and whatever else might be useful is grabbed, I know it's TOO much data to collect, but for testing, figured "if its there, grab it".

Working on a separate python script (probably module as well) that will dig through the JSON file, do a quick if "day_or_night" = "NIGHT", and "starcount" > 90, add file name to list ${ALLSKY_TMP}/starcount_trails.txt, then try and feed it into generateForDay as a --images and see what happens. Comparing it to the current startrails. I'll probably modify the generateForDay.sh a bit to get the --files working with the txt file, but should be able to farmer it together.

Not sure if this is going to work the way my mind wants it to, or even work at all... BUT - what are y'all's thoughts on a method like this? Anything I should try? or consider in the testing?

[EricClaeys]

@jcauthen78, Jim,
Thanks a ton for doing this!

I think this will most likely work and could easily form the basis for a script that checks multiple criteria - mean brightness, star count, cloud coverage, day/night, etc.

Some things to take into account when creating it:

1. Assume it will use the database when we know how to use it, so try to compartmentalize references to the ${ALLSKY_IMAGES}/DATA folder so we can replace just that code.
2. Assume there will be multiple criteria that could be used, and each will have a setting or something similar that determines if that criteria is actually used. For example, if a user doesn't have the Cloud Detection installed, the code would not include that criteria.
3. If there are NO criteria that can be applied, then the list of files that startrails looks at would be the current list, which is everything.

You will probably want to install the newest startrails command since it accepts a "--images file_name" argument which you'll want to use. All the criteria checks should result in a file that contains image names for startrails to look at. To get the file:

cd ~/allsky/src
mv startrails.cpp startrails.cpp- ORIGINAL
# in GitHub, change from the `master` branch to `v2025.xx.xx` and go to the `src` directory
# click on the `startrails.cpp` file
# Use one of the icons to download the file to the directory you are in.
make
cp startrails ../bin

[craigim]

Echoing some bullets from the other discussion thread (AllskyTeam/allsky-modules#276):

• Agree that having the option to specify darkness level (civil, nautical, astro) makes a lot of sense to me. It could be a simple dropdown with an "other" option for power users to specify a different angle. Only near the equator does a fixed time really make sense. I think that people who are looking for specific times every night are probably power users rolling their own scripts.
• Because the auto-level algorithm shoots for a fixed brightness by adjusting exposure time and gain, and it take a few frames for responses to changes in brightness, the fixed image brightness is not a good measurement of how dark it is outside. You may well have night images that are brighter than day images, or cloudy images at night that are similar to clear. If you calculate brightness after dividing by the gain and the exposure time, that would normalize out those changes and give you a much more physical value of the scene brightness.
• Being able to provide a mask for where to determine brightness is pretty key in my mind.