Opt-in to scraping for building a data set of all-sky camera captures

[jrussek]

Hi,
I'm currently in the early phase of building a model to automatically identify weather patterns like clouds, rain, fog etc from all sky cameras for astronomical purposes (i.e. automatically assess whether imaging might not be possible.

For this and other purposes, it would be immensely helpful to build a large dataset of images and this software provides a great opportunity.

Would it be feasible to add a feature to:
a) opt-in to have images scraped/downloaded for the purpose of collecting an (open, public) dataset of captures.
b) provide an end-point in the all-sky map to allow a scraper to get a list of cameras that have opted in.

I'd be very happy to contribute, and collaborate with the authors.

Thank you!

[jrussek]

And just now I found the allskyai module - that's exactly what I was thinking of.
Would it be possible to share the data?

[christian-kardach]

Hi!
I think it would be possible, I will have to look into how as right now it’s close to 300k samples and several hundred of GB

Help would be great as tagging none-tagged images is quite time consuming. I have a rudimentary system for auto-tagging but they still need to be verified after that.

cheers,
Chris

[jrussek]

@christian-kardach I'd be very glad to help with that. There is some research into using unsupervised training for modelling weather pattern detection that I would like to try and implement.
I'm also happy to help with hosting the dataset (if we make it available on huggingfaces, it'd be basically free and could enable others to iterate on it).
Having opt in when installing would be amazing to get a ton of data