AWS s3 ERA5 deprecation

[mickwelli]

It seems ECMWF ERA5 Reanalysis has been deprecated on AWS S3 , which is a shame. Wondering if anyone here has insights on alternative public bucket options? Thanks!

[aluhamaa]

It seems ECMWF ERA5 Reanalysis has been deprecated on AWS S3 , which is a shame. Wondering if anyone here has insights on alternative public bucket options? Thanks!

@mickwelli Hi, just noticed as well. As a developer of the original version (but moved to another company two years ago), I could take over the maintenance, but it requires some planning and will not happen overnight.

[mickwelli]

It seems ECMWF ERA5 Reanalysis has been deprecated on AWS S3 , which is a shame. Wondering if anyone here has insights on alternative public bucket options? Thanks!

@mickwelli Hi, just noticed as well. As a developer of the original version (but moved to another company two years ago), I could take over the maintenance, but it requires some planning and will not happen overnight.

Hi @aluhamaa, thanks for your response. It's possible that myself or colleagues could assist. We have some dependent functions that we'd like to maintain e.g. load_era5(). Please get in contact if we may be of assistance.

[aluhamaa]

Thanks, @mickwelli I'll try to contact AWS and let you know here. If I continue, it would be a fully open solution with all contributions welcome.

[Gonzalo1985]

Hello,
Sorry I think I opened other issue about the same problem. We have a R package for management of ERA5 data from AWS. It would be a shame to lose this. If there is a possibility of helping on this issue please let me know.
Thank you!

[aluhamaa]

@Gonzalo1985 @mickwelli I've got a response from AWS, they have several interested persons already and they will continue providing the dataset. No details about when and if there will be any changes in formats, etc.

[Gonzalo1985]

Thank you very much @aluhamaa. I hope we can get it soon.

[mickwelli]

Hi again folks. We've managed to re-route our load_era5() function to google, but have left the AWS option open when access is reinstated. Sharing in case it's useful for others digitalearthafrica/deafrica-sandbox-notebooks#491

[Gonzalo1985]

Ok, thanks @mickwelli! We know something new about the data availability on AWS?

[mickwelli]

Ok, thanks @mickwelli! We know something new about the data availability on AWS?

Only the above, which indicates they are working on it.

[rmnvncnt]

Any news on this topic? I'm accessing ERA5 data through the Climate Data Store API, but I would prefer access it through blob storage to speed up the process.

[aluhamaa]

@rmnvncnt There seems to be a new version by NCAR, but that is NetCDF https://registry.opendata.aws/nsf-ncar-era5/ and seems not to be very up-to date, currently data till end of 2024
Would be interesting to know if it can be used with kerchunk or virtualizarr etc.

Limited amount of ERA5 is also available via openmeteo API. I do not understand their format immediately, but they seem to provide open software to access it, so should be doable. https://registry.opendata.aws/open-meteo/

Also there is a version in google cloud, which is proper zarr, but I think they do not update it regularly as well.

[rmnvncnt]

Thanks @aluhamaa for the hints!

[Gonzalo1985]

Thanks @aluhamaa! I am reaching to https://registry.opendata.aws/nsf-ncar-era5/, but it is difficult to understand which variable is each file. Thanks!

[Gonzalo1985]

I think that the precipitation from the analysis is not in here: https://registry.opendata.aws/nsf-ncar-era5/