ECMWF AWS issues? (Oct 30th)

[williamhobbs]

Is anyone having issues getting data from ECMWF via AWS? I think it was working earlier today, but I've also been troubleshooting network issues all day...

For example,

from herbie import Herbie
H = Herbie("2024-03-01", model="ifs", product="oper")

# Get u and v wind component
ds = H.xarray(":10[u|v]:").herbie.with_wind()
ds

(from https://herbie.readthedocs.io/en/stable/gallery/ecmwf_models/ecmwf.html)

returns a JSONDecodeError: Expecting value: line 1 column 1 (char 0) error.

Something more direct, like going after a grib file directly in my web browser, https://ecmwf-forecasts.s3.eu-central-1.amazonaws.com/20250802/00z/ifs/0p25/oper/20250802000000-0h-oper-fc.grib2, returns:

<Error>
<Code>SlowDown</Code>
<Message>Please reduce your request rate.</Message>
<RequestId>DKYT2TF1CBJXHYKH</RequestId>
<HostId>u8034DO9RwteEq3wvRZM3jptOH7s86NKi6IXprw52SNcklQAh6Aku8ss7YQz0CvpC/3BG7cvOGVjqjlixJdSgMFPc9201Vz2</HostId>
</Error>

[rafa-guedes]

I am also having issues trying to download grib files from their aws buckets as well for this latest cycle 2025-10-30 12:00:00+00:00 but I haven't done much troubleshooting yet.

[williamhobbs]

Ok, thanks. That helps me feel less crazy 😀

[williamhobbs]

Google seems to work, so I submitted this PR to add it as an optional source for ECMWF IFS and AIFS: #489.

[rafa-guedes]

It worked here from the GCS buckets too.

[williamhobbs]

I'm having similar issues with IFS via AWS again. I first noticed it a few days ago and it seems to be ongoing but intermittent. Anyone else having problems?

[briwg]

@williamhobbs
Thanks for the suggestions. Unfortunately my work flows are based around having grib files locally, rather than xarray datasets, so I don't think FastHerbie is applicable?

I've raised the regional subsetting with ECMWF, but I've not seen any indication that it is likely in the near future.

[williamhobbs]

Using FH = FastHerbie(…) and then FH.download(search_str) will download grib files, and multiple at once if there are multiple fxx values in the FastHerbie call, so it could/should be faster than getting them one at a time.

[karlwx]

I had to completely give up on AWS and go back to the ECMWF open data server. Are google and azure reliable for you guys? Is the data availability at the same time as other sources, or slightly delayed?

[briwg]

@williamhobbs thanks for that. For the moment I've flipped back to using the ECMWF open data python interface, but I'll investigate FastHerbie too.

PS: ECMWF confirmed to me that regional subsetting isn't available unless you buy a service level agreement.

[williamhobbs]

I had to completely give up on AWS and go back to the ECMWF open data server. Are google and azure reliable for you guys? Is the data availability at the same time as other sources, or slightly delayed?

Google seems to have been working pretty well. I haven't used Azure much. On data availability, I haven't been running much in "real-time", just looking at archived outputs, but based on a quick browse, it looks like the "created" timestamps on Google are on time. E.g.,

https://console.cloud.google.com/storage/browser/ecmwf-open-data/20260504/06z/ifs/0p25/scda?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))

[briwg]

@karlwx I'm using a combination of Google and Azure at the moment. As far as I can tell, the data is available at more or less the same time.

The performance in recent days via the ECMWF open data python interface has been acceptable. That said, in my own scripts I've been able to achieve much better performance by downloading up to 6 steps concurrently. However, I'm not yet confident about using this on a live server for several reasons.

I've also tested with Herbie (that's why I'm here!) and found performance to be very similar, but I've not yet tried out FastHerbie which @williamhobs has suggested.

[blaylockbk]

This is good discussion.

Side note: I just pushed a script to show download times for a model. If you come up with something better, please share it :)

Downloads for IFS model

$ uv run download_timer.py --model ifs --search ":t:"
                                        
Download Summary:
model='ifs', date=datetime.date(2026, 5, 4), search=':t:'

google     █                               5.449 s  (13 fields, 7.75 MB)
ecmwf      ████████████████████▌           14.629 s  (13 fields, 7.75 MB)
aws        █████████████████████▌          15.036 s  (13 fields, 7.75 MB)
azure      ███████████████████████         15.818 s  (13 fields, 7.75 MB)
azure-waef ██████████████████████████████  18.849 s  (13 fields, 7.75 MB)
azure-scda ERROR

Of course, the download times are different every time you run the script and will vary by network. For IFS, the google source seems to be the quickest for the IFS model from what I've seen.

For comparison, here is a benchmark for HRRR and GFS

Downloads for HRRR model

$ uv run download_timer.py --model hrrr --search ":TMP:"

Download Summary:
model='hrrr', date=datetime.date(2026, 5, 4), search=':TMP:'

nomads     █                               2.338 s  (7 fields, 5.27 MB)
aws        █████▌                          2.766 s  (7 fields, 5.27 MB)
azure      ██████████                      3.221 s  (7 fields, 5.27 MB)
google     ██████████████████████▌         4.368 s  (7 fields, 5.27 MB)
pando2     ████████████████████████████    4.903 s  (7 fields, 5.27 MB)
pando      ██████████████████████████████  5.047 s  (7 fields, 5.27 MB)

Downloads for GFS model

$ uv run download_timer.py --model gfs --search ":TMP:"
                                        
Download Summary:
model='gfs', date=datetime.date(2026, 5, 4), search=':TMP:'

azure      █                               22.715 s  (54 fields, 40.00 MB)
aws        █████▌                          23.614 s  (54 fields, 40.00 MB)
nomads     ████████████████▌               25.712 s  (54 fields, 40.00 MB)
google     ██████████████████████████████  28.248 s  (54 fields, 40.00 MB)
ncar_rda   ERROR

[williamhobbs]

This isn't the same thing, but it's somewhat related: https://dynamical.org/status/. I also notice that dynamical seems to be successfully downloading the IFS ensemble from AWS. I wonder if they are seeing the issues we're seeing, and/or if they have a workaround....

[briwg]

Using the ecmwf-opendata Python client this evening, I’m seeing download speeds of about 10–15 Mbps when using ECMWF as the client, about 1 Mbps with AWS, and when I switch to Google and try to download AIFS ENS it errors. My servers are in the UK on the Azure network.

[williamhobbs]

@briwg, what error are you getting from Google?

[briwg]

@williamhobbs I can't remember the exact error, but it was unable to find the file/s.
[Edit: Sorry, that's wrong. It seems the problem is that Google Cloud Storage doesn't support multi-range HTTP requests]

I'm now testing with Azure via the ecmwf-opendata Python client and seeing speeds of about 1 Mbps, despite being on the Azure network.