Potential bug? Herbie produced wrong source URL for recent RRFS data files

[jllmoe]

Hi There;

I was trying to download recent RRFS files and found that Herbie produced wrong source URL for recent RRFS data files, consequently it cannot find the files, for example, the correct URL is supposed to be

https://noaa-rrfs-pds.s3.amazonaws.com/rrfs_a/rrfs.20260318/12/rrfs.t12z.natlev.3km.f000.na.grib2

but Herbie produced the following wrong URL

https://noaa-rrfs-pds.s3.amazonaws.com/rrfs_a/rrfs_a.20260318/12/control/rrfs.t12z.natlev.f000.grib2

Here is the python code, in which member and domain have to be present

from herbie import Herbie

H = Herbie(
    "2026-03-18 12:00",
    model="rrfs",
    member="control",
    product="natlev",
    domain=None,
    fxx=0
)

print(H.SOURCES)

Is this a bug or I am missing something? Any work around?

Thx for your help.

John

[blaylockbk]

Thanks for creating an issue. @jacob1299, would you be willing to take a look at this since you recently updated the RRFS template?

[jllmoe]

Anyone Herbie developers can help on this one? Much appreciated!

[jacob1299]

Yeah, I can take a look at this tonight and see what's going on. I missed the first notification, so my apologies for that.

The URL format you're seeing (rrfs_a/rrfs_a.YYYYMMDD/.../control/...) is from an older template that doesn't match the current S3 bucket structure. That was already fixed in PR #511, so updating should get you most of the way there:

pip install git+https://github.com/blaylockbk/Herbie.git

There's also a second issue — natlev files only exist with domain "na" on S3, but Herbie was defaulting to "conus" and generating a bad URL. PR #520 fixes that so it just works:

H = Herbie("2026-03-18 12:00", model="rrfs", product="natlev", fxx=0)

Also, member is only needed for ensemble runs (e.g. member=1). For the deterministic run you can leave it out.

@jllmoe

[jllmoe]

Thank you so much @jacob1299 for your quick fix, now it is working with domain='na', (without this, Herbie still goes to conus domain, and consequently results "not found").

Now I found that the variable-subset download is not working properly, it hangs after finish the 18th hybrid level no matter how many variables I choose to subset (even just 1 variable), but Herbie has no such issue in downloading the full file. I am attaching my code, any easy fix?

Thx again for your help

Cheers
John

getRRFS3.py

[blaylockbk]

Reopening for this comment

Now I found that the variable-subset download is not working properly, it hangs after finish the 18th hybrid level no matter how many variables I choose to subset (even just 1 variable), but Herbie has no such issue in downloading the full file. I am attaching my code, any easy fix?

[jllmoe]

Reopening for this comment

Now I found that the variable-subset download is not working properly, it hangs after finish the 18th hybrid level no matter how many variables I choose to subset (even just 1 variable), but Herbie has no such issue in downloading the full file. I am attaching my code, any easy fix?

I did more tests and found, the shared getRRFS3.py worked on my desktop, but still not working properly on my laptop, on which I left getRFSS3.py run for about 1 hour 20 minutes then canceled it, it produced a 145GB temporary file (subset?????.grib2) while a single 1 hour full file is only 9GB and my subset is supposed to be about 4GB. so it may not be Herbie's issue. I will continue to figure this out and keep you posed.

[jacob1299]

Hi @jllmoe, thank you for digging in more. I am not able to reproduce your issue, though I have a few questions:

1. Are you using the most recent version of Herbie? Pull the latest main branch and try with that.
2. Are you using a VPN or any sort of corporate device to run the script?
3. Are you using Windows antivirus or anything related?
4. Can you run this script on both your laptop and desktop and paste the output here?

import requests, os, platform, sys

url = (
    "https://noaa-rrfs-pds.s3.amazonaws.com/rrfs_a/"
    "rrfs.20260401/00/rrfs.t00z.natlev.3km.f000.na.grib2"
)

print("=== Test 1: Range bytes=0-99 ===")
r = requests.get(url, headers={"Range": "bytes=0-99"}, timeout=15)
print(f"  Status: {r.status_code} (should be 206)")
print(f"  Received: {len(r.content)} bytes (should be 100)")
print(f"  Content-Range: {r.headers.get('Content-Range', 'NOT PRESENT')}")

print("\n=== Test 2: Range past 2GB boundary ===")
r = requests.get(url, headers={"Range": "bytes=2253850579-2253850678"}, timeout=15)
print(f"  Status: {r.status_code} (should be 206)")
print(f"  Received: {len(r.content)} bytes (should be 100)")

print("\n=== Environment ===")
print(f"  Python: {sys.version}")
print(f"  requests: {requests.__version__}")
print(f"  Platform: {platform.platform()}")
print(f"  HTTPS_PROXY: {os.environ.get('HTTPS_PROXY', 'not set')}")

[jacob1299]

Opened #521 to address this. It checks that the server actually returns a partial response (HTTP 206) before reading the body, so if something in the network is interfering with Range requests, the download fails immediately instead of silently ballooning to hundreds of GB on disk.

[jllmoe]

Thx Jacob!

1. I followed your instruction 2 days ago and installed the git+Herbie, so I suppose I am using the latest Herbie on bot computers.

2. Both computer are running through the same VPN.

3. both computers are using windows Defender.

4. The following are the returns to your script, the only difference I can see is the version of the requests package: 2.23.4 on desktop and 2.23.5 on laptop.on Desktop

C:\RRFSSubset>python test.py
=== Test 1: Range bytes=0-99 ===
Status: 206 (should be 206)
Received: 100 bytes (should be 100)
Content-Range: bytes 0-99/9429696086

=== Test 2: Range past 2GB boundary ===
Status: 206 (should be 206)
Received: 100 bytes (should be 100)

=== Environment ===
Python: 3.13.12 (tags/v3.13.12:1cbe481, Feb 3 2026, 18:22:25) [MSC v.1944 64 bit (AMD64)]
requests: 2.32.4
Platform: Windows-11-10.0.26100-SP0
HTTPS_PROXY: not set

on Laptop

C:\RRFSSubset>python test.py
=== Test 1: Range bytes=0-99 ===
Status: 206 (should be 206)
Received: 100 bytes (should be 100)
Content-Range: bytes 0-99/9429696086

=== Test 2: Range past 2GB boundary ===
Status: 206 (should be 206)
Received: 100 bytes (should be 100)

=== Environment ===
Python: 3.13.12 (tags/v3.13.12:1cbe481, Feb 3 2026, 18:22:25) [MSC v.1944 64 bit (AMD64)]
requests: 2.32.5
Platform: Windows-11-10.0.26100-SP0
HTTPS_PROXY: not set

[jacob1299]

Can you also run this on both machines? python -c "import herbie; print(herbie.__version__)"

[jllmoe]

Can you also run this on both machines? python -c "import herbie; print(herbie.__version__)"

they are identical

on Desktop

C:\RRFSSubset>python -c "import herbie; print(herbie.version)"
2026.3.1.dev3+g773b410af

on Laptop

C:\RRFSSubset>python -c "import herbie; print(herbie.version)"
2026.3.1.dev3+g773b410af