Using local index file #367

[vieramercado]

It downloads the index file and stores it locally for subsequent use.

Closes #367

[vieramercado]

I'm pretty sure I might be missing an edge case or something else. This will require some careful review. Please let me know and I'll update the code accordingly.

Here is a simple script to test the functionality.

from herbie import Herbie

H = Herbie("2020-10-29", 
           model="gfs", 
           product="0.5-degree",
           verbose=True,
           overwrite=False
           )

print(f"{H.save_dir=}")

H.download(verbose=True)
print(f"{H.idx_source=}")

print(H.inventory())

The first time you run the script, it will download the grib and index files from the internet.

$ python test.py 
✅ Found ┊ model=gfs ┊ product=0.5-degree ┊ 2020-Oct-29 00:00 UTC F00 ┊ GRIB2 @ ncei_analysis ┊ IDX @ ncei_analysis
H.save_dir=WindowsPath('C:/Users/vieramercado/data')
👨🏻‍🏭 Created directory: [C:\Users\vieramercado\data\gfs\20201029]
✅ Success! Downloaded GFS from ncei_analysis       
        src: https://www.ncei.noaa.gov/data/global-forecast-system/access/grid-004-0.5-degree/analysis/202010/20201029/gfs_4_20201029_0000_000.grb2
        dst: C:\Users\vieramercado\data\gfs\20201029\gfs_4_20201029_0000_000.grb2
H.idx_source='ncei_analysis'
Downloading inventory file from self.idx='https://www.ncei.noaa.gov/data/global-forecast-system/access/grid-004-0.5-degree/analysis/202010/20201029/gfs_4_20201029_0000_000.grb2.inv'
     grib_message  start_byte    end_byte              range reference_time valid_time variable                          level forecast_time                              search_this
0               1           0     10320.0            0-10320     2020-10-29 2020-10-29    CLWMR                 1 hybrid level           anl                :CLWMR:1 hybrid level:anl
1               2       10321     44320.0        10321-44320     2020-10-29 2020-10-29     ICMR                 1 hybrid level           anl                 :ICMR:1 hybrid level:anl
2               3       44321     64036.0        44321-64036     2020-10-29 2020-10-29     RWMR                 1 hybrid level           anl                 :RWMR:1 hybrid level:anl
3               4       64037     71931.0        64037-71931     2020-10-29 2020-10-29     SNMR                 1 hybrid level           anl                 :SNMR:1 hybrid level:anl
4               5       71932     76323.0        71932-76323     2020-10-29 2020-10-29     GRLE                 1 hybrid level           anl                 :GRLE:1 hybrid level:anl
..            ...         ...         ...                ...            ...        ...      ...                            ...           ...                                      ...
517           518    90819437  91075890.0  90819437-91075890     2020-10-29 2020-10-29     PRES  PV=-2e-06 (Km^2/kg/s) surface           anl  :PRES:PV=-2e-06 (Km^2/kg/s) surface:anl
518           519    91075891  91249860.0  91075891-91249860     2020-10-29 2020-10-29     VWSH  PV=-2e-06 (Km^2/kg/s) surface           anl  :VWSH:PV=-2e-06 (Km^2/kg/s) surface:anl
519           520    91249861  91451131.0  91249861-91451131     2020-10-29 2020-10-29    PRMSL                 mean sea level           anl                :PRMSL:mean sea level:anl
520           521    91451132  91545878.0  91451132-91545878     2020-10-29 2020-10-29    5WAVH                         500 mb           anl                        :5WAVH:500 mb:anl
521           522    91545879         NaN          91545879-     2020-10-29 2020-10-29    LANDN                        surface           anl                       :LANDN:surface:anl

[522 rows x 10 columns]

The second time you run the script, it will use the grib and index files from your local path. You will notice it is slightly faster as well.

$ python test.py
✅ Found ┊ model=gfs ┊ product=0.5-degree ┊ 2020-Oct-29 00:00 UTC F00 ┊ GRIB2 @ local ┊ IDX @ local
H.save_dir=WindowsPath('C:/Users/vieramercado/data')
🌉 Already have local copy --> C:\Users\vieramercado\data\gfs\20201029\gfs_4_20201029_0000_000.grb2
H.idx_source='local'
     grib_message  start_byte    end_byte              range reference_time valid_time variable                          level forecast_time                              search_this
0               1           0     10320.0            0-10320     2020-10-29 2020-10-29    CLWMR                 1 hybrid level           anl                :CLWMR:1 hybrid level:anl
1               2       10321     44320.0        10321-44320     2020-10-29 2020-10-29     ICMR                 1 hybrid level           anl                 :ICMR:1 hybrid level:anl
2               3       44321     64036.0        44321-64036     2020-10-29 2020-10-29     RWMR                 1 hybrid level           anl                 :RWMR:1 hybrid level:anl
3               4       64037     71931.0        64037-71931     2020-10-29 2020-10-29     SNMR                 1 hybrid level           anl                 :SNMR:1 hybrid level:anl
4               5       71932     76323.0        71932-76323     2020-10-29 2020-10-29     GRLE                 1 hybrid level           anl                 :GRLE:1 hybrid level:anl
..            ...         ...         ...                ...            ...        ...      ...                            ...           ...                                      ...
517           518    90819437  91075890.0  90819437-91075890     2020-10-29 2020-10-29     PRES  PV=-2e-06 (Km^2/kg/s) surface           anl  :PRES:PV=-2e-06 (Km^2/kg/s) surface:anl
518           519    91075891  91249860.0  91075891-91249860     2020-10-29 2020-10-29     VWSH  PV=-2e-06 (Km^2/kg/s) surface           anl  :VWSH:PV=-2e-06 (Km^2/kg/s) surface:anl
519           520    91249861  91451131.0  91249861-91451131     2020-10-29 2020-10-29    PRMSL                 mean sea level           anl                :PRMSL:mean sea level:anl
520           521    91451132  91545878.0  91451132-91545878     2020-10-29 2020-10-29    5WAVH                         500 mb           anl                        :5WAVH:500 mb:anl
521           522    91545879         NaN          91545879-     2020-10-29 2020-10-29    LANDN                        surface           anl                       :LANDN:surface:anl

[522 rows x 10 columns]

[vieramercado]

Also, I would like to add unit test but I'm not sure how you feel about adding grib and index (data) files to the repo for testing purposes. The index file can be small (<1MB), but the grib files can be bigger (>20MB). I'm open to suggestions.

[blaylockbk]

Sorry it's taken so long. Haven't had much free time the last few months.

Also, I would like to add unit test but I'm not sure how you feel about adding grib and index (data) files to the repo for testing purposes. The index file can be small (<1MB), but the grib files can be bigger (>20MB). I'm open to suggestions.

I would really appreciate a test. I would allow adding a test file to the repo, but only for a GRIB file with 2-3 fields for a coarse model to keep it small. We'd have to engineer that file and create an index file with wgrib2.

I have one GRIB file in the repo here
https://github.com/blaylockbk/Herbie/tree/main/sample_data/hrrr/20201214

Alternatively, the test can download a file and index file, then run a test with newly downloaded local data.

[vieramercado]

Hi @blaylockbk any chance we can get this approved and merged soon? Thanks!
Please let me know if there is any improvement you want me to make on this PR.

[blaylockbk]

Thanks for the ping. Just merged.

[blaylockbk]

And thanks for the contribution. I know people have asked for this feature for some time.