Search and download files in AWS bucket with s3fs

[blaylockbk]

You don't have to use Herbie to explore the AWS public buckets. You can use s3fs to list files and download them to your computer.

import s3fs
# Connect to AWS public buckets
fs = s3fs.S3FileSystem(anon=True)

# List directories in hrrr big data program public data set
fs.ls('noaa-hrrr-bdp-pds')

# List files in a bucket path
fs.ls('noaa-hrrr-bdp-pds/hrrr.20210917/conus')

# Download file
fs.get('noaa-hrrr-bdp-pds/hrrr.20210917/conus/hrrr.t05z.wrfnatf13.grib2.idx', './myLocal.grib2.idx')