Issues and suggested enhancements for MRMS Dataset

[kevinyang-cky]

Issue

The Dataset won't return the target/label when the data loading mode is set to "remote."

__getitem__ won't return target/label when self.file_dict.get(field_type) returns None:

miles-credit/credit/datasets/MRMS.py

Lines 227 to 228 in f11aed3

	if self.file_dict.get(field_type) and field_type in self.var_dict:
	self._extract_field(field_type, t_target, target_data)

which is always the case (see _register_field()):

miles-credit/credit/datasets/MRMS.py

Lines 268 to 270 in f11aed3

	else:
	# Remote mode: S3 path is constructed at runtime from the timestamp
	self.file_dict[field_type] = None

One can remove the safeguard if self.file_dict.get(field_type) to temporarily bypass the issue, but the question remains: where is the appropriate place for the safeguard.

Enhancement

1. Local files may be stored in subdirectories:

Current code assumes all files are stored in the same directory (see _register_field()):

miles-credit/credit/datasets/MRMS.py

Lines 264 to 265 in f11aed3

	if self.mode == "local":
	files = sorted(glob(d.get("path", "")))

It would be nice to support a recursive search through all subdirectories.

2. Handle irregular timestamps in files:

The current code can only retrieve data for the exact requested timestamp, but since some MRMS products have filenames like MRMS_MergedReflectivityQC_00.50_20201014-000227.grib2.gz, a scheme to search for the nearest timestamp is needed. The current GOES Dataset can handle this and should be straightforward to implement here.

3. Initialize the s3fs on the first call to __getitem__

Next steps

Would like to hear feedback from @charlie-becker and decide how to move forward. Happy to take the lead on the modification as well.

[djgagne]

Yes, glob.glob with ** and recursive=True searches subdirectories, so that should be an easy fix:

  import glob

  # Find all files matching a pattern in any subdirectory
  matches = glob.glob("**/my_pattern*.nc", recursive=True)

Or pathlib.Path.rglob, which is often cleaner:

  from pathlib import Path

  # Find all .nc files with "ERA5" in the name, anywhere under the current dir
  matches = list(Path(".").rglob("*ERA5*.nc"))

  # Or from a specific root
  matches = list(Path("/data/credit").rglob("static_variables*.nc"))

rglob(pattern) is equivalent to glob("**/pattern", recursive=True). The pathlib version returns Path objects, which are usually more convenient to work with downstream.