Reader failing, and opendrift is trying to load entire forcing field?

[brucelaughlin]

02:01:27 ERROR opendrift.models.basemodel.environment:666: Unable to allocate 26.8 GiB for an array with shape (365, 50, 469, 421) and data type float64

And then later

02:01:27 DEBUG opendrift.models.basemodel.environment:498: Discarding reader (failed more than 1 times): /home/blaughli/symbolic_links/mercator_reanalysis12/*.nc

Are these errors related?

Why would Opendrift be trying to allocate "26.8 GiB for an array with shape (365, 50, 469, 421) and data type float64" ?? Those are the dimensions of the entire domain, and my releases are from a small region within the domain. I thought Opendrift does some clever memory management and only loads necessary data, local to the drifters, rather than the entire domain.

And, since I apparently don't have this much memory available, is this why the reader failed and is being discarded?

Thanks,
Bruce

[knutfrode]

Yes, OpenDrift should use just a minimum of memory.
How to you create this reader?
Depending on circumstances, you might need to add quotes ( " " ) when using a path with wildcards.

[brucelaughlin]

I'm using what I believe to be Mercator data, perhaps from Glorys 12, which is perhaps output from Nemo...

I tried importing reader_netCDF_CF_generic, hoping this would work (I didn't know which reader to use!). Is there a better choice? I've only used ROMS forcing in the past; now I wonder how one chooses a reader for forcing from other models. I'm not sure how to know if forcing files are CF-compliant...

Anyway, here's my code (note that I have symbolic links to all of my forcing files in "his_dir"):

from opendrift.readers.reader_netCDF_CF_generic import Reader
...
his_file_wildcard = '*.nc'
his_files = his_dir + '/' + his_file_wildcard
reader_current_job = Reader(his_files)
o.add_reader(reader_current_job)

[brucelaughlin]

Some more log output:

02:00:04 DEBUG opendrift.readers.basereader.variables:752: Fetching variables from /home/blaughli/symbolic_links_ROMS/mercator_reanalysis12/*.nc covering 11844 elements

02:01:27 ERROR opendrift.models.basemodel.environment:666: Unable to allocate 26.8 GiB for an array with shape (365, 50, 469, 421) and data type float64

It's curious to me that (365, 50, 469, 421) is the the correct shape of a 4D variable in my forcing data... How is that determined correctly, before things fail more generally?

[brucelaughlin]

I tried using double-quotes instead of single-quotes, with the same result. Note also that my original code (as shared) has been working with ROMS forcing and the ROMS native reader.

[knutfrode]

Type of quotes does not matter, and reader_netCDF_CF_generic is normally the correct choice (unless you e.g. have native ROMS data).

The problem here seems to be the way Xarray open_mfdataset tries to combine the files.
If you update OpenDrift to the latest, there are some recent fixes that tries to make correct chunking in the time dimension.

If you get same problem with latest OpenDrift, you can try to make an Xarray dataset first, and then an OpenDrift reader from this:

>>> ds = xr.open_mfdataset(<your files>, <mfdataset_options to optimize>)
>>> r = Reader(ds)

You may then play around with mfdataset options until you find something that is fast.

A third options is to obtain GLORYS data directly from CMEMS server:
r = Reader('cmems_mod_glo_phy_myint_0.083deg_P1D-m')

[brucelaughlin]

Thank you; after updating, I am not seeing the same memory errors. I'll update my post once a test run has finished