Improve performance in `cfdm.read` by caching any array values retrieved from disk

[davidhassell]

First and last array values are cached in NetCDFRead.read._cache_data_elements, and it currently could do this multiple times per variable.

It would be straight forward to cache these values to save going back to the dataset more often than is required.

PR to follow.

[sadielbartholomew]

I'm a bit confused by your description @davidhassell, so just to check - from re-reading a bit my interpretation is that you mean we already cache first and last values for the array of data variables, but we can do this more often that we currently do? Would appreciate a clarification - on the following PR or here. Thanks.

[davidhassell]

Hi Sadie - sorry - the word "cache " is doing some heavy lifting, here! I'll try to re-word (although you're already correct!)

Selected array values (typically the first and last in the array) are read from disk and cached inside the Data objects of the returned constructs. A given netCDF variable might end up providing the data for multiple constructs, in which case the selected values end up getting read from disk multiple times. Therefore we need a new cache of these values in NetCDFRead that means we're not going to disk more often than we need to. I imagine that this new cache will look something like:

self.read_vars['_cached_data_elements'] = {
    <netCDF variable name>: <selected array values>,  
    <netCDF variable name>: <selected array values>,  
    ...
}

Then, each time we want to get the selected array values, we get them from the NetCDFRead cache, only going to disk if they're not there.

[sadielbartholomew]

Ah I see - thanks for clarifying, I guess I'll be reviewing the PR eventually so it's out of more than just curiosity 🙂

[davidhassell]

PR now up - empirical evidence on my laptop shows that this change speeds up the units tests by ~5%

[davidhassell]

PR now up - empirical evidence on my laptop shows that this change speeds up the units tests by ~5%

Not been able to repeat this, and further investigation shows that the new cache is not accessed in the unit tests, anyway.

That said, there is at least one case when the cache does get accessed - when there is a scalar coordinate variable shared between two data variables, so we might as well leave it all in for that.