Cache warming #526
Description
Is your feature request related to a problem? Please describe.
Cold cache file reads on lustre are really slow. We should be able to speed things up by warming the cache, ie. touching relevant files ahead of time. Think of this as the filesystem version of prefetching data.
Describe the feature you'd like
The intake-df-catalog (top level catalog) contains about 200 files. It seems like an easy win to me start a subprocess that touches all of those files (jupyter notebooks have an async event loop we can submit tasks to so this happens as a background process with minimal effort if we find it. The display module there also contains some code to work out if we are in a Notebook anyway, so we can hook into that.