Investigate Proof of Concept Partitions for Behaviour with Dask

[stephenworsley]

📰 Custom Issue

Subtask of #427.

Before designing the rest of the Partition class, it's worth determining what obstacles there may be to the core idea of loading and applying regridders lazily.

The proof of concept should:

• Reference a collection of saved regridders and indexing information (probably with some overlaps).
• Use those regridders to map a dask array to another dask array.
• Only load the regridders when the result dask array is being realised/computed.

The investigation should check:

• How flexible is a partition with repsect to different source chunking? Is it necessary to have a specific partition for a specific source chunking?
• How should the source chunking affect the target chunking?
• How can vertical chunks be handled efficiently? Is it possible to not have to reload the same regridder multiple times?
• Are there any blockers to loading during a chunkwise opperation?

[trexfeathers]

So excited to see this in action (hopefully I can be some use in helping)!

[stephenworsley]

See this gist for an initial proof of concept for this idea: https://gist.github.com/stephenworsley/2235d21cdee8a84a83e889362a83b994