Skip to content

Conversation

avolu
Copy link
Contributor

@avolu avolu commented May 15, 2025

See https://3.basecamp.com/5595649/buckets/32341364/todos/8649700954 for discussion

I added a new xrutils function for memory and cpu-efficient multiplication of large matrices. xrutils.chunked_eff_xr_matmult performs the mutliplication using numpy only, chunks up the data and streams chunks to disk, before it reassembles everything and puts it back into an xarray. The output should be identical to xarrayA @ xarrayB, but less hard on memory and cpu overhead.

This function is then used in the updated forward_model.apply_inv_sensitivity function whenever timeseries data with more than 1000 samples is handed in, which can however be force-skipped by setting the optional "chunk" flag to false (true by default).

I did some testing on the output of both the conventional and new chunked conversion, and (ignoring very small floating point errors) both yield the same result.

Maybe its a quick review and you can merge it, otherwise this has time until later

@avolu avolu requested a review from emiddell May 15, 2025 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant