Avoid densifying matrix in ProjectCellEmbeddings#10337
Open
assaron wants to merge 2 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes an issue with large transient memory usage in
FindTransferAnchorsarising from generation of a dense matrix insideProjectCellEmbeddings. I had this problem when working with Xenium 5K data and there was a similar report here: #7929 This PR removes explicit dense matrix used for the projection by moving the adjustments down the chain. The behavior doesn't change at all and the results match exactly (up to floating point arithmetic errors). It might also be a little faster.Unfortunately, I was not able to come up with a good illustrative example on a standard dataset—apparently some other memory allocations dominate in such cases—so the example below is on synthetic data. Still, in my workflow it drastically improves memory usage (from 30GB+ to ~16GB, but I didn't check it carefully).
Example:
For the current CRAN version the peak memory usage is 2.7GB, for the PR version it's 1.1GB.
Disclaimer: the code was generated with a help of Claude Code.