Ultra-large DelayedMatrix backend and portable single-file Seurat objects (#9798)#10405
Open
BenjaminDEMAILLE wants to merge 17 commits into
Open
Ultra-large DelayedMatrix backend and portable single-file Seurat objects (#9798)#10405BenjaminDEMAILLE wants to merge 17 commits into
BenjaminDEMAILLE wants to merge 17 commits into
Conversation
Declare DelayedMatrixStats, HDF5Array, rhdf5 and spam in Suggests and register R/spam.R, R/delayedarray.R, R/save-portable.R and R/save-h5.R in Collate for the ultra-large matrix and portable-object work (satijalab#9798). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CheckMatrixSize reports a matrix's (non-zero) element count against R's 2^31-1 indexing limit and points users to on-disk backends when the limit is approached. RowSumSparse/RowMeanSparse/RowVarSparse now compute natively on DelayedMatrix via DelayedMatrixStats instead of force-converting to dgCMatrix, which would fail past 2^31 entries (satijalab#9798). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror the BPCells IterableMatrix backend for DelayedArray/HDF5Array-backed layers: as.DelayedMatrix() coercion, as.sparse/.CalcN methods, and block-wise LogNormalize and VST that never materialize the full matrix. DelayedMatrix carries dimnames and uses 64-bit indexing, so it works as a Seurat layer past the dgCMatrix 2^31 limit. NormalizeData -> FindVariableFeatures now run natively on DelayedMatrix with results identical to dgCMatrix (satijalab#9798). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
as.sparse.spam converts a 64-bit-indexed spam matrix into a dgCMatrix for use in Seurat. spam objects cannot be Seurat layers directly because they do not carry dimnames; for out-of-memory ultra-large layers use the DelayedMatrix or BPCells backends. This method covers the conversion path (satijalab#9798). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AsInMemory converts BPCells/DelayedMatrix layers of a Seurat object into in-memory dgCMatrix, yielding a self-contained object that plain saveRDS() writes to a single portable file -- the AnnData .h5ad experience for data that fits in RAM. Errors with guidance toward SaveSeurat()/SaveSeuratH5() when a layer exceeds the 2^31 limit (satijalab#9798). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add StitchMatrix.DelayedMatrix so multi-layer (split-by-batch) DelayedMatrix objects can be joined and scaled, using DelayedArray::arbind/acbind to avoid base/S4Vectors cbind mis-dispatch. Extend CCAIntegration's on-disk handling to convert DelayedMatrix (like BPCells) to dgCMatrix for CCA. With these, JoinLayers, ScaleData, RunPCA and CCA/RPCA/Harmony IntegrateLayers all run on DelayedMatrix-backed objects (satijalab#9798). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SaveSeurat() writes a Seurat object plus all of its out-of-memory layers (BPCells, DelayedMatrix/HDF5) into one gzip tar archive, copying each on-disk store into the bundle and recording it in a manifest. LoadSeurat() extracts the archive and reconnects the layers to the extracted stores, so a bundle can be moved or shared like an AnnData .h5ad with no absolute-path breakage. Fully in-memory objects are simply serialized inside the archive (satijalab#9798). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SaveSeuratH5() writes a Seurat object to one HDF5 file -- the closest analogue to an AnnData .h5ad: each assay layer is stored as a 10x-style sparse HDF5 dataset (via writeTENxMatrix) and reopened lazily as a DelayedMatrix by LoadSeuratH5(), so the file is portable, cross-tool readable and not bound by the dgCMatrix 2^31 limit. meta.data (numeric/character/factor/logical) and dimensional reductions are stored alongside; split layers are joined per assay. v1 omits graphs/images/commands/misc (use SaveSeurat for a full bundle) (satijalab#9798). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add man pages for AsInMemory, SaveSeurat, LoadSeurat, SaveSeuratH5 and LoadSeuratH5 (satijalab#9798). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cover CheckMatrixSize, the DelayedMatrix backend (coercion, .CalcN, LogNormalize/VST equivalence to dgCMatrix, PCA), multi-layer JoinLayers and RPCA integration, spam coercion, AsInMemory + single-file saveRDS, the SaveSeurat/LoadSeurat bundle across a file move, and the SaveSeuratH5/ LoadSeuratH5 HDF5 container. All guarded with skip_if_not_installed (satijalab#9798). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a section covering SaveSeurat/LoadSeurat, SaveSeuratH5/LoadSeuratH5 and AsInMemory for writing portable, self-contained objects with on-disk layers (satijalab#9798). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
as.DelayedMatrix(x, ondisk = TRUE) realizes x to a 10x-style HDF5 TENxMatrix store whose non-zero-count pointer (indptr) is 64-bit, so the resulting DelayedMatrix can represent more than 2^31 non-zero entries and is processed block-wise. The default in-memory wrapping is bounded by its seed (a dgCMatrix seed stays under 2^31). Documents the 64-bit path and tests a >2^31-element on-disk matrix (satijalab#9798). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
as.DelayedMatrix now wraps sparse input in a SparseArray SVT_SparseMatrix seed (when available) rather than a dgCMatrix. Because SVT stores non-zeros in a per-column tree instead of one flat index vector, the layer can hold more than 2^31 non-zeros entirely in RAM while carrying dimnames, so a plain saveRDS writes the whole object to one self-contained file -- an in-memory AnnData equivalent. ondisk = TRUE remains the out-of-memory HDF5 path. Adds SparseArray to Suggests and a round-trip test (satijalab#9798). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a .AsSparseIfFits helper that materializes an on-disk layer (DelayedMatrix, BPCells) to dgCMatrix for operations that are not block-native, erroring with guidance when the layer exceeds the 2^31 limit. Apply it to CLR/RC NormalizeData and mean.var.plot/dispersion feature selection, and compute PseudobulkExpression (AverageExpression/AggregateExpression) on DelayedMatrix via a block-wise multiply instead of rejecting it. All produce results identical to dgCMatrix (satijalab#9798). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop Seurat's StitchMatrix.DelayedMatrix in favor of the canonical method now provided by SeuratObject (>= 5.4.0.9000), alongside StitchMatrix.IterableMatrix. Bumps the SeuratObject dependency accordingly. Multi-layer DelayedMatrix JoinLayers/ScaleData/integration continue to work via the SeuratObject method. Companion to satijalab/seurat-object#295 (satijalab#9798). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
seurat-object main advanced to 5.4.0.9001 (satijalab#294) after this branch was cut, so the StitchMatrix.DelayedMatrix dev snapshot is now 5.4.0.9002. Pin the dependency to that version; a user on 5.4.0.9001 lacks the method and would otherwise satisfy the old >= 5.4.0.9000 constraint while hitting a runtime error in JoinLayers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Closes the actionable part of #9798: enabling Seurat to work with matrices larger than the
dgCMatrix2³¹ limit, and making such objects portable as a single file — "a Seurat object in one file, the way you don't fuss over an AnnData.h5ad."What's in this PR
1.
DelayedMatrix(DelayedArray/HDF5Array) layer backend mirroring the BPCellsIterableMatrixbackend (as.DelayedMatrix,as.sparse,.CalcN,LogNormalize,VST).DelayedMatrixcarries dimnames and 64-bit indexing, so it works as a Seurat layer past 2³¹.as.DelayedMatrix(x)uses aSparseArray::SVT_SparseMatrixseed (per-column tree, no flat 2³¹ index vector), so a layer can hold >2³¹ non-zeros in memory and a plainsaveRDSwrites one self-contained file — an in-memory AnnData equivalent.as.DelayedMatrix(x, ondisk = TRUE)realizes to a 10x-style HDF5TENxMatrix(64-bitindptr).2. "Total" pipeline coverage on
DelayedMatrix-backed objects, verified numerically equal todgCMatrix:NormalizeData(LogNormalize native; CLR/RC via materialize-when-fits),FindVariableFeatures(vst native; mvp/dispersion materialize-when-fits),ScaleData,RunPCA, neighbors/clusters/UMAP,SCTransform,FindMarkers/FindAllMarkers/FoldChange,AverageExpression/AggregateExpression/PseudobulkExpression(block-wise), and multi-layersplit→JoinLayers→ScaleData→RunPCA→IntegrateLayers(CCA, RPCA, Harmony). Non-block-native steps materialize-when-fits with a warning (never silent subsampling), matching BPCells.3. Ultra-large interoperability + portability.
CheckMatrixSize()(actionable 2³¹ guidance),as.sparse.spam(spam is coercion-only — no dimnames),AsInMemory()(on-disk → in-memory, single-rds), and single-file objects:SaveSeurat/LoadSeurat(bundle, survives a file move) andSaveSeuratH5/LoadSeuratH5(one.h5, a.h5adanalogue).Tests
New
tests/testthat/test_large_matrix.R(guarded withskip_if_not_installed) — backend equivalence to dgCMatrix, in-RAM SVT + single-rds round-trip, 64-bit on-disk, multi-layer integration, CLR/RC + mvp + pseudobulk, bundle/HDF5 round-trips across a file move. Locally: new tests pass;test_utilities(48) andtest_preprocessing(97) unchanged. Addsspam,HDF5Array,DelayedMatrixStats,rhdf5,SparseArraytoSuggests.Kept intentionally separate from #10017.
🤖 Generated with Claude Code