You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: add 3D tensor scaling section to PR description
Explains factored storage pattern for large tensors: register_section
for compact rank-R factors + register_anndata_namespace for on-demand
tensor reconstruction and O(rank) point queries. Includes compression
ratio (1.5M× for 1M cells) and sparse.COO note.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: PR4_DESCRIPTION.md
+61-1Lines changed: 61 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -191,6 +191,66 @@ True
191
191
|`repr(adata)`| Shows when non-empty |
192
192
| View copy-on-write | Writing to a view triggers copy |
193
193
194
+
### Scaling 3D tensors: factored storage + accessor
195
+
196
+
A dense `(n_obs × n_obs × n_vars)` tensor is infeasible for large datasets (1M cells × 1M cells × 30K genes ≈ 10^16 entries). The practical pattern is to store compact rank-R factors and reconstruct on demand:
>>> adata.comm.query("cell_0", "cell_1", "CD8A", "lr") # O(rank), no tensor
238
+
0.7386
239
+
240
+
>>> t_cells = adata[adata.obs["cell_type"] =="T"]
241
+
>>> t_cells.comm.tensor("lr").shape # factors were subsetted
242
+
(50, 50, 50)
243
+
244
+
>>> adata.write("test.h5ad") # only factors written
245
+
>>> adata2 = ad.read_h5ad("test.h5ad")
246
+
>>> adata2.comm.tensor("lr").shape # reconstructs from factors
247
+
(100, 100, 50)
248
+
```
249
+
250
+
This combines `@register_section` (factor storage with automatic subsetting andIO) with`@register_anndata_namespace` (tensor APIand point queries). For 1M cells with rank 20, the factors are ~160MBwhile the dense tensor would be ~240TB — a 1,500,000× compression.
251
+
252
+
For moderately-sized datasets, `sparse.COO`from the PyData sparse package also works directly in registered sections (subsetting handles N-D sparse arrays).
67 tests covering all alignment patterns, custom validation, custom IO (JSON, xarray), 3D tensor subsetting, copy-on-write, and end-to-end workflows for TreeData-like, SpatialData-like, CellChat-like, andSCENIC-like scenarios.
262
+
73 tests covering all alignment patterns, custom validation, custom IO (JSON, xarray), 3D tensor subsetting, factored tensor with accessor, copy-on-write, and end-to-end workflows for TreeData-like, SpatialData-like, CellChat-like, SCENIC-like, and factored communication scenarios.
0 commit comments