Skip to content

Commit 85f0b88

Browse files
alxndrkalininsrivarraedyoshikunclaudeCopilot
authored
feat(dynacell): HF demo + eval metrics/perf refactor + pix2pix3d leaves + dataset notebook (#449)
* VisCy Modular: Transforms and Monorepo Skeleton (#356) * refactor: restructure viscy into uv workspace monorepo with viscy-transforms subpackage BREAKING CHANGE: - Import path changed: from viscy.transforms import X → from viscy_transforms import X - Removed legacy modules: viscy.utils, viscy.cli, viscy.unet, viscy.evaluation - Removed applications/, examples/, and docs/ directories Signed-off-by: Sricharan Reddy Varra <sricharan.varra@biohub.org> * docs: updated readme with citations / examples from main Signed-off-by: Sricharan Reddy Varra <sricharan.varra@biohub.org> * docs: added symlinking uv cache on hpc systems Signed-off-by: Sricharan Reddy Varra <sricharan.varra@biohub.org> * add the jupyter and ipykernel to a optional visual group * re organize the transforms * add jupyternotebook back * build: updated some dep groups Signed-off-by: Sricharan Reddy Varra <sricharan.varra@biohub.org> * build: added matplotlib back in Signed-off-by: Sricharan Reddy Varra <sricharan.varra@biohub.org> * build: add ruff and prek to the dev dep group * docs: correct the docstring for the transform that doesn't exist lol Signed-off-by: Sricharan Reddy Varra <sricharan.varra@biohub.org> --------- Signed-off-by: Sricharan Reddy Varra <sricharan.varra@biohub.org> Co-authored-by: Sricharan Reddy Varra <sricharan.varra@biohub.org> Co-authored-by: Eduardo Hirata-Miyasaki <edhiratam@gmail.com> Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com> * docs: map existing codebase * Modularizing the model's folder (#365) * add planning roadmap * docs: start milestone v1.1 Models * docs: define milestone v1.1 requirements * docs: create milestone v1.1 roadmap (5 phases) * docs(package-scaffold-shared-components): research phase domain * docs(06): create phase plan - package scaffold and shared components * feat(06-01): create viscy-models package scaffold - Add pyproject.toml with hatchling build, torch/timm/monai/numpy deps - Create src layout with _components, unet, contrastive, vae subpackages - Add PEP 561 py.typed marker - Add test scaffolding with device fixture Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(06-01): register viscy-models in workspace - Add viscy-models to root dependencies and uv sources - Update lockfile with timm and viscy-models dependencies - Verified: uv sync, import, and pytest collection all succeed Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(06-01): complete package scaffold plan - Add 06-01-SUMMARY.md with execution results - Update STATE.md with position, metrics, and decisions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(06-02): extract shared components into _components/ module - stems.py: UNeXt2Stem, StemDepthtoChannels from v0.3.3 unext2.py - heads.py: PixelToVoxelHead, UnsqueezeHead, PixelToVoxelShuffleHead - blocks.py: icnr_init, _get_convnext_stage, UNeXt2UpStage, UNeXt2Decoder - __init__.py: re-exports all 8 public components - Zero imports from unet/, vae/, or contrastive/ - All attribute names preserved for state dict compatibility * feat(06-03): migrate ConvBlock2D and ConvBlock3D to unet/_layers/ - Copy ConvBlock2D from v0.3.3 source to snake_case file - Copy ConvBlock3D from v0.3.3 source to snake_case file - Preserve register_modules/add_module pattern for state dict key compatibility - Update _layers/__init__.py with public re-exports - Fix docstring formatting for ruff D-series compliance * test(06-03): add tests for ConvBlock2D and ConvBlock3D - 6 tests for ConvBlock2D: forward pass, state dict keys, residual, filter steps, instance norm - 4 tests for ConvBlock3D: forward pass, state dict keys, dropout registration, layer order - All 10 tests verify shape, naming patterns, and module registration * test(06-02): add forward-pass tests for all _components - test_stems.py: UNeXt2Stem shape, StemDepthtoChannels shape + mismatch error - test_heads.py: PixelToVoxelHead, UnsqueezeHead, PixelToVoxelShuffleHead shapes - test_blocks.py: icnr_init, _get_convnext_stage, UNeXt2UpStage, UNeXt2Decoder - 10 tests total, all passing on CPU * docs(06-03): complete UNet ConvBlock layers plan - SUMMARY.md with migration details and self-check - STATE.md updated: phase 6 plan 3/3, decisions recorded * docs(06-02): complete shared components extraction plan - SUMMARY.md documents 8 extracted components with 10 tests - STATE.md updated with decisions from 06-02 execution * docs(phase-6): complete phase execution and verification Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(07-core-unet-models): research phase domain * docs(07): create phase plan for core UNet models * feat(07-01): migrate UNeXt2 model class to viscy-models - Copy UNeXt2 class (~70 lines) from monolithic unext2.py - Update imports to use viscy_models._components (stems, heads, blocks) - Preserve all attribute names for state dict compatibility - Export UNeXt2 from viscy_models.unet public API Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(07-01): add 6 UNeXt2 forward-pass tests and fix deconv tuple bug - Add tests: default, small backbone, multichannel, diff stack depths, deconv, stem validation - Fix deconv decoder tuple bug in UNeXt2UpStage (trailing comma created tuple not module) - Mark deconv test xfail: original code has channel mismatch in deconv forward path - All 26 tests pass (25 passed, 1 xfailed) with no regressions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(07-01): complete UNeXt2 migration plan - Add 07-01-SUMMARY.md with execution results and deviation documentation - Update STATE.md: phase 7, plan 1/2 complete, new decisions logged Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(07-02): migrate FullyConvolutionalMAE to viscy-models - Copy FCMAE and all helper classes/functions to unet/fcmae.py - Replace old viscy imports with viscy_models._components imports - Remove duplicated PixelToVoxelShuffleHead (import from _components.heads) - Fix mutable list defaults to tuples (encoder_blocks, dims) - Export both UNeXt2 and FullyConvolutionalMAE from unet/__init__.py * test(07-02): migrate 11 FCMAE tests to viscy-models - Copy all 11 test functions with zero logic changes - Update imports from viscy.unet.networks.fcmae to viscy_models.unet.fcmae - Import PixelToVoxelShuffleHead from viscy_models._components.heads - All 37 tests pass across full suite (no regressions) * docs(07-02): complete FCMAE migration plan (Phase 7 complete) - Add 07-02-SUMMARY.md with execution results - Update STATE.md: Phase 7 complete, 12 plans total, decisions logged * docs(phase-7): complete phase execution and verification Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(phase-8): research representation models migration * docs(08): create phase plan for representation models * feat(08-02): migrate BetaVae25D and BetaVaeMonai to viscy-models - Add BetaVae25D with VaeUpStage, VaeEncoder, VaeDecoder helpers - Add BetaVaeMonai wrapping MONAI VarAutoEncoder - Fix VaeDecoder mutable list defaults to tuples (COMPAT-02) - Change VaeEncoder pretrained default to False - Preserve all attribute names for state dict compatibility Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(08-01): migrate ContrastiveEncoder and ResNet3dEncoder to viscy-models - Add ContrastiveEncoder with convnext/resnet50 backbone support via timm - Add ResNet3dEncoder with MONAI ResNetFeatures backend - Fix ResNet50 bug: use encoder.num_features instead of encoder.head.fc.in_features - Add pretrained parameter (default False) for pure nn.Module semantics - Preserve state dict attribute names (stem, encoder, projection) - Share projection_mlp utility between both encoder classes * test(08-01): add 5 forward-pass tests for contrastive models - 3 tests for ContrastiveEncoder: convnext_tiny, resnet50, custom stem - 2 tests for ResNet3dEncoder: resnet18, resnet10 - Verify embedding and projection output shapes - ResNet50 test uses in_stack_depth=10 for valid stem channel alignment * test(08-02): add forward-pass tests for BetaVae25D and BetaVaeMonai - 2 BetaVae25D tests: resnet50 and convnext_tiny backbones - 2 BetaVaeMonai tests: 2D and 3D spatial configurations - Verify SimpleNamespace output with recon_x, mean, logvar, z - Fix ResNet50 expected spatial dims (64x64 not 128x128) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(08-01): complete contrastive model migration plan - Add 08-01-SUMMARY.md with execution results - Update STATE.md to Phase 8, plan 1/2 * docs(08-02): complete VAE migration plan (Phase 8 complete) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(phase-8): complete phase execution and verification Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(09): research legacy UNet models migration * docs(09): create phase plan for legacy UNet models * feat(09-01): migrate Unet2d and Unet25d to viscy-models - Copy Unet2d from v0.3.3 with import path update to viscy_models.unet._layers - Copy Unet25d from v0.3.3 with import path update to viscy_models.unet._layers - Fix mutable default num_filters=[] to num_filters=() in both models - Add module docstrings and __all__ exports - Update unet/__init__.py to export all 4 models (UNeXt2, FCMAE, Unet2d, Unet25d) - Preserve register_modules/add_module pattern for state dict compatibility - All 45 existing tests still pass Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(09-01): add pytest tests for Unet2d and Unet25d - 12 tests for Unet2d: default forward, variable depth, multichannel, residual, task mode, dropout, state dict keys, custom num_filters - 11 tests for Unet25d: default Z-compression, preserved depth, variable depth, multichannel, residual, task mode, state dict keys with skip_conv_layer, custom filters - Fix list(num_filters) conversion in both models for tuple default compatibility - Total test suite: 68 passed, 1 xfailed, 0 failures Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(09-01): complete legacy UNet migration plan - SUMMARY.md with task commits, deviations, and self-check - STATE.md updated to Phase 9 complete (15 plans total) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(phase-9): complete phase execution and verification Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(10): create phase plan for public API and CI integration * feat(10-01): add top-level re-exports for all 8 model classes - Import UNeXt2, FullyConvolutionalMAE, Unet2d, Unet25d from unet subpackage - Import ContrastiveEncoder, ResNet3dEncoder from contrastive subpackage - Import BetaVae25D, BetaVaeMonai from vae subpackage - Update __all__ with all 8 classes in alphabetical order Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(10-01): add state dict key compatibility regression tests - 24 tests covering all 8 migrated model architectures - Each model tested for parameter count, top-level prefixes, and sentinel keys - Guards COMPAT-01: state dict keys must match for checkpoint loading - Tests import from top-level viscy_models package (validates public API) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore(10-01): add viscy-models to CI test matrix - Add package dimension to test matrix (viscy-transforms, viscy-models) - Use cross-platform --cov=src/ instead of named package coverage - Matrix now produces 18 jobs (3 OS x 3 Python x 2 packages) - check job automatically aggregates all test results via alls-green Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(10-01): complete public API & CI integration plan (v1.1 milestone complete) - Add 10-01-SUMMARY.md with execution results - Update STATE.md: phase 10 complete, v1.1 milestone done, 100% progress Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(phase-10): complete phase execution and verification (v1.1 milestone complete) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: consolidate ConvBlock2D/3D into _components Move conv_block_2d.py and conv_block_3d.py from unet/_layers/ to _components/ alongside all other shared building blocks. All reusable layers now live in one place. unet/_layers/ retained as backward- compatible re-export shim. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * remove the _layers * update the readme * update the main readme * fix description in toml * changing ruff formatting , dosctrings and imports * renaming folder to components * updatet planning docs * update to components * numpy docstring * add claude.md and contributing.md --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * modularizing data package (#366) * add planning roadmap * docs: start milestone v1.1 Extract viscy-data * docs: complete viscy-data project research * docs: define milestone v1.1 requirements * docs: create milestone v1.1 roadmap (4 phases) * docs(06-package-scaffolding-and-foundation): create phase plan * feat(06-01): create viscy-data package directory structure with pyproject.toml - Add pyproject.toml with hatchling build, uv-dynamic-versioning, all base deps - Declare optional dependency groups: triplet, livecell, mmap, all - Add PEP 561 py.typed marker and tests/__init__.py - Configure pattern-prefix for independent versioning Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(06-01): add type definitions and package init with re-exports - Copy all type definitions from viscy/data/typing.py into _typing.py - Add INDEX_COLUMNS from viscy/data/triplet.py for shared access - Update typing_extensions.NotRequired to typing.NotRequired (Python >=3.11) - Create __init__.py with full re-export of all public types - Add README.md required by hatchling build Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(06-01): integrate viscy-data as workspace dependency in root pyproject.toml - Add viscy-data to root dependencies list - Register viscy-data as workspace source in [tool.uv.sources] - Verified editable install and full import chain works Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(06-01): complete package scaffolding plan with summary and state update - Add 06-01-SUMMARY.md documenting viscy-data package creation - Update STATE.md with plan position, metrics, and decisions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(06-02): extract shared utility functions into _utils.py - Extract _ensure_channel_list, _search_int_in_str, _collate_samples, _read_norm_meta from hcs.py - Extract _scatter_channels, _gather_channels, _transform_channel_wise from triplet.py - Update imports to use viscy_data._typing instead of viscy.data.typing - Add __all__ listing all 7 utility functions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(06-02): complete utility module extraction plan - Add 06-02-SUMMARY.md documenting utility extraction - Update STATE.md: Phase 6 complete, progress 80% Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(phase-6): complete phase execution * docs(07-code-migration): create phase plan * feat(07-01): migrate select.py, distributed.py, segmentation.py to viscy-data - Copy select.py with well/FOV filtering utilities (no internal viscy imports) - Copy distributed.py with ShardedDistributedSampler (no internal viscy imports) - Copy segmentation.py with viscy.data.typing -> viscy_data._typing import update - Add missing docstrings to satisfy ruff D rules Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(07-01): migrate hcs.py to viscy-data with utility import rewiring - Copy HCSDataModule, SlidingWindowDataset, MaskTestDataset from main - Replace viscy.data.typing imports with viscy_data._typing - Remove 4 utility function definitions (now in _utils.py) - Add import from viscy_data._utils for shared utilities - Remove unused re and collate_meta_tensor imports - Add missing docstrings for ruff D compliance Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(07-01): migrate gpu_aug.py to viscy-data with dependency rewiring - Copy GPUTransformDataModule, CachedOmeZarrDataset, CachedOmeZarrDataModule - Rewire viscy.data.distributed -> viscy_data.distributed - Rewire viscy.data.hcs utility imports -> viscy_data._utils - Rewire viscy.data.select -> viscy_data.select - Rewire viscy.data.typing -> viscy_data._typing - Add missing docstrings for ruff D compliance Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(07-01): complete core data module migration plan - Add 07-01-SUMMARY.md documenting migration of 5 core modules - Update STATE.md: phase 7 plan 1 of 4, decisions, metrics Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(07-03): migrate mmap_cache.py and ctmc_v1.py to viscy-data - Rewire all imports from viscy.data to viscy_data prefix - Add lazy import for tensordict with clear error message - Add docstrings for ruff D compliance Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(07-03): migrate livecell.py with lazy optional dependency imports - Rewire imports from viscy.data to viscy_data prefix - Add lazy imports for pycocotools, tifffile, torchvision - Add import guards in LiveCellDataset and LiveCellTestDataset __init__ - Add docstrings for ruff D compliance Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(07-03): migrate combined.py as-is with import rewiring - Rewire viscy.data.distributed to viscy_data.distributed - Rewire viscy.data.hcs._collate_samples to viscy_data._utils._collate_samples - Preserve all 6 public classes without structural changes - Add docstrings for ruff D compliance Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(07-02): migrate cell_classification.py and cell_division_triplet.py - Rewire imports from viscy.data to viscy_data prefix - Add lazy import for pandas in cell_classification.py with clear error message - Import _transform_channel_wise from viscy_data._utils (not triplet.py) - Import INDEX_COLUMNS and AnnotationColumns from viscy_data._typing - Add docstrings for ruff D compliance * docs(07-03): complete optional dependency module migration plan Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(07-02): complete specialized module migration plan - Add 07-02-SUMMARY.md documenting triplet, classification, and cell division module migration - Update STATE.md with position, decisions, and metrics Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(07-04): add complete public API exports to viscy_data __init__.py - Export all 45 public names (17 types, 2 utilities, 26 DataModules/Datasets/enums) - Eager imports from all 13 modules (lazy guards handled internally by each module) - Comprehensive __all__ list for IDE autocompletion and star-import support - Ruff-sorted import ordering passes all lint checks Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(07-04): complete public API exports plan - phase 7 fully done - 07-04-SUMMARY.md documenting 45 public exports and full package verification - STATE.md updated: phase 7 complete (4/4 plans), 12 total plans done Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(phase-7): complete code migration execution * docs(08-test-migration-and-validation): create phase plan * test(08-01): add conftest.py with HCS OME-Zarr fixtures for viscy-data - Copy all 6 fixtures and _build_hcs helper from main branch conftest - Replace legacy np.random.rand with np.random.default_rng (NPY002) - No viscy import changes needed (only uses third-party libs) - Provides preprocessed_hcs_dataset, small_hcs_dataset, tracks fixtures Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(08-02): complete smoke tests plan - phase 8 test migration done - Created 08-02-SUMMARY.md documenting 52 smoke tests for viscy_data - Updated STATE.md: phase 8 complete, 14 total plans executed - DATA-TST-02 satisfied: import, __all__, optional dep messages, no legacy namespace Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(08-01): migrate test_hcs, test_triplet, test_select to viscy-data package - Update imports from viscy.data.X to viscy_data - Add BatchedCenterSpatialCropd to _utils.py (fixes batch dim handling) - Fix triplet.py to use BatchedCenterSpatialCropd instead of CenterSpatialCropd - Add tensorstore to test dependency group for triplet tests - All 19 tests pass across 3 test files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(08-01): complete data test migration plan summary - Create 08-01-SUMMARY.md documenting test migration and bug fixes - Update STATE.md with BatchedCenterSpatialCropd decision revision Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(phase-8): complete test migration and validation * docs(09-ci-integration): create phase plan * feat(09-01): add viscy-data CI test jobs to GitHub Actions workflow - Add test-data job with 3x3 matrix (3 OS x 3 Python) for viscy-data - Add test-data-extras job (ubuntu-latest, Python 3.13) for extras validation - Update check job needs to aggregate all test jobs: test, test-data, test-data-extras Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(09-01): complete CI integration plan - Add 09-01-SUMMARY.md documenting viscy-data CI jobs - Update STATE.md: phase 9 complete, v1.0 milestone complete Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(phase-9): complete CI integration - milestone v1.1 done * chore: complete v1.1 milestone — Extract viscy-data Delivered: viscy-data package with 15 modules, 45 public exports, optional dependency groups, 71 tests, and tiered CI. Archives: - milestones/v1.1-ROADMAP.md - milestones/v1.1-REQUIREMENTS.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add missing pandas guards, restore conftest fixture, remove no-op CI filter - Add `if pd is None` guard in ClassificationDataModule.setup() and TripletDataModule._align_tracks_tables_with_positions() to raise helpful ImportError instead of AttributeError when pandas is absent - Fix ClassificationDataset error message to suggest `pip install pandas` instead of `pip install 'viscy-data[triplet]'` (classification doesn't need tensorstore) - Restore `num_timepoints` parameter on `_build_hcs()` and add `temporal_hcs_dataset` fixture from upstream commit 44b25b9 - Remove no-op `-m "not slow"` from test-data-extras CI job (no tests use @pytest.mark.slow) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add CLAUDE.md and update CONTRIBUTING.md Add CLAUDE.md with project-specific instructions for Claude Code sessions. Update CONTRIBUTING.md with ruff config centralization warning and numpy docstring convention note. Synced from 71009b5. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update uv.lock after rebase onto modular-viscy-staging Regenerate lockfile to include viscy-data workspace dependencies alongside viscy-models from the updated base branch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * port changes from tests-zarrv3 branch * ruff * add additional test for the main dataloaders * redundant tets 3. I think test 2 alreaady takes care of this. * rename INDEX_COLUMNS * remove unused LABEL classes * fix(livecell): assign transform result and avoid mutable defaults LiveCellTestDataset.__getitem__ discarded the return value of self.transform(sample), so MONAI transforms had no effect. Also replace mutable default lists in LiveCellDataModule.__init__ with None to prevent cross-instance state sharing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cell_classification): raise ValueError, tighten val_fovs type, fix mutable default - `raise (f"Unknown stage: {stage}")` raised a string instead of an exception — use `ValueError`. - `val_fovs: list[str] | None` was unconditionally indexed in `setup()` — remove the `None` option since it's always required. - `_subset(..., exclude_timepoints=[])` used a mutable default — replace with `None`. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(gpu_aug): remove _filter_fit_fovs override so exclude_fovs is applied CachedOmeZarrDataModule accepted exclude_fovs but its local _filter_fit_fovs override only filtered wells, silently ignoring excluded FOVs. Remove the override so the SelectWell mixin's implementation (which filters both wells and FOVs) is used. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: rename select.py to _select.py (private module) The module contains mostly private helpers (_filter_wells, _filter_fovs) and a mixin dataclass (SelectWell). Renaming to _select.py signals it is internal implementation detail. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update package descriptions to "AI x Imaging" and CLAUDE.md Update viscy-data description from "virtual staining microscopy" to "AI x Imaging tasks" in README, pyproject.toml, and __init__.py. Add viscy-models test example to CLAUDE.md. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ctmc_v1): add missing prefetch_factor attribute CTMCv1DataModule.__init__ did not set self.prefetch_factor, causing an AttributeError when train_dataloader() or val_dataloader() was called (inherited from GPUTransformDataModule). Set it to None. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add functional tests for ctmc, livecell, segmentation, classification Add unit tests for the four data modules that previously only had import smoke tests: - test_ctmc_v1.py: setup, val subsample ratio, batch shape - test_livecell.py: dataset/datamodule with mock TIFF + COCO data - test_segmentation.py: paired pred/target datasets, z-slice - test_cell_classification.py: annotation CSV, FOV split, timepoint exclusion Also adds shared fixtures to conftest.py (single_channel_hcs_pair, segmentation_hcs_pair, classification_hcs_dataset). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(livecell): add missing prefetch_factor; add dataloader iteration tests Add prefetch_factor attribute to LiveCellDataModule (same fix as 97455eb for CTMC). Add batch iteration + shape validation to classification and livecell datamodule tests to match coverage patterns in test_hcs/test_triplet. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: delete leftover select.py after rename to _select.py Commit 5b132f9 renamed select.py to _select.py but did not remove the original file. Nothing imports from it. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use explicit positions[0] instead of leaked loop variable CachedOmeZarrDataset and MmappedDataset both built self.channels using the loop variable `position` after iterating, implicitly depending on the last element. Use positions[0] to be explicit and avoid UnboundLocalError on empty input. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(livecell): handle empty annotations and correct return type Guard torch.stack against empty annotation lists in LiveCellTestDataset.__getitem__ when load_labels=True, returning properly shaped empty tensors instead of crashing. Fix _parse_image_names return type annotation: list[Path] -> list[str]. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(segmentation): defer open_ome_zarr from __init__ to setup() Store only paths in __init__ and open OME-Zarr stores in setup("test"), consistent with other DataModules in the package and Lightning conventions for resource lifecycle. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * print to logger * refactor(triplet): remove BatchedCenterSpatialCropd from viscy-data The transform already exists in viscy-transforms and viscy-data should not depend on it. Replace the final crop with a shape validation check in on_after_batch_transfer() and require initial_yx_patch_size to match the desired output size. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(docs): convert Sphinx-style docstrings to numpy style in _utils.py Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(init): lazy-load submodules via PEP 562 __getattr__ Replace eager imports of all DataModule/Dataset submodules with on-demand loading. Modules with optional dependencies (triplet, livecell, mmap_cache) are no longer imported at `import viscy_data` time. Add __init__.pyi stub for type-checker/IDE support. Also split CI test-data job to run without --all-extras so the base package is validated independently from optional dependencies. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: guard None dereferences and replace mutable default arguments - cell_classification.py: guard _read_norm_meta() returning None - hcs.py: guard MaskTestDataset with ground_truth_masks=None, add missing array_key parameter, replace mutable default [] with None - triplet.py, cell_division_triplet.py: replace mutable default [] with None for normalizations/augmentations parameters Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: correct return type annotations in _utils.py _gather_channels and _transform_channel_wise return Tensor, not list[Tensor]. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(combined): check stage before calling dm.setup() Move the unsupported-stage guard to the top of setup() in ConcatDataModule and CachedConcatDataModule so constituent data modules are not set up for stages that will be rejected. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add tests for cell_division_triplet, mmap_cache, and HCS test stage - test_cell_division_triplet.py: 11 smoke tests for dataset and datamodule - test_mmap_cache.py: 5 smoke tests (skipped when tensordict missing) - test_hcs.py: add setup("test") coverage for MaskTestDataset Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Eduardo Hirata-Miyasaki <edhiratam@gmail.com> Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Applications: Cytoland refactor (#379) * test(10-01): add state dict key compatibility regression tests - 24 tests covering all 8 migrated model architectures - Each model tested for parameter count, top-level prefixes, and sentinel keys - Guards COMPAT-01: state dict keys must match for checkpoint loading - Tests import from top-level viscy_models package (validates public API) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore(10-01): add viscy-models to CI test matrix - Add package dimension to test matrix (viscy-transforms, viscy-models) - Use cross-platform --cov=src/ instead of named package coverage - Matrix now produces 18 jobs (3 OS x 3 Python x 2 packages) - check job automatically aggregates all test results via alls-green Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(10-01): complete public API & CI integration plan (v1.1 milestone complete) - Add 10-01-SUMMARY.md with execution results - Update STATE.md: phase 10 complete, v1.1 milestone done, 100% progress Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(phase-10): complete phase execution and verification (v1.1 milestone complete) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: consolidate ConvBlock2D/3D into _components Move conv_block_2d.py and conv_block_3d.py from unet/_layers/ to _components/ alongside all other shared building blocks. All reusable layers now live in one place. unet/_layers/ retained as backward- compatible re-export shim. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * remove the _layers * update the readme * docs: start milestone v1.1 Extract viscy-data * update the main readme * docs: complete viscy-data project research * docs: define milestone v1.1 requirements * docs: create milestone v1.1 roadmap (4 phases) * docs(06-package-scaffolding-and-foundation): create phase plan * feat(06-01): create viscy-data package directory structure with pyproject.toml - Add pyproject.toml with hatchling build, uv-dynamic-versioning, all base deps - Declare optional dependency groups: triplet, livecell, mmap, all - Add PEP 561 py.typed marker and tests/__init__.py - Configure pattern-prefix for independent versioning Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(06-01): add type definitions and package init with re-exports - Copy all type definitions from viscy/data/typing.py into _typing.py - Add INDEX_COLUMNS from viscy/data/triplet.py for shared access - Update typing_extensions.NotRequired to typing.NotRequired (Python >=3.11) - Create __init__.py with full re-export of all public types - Add README.md required by hatchling build Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(06-01): integrate viscy-data as workspace dependency in root pyproject.toml - Add viscy-data to root dependencies list - Register viscy-data as workspace source in [tool.uv.sources] - Verified editable install and full import chain works Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(06-01): complete package scaffolding plan with summary and state update - Add 06-01-SUMMARY.md documenting viscy-data package creation - Update STATE.md with plan position, metrics, and decisions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(06-02): extract shared utility functions into _utils.py - Extract _ensure_channel_list, _search_int_in_str, _collate_samples, _read_norm_meta from hcs.py - Extract _scatter_channels, _gather_channels, _transform_channel_wise from triplet.py - Update imports to use viscy_data._typing instead of viscy.data.typing - Add __all__ listing all 7 utility functions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(06-02): complete utility module extraction plan - Add 06-02-SUMMARY.md documenting utility extraction - Update STATE.md: Phase 6 complete, progress 80% Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(phase-6): complete phase execution * docs(07-code-migration): create phase plan * feat(07-01): migrate select.py, distributed.py, segmentation.py to viscy-data - Copy select.py with well/FOV filtering utilities (no internal viscy imports) - Copy distributed.py with ShardedDistributedSampler (no internal viscy imports) - Copy segmentation.py with viscy.data.typing -> viscy_data._typing import update - Add missing docstrings to satisfy ruff D rules Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(07-01): migrate hcs.py to viscy-data with utility import rewiring - Copy HCSDataModule, SlidingWindowDataset, MaskTestDataset from main - Replace viscy.data.typing imports with viscy_data._typing - Remove 4 utility function definitions (now in _utils.py) - Add import from viscy_data._utils for shared utilities - Remove unused re and collate_meta_tensor imports - Add missing docstrings for ruff D compliance Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(07-01): migrate gpu_aug.py to viscy-data with dependency rewiring - Copy GPUTransformDataModule, CachedOmeZarrDataset, CachedOmeZarrDataModule - Rewire viscy.data.distributed -> viscy_data.distributed - Rewire viscy.data.hcs utility imports -> viscy_data._utils - Rewire viscy.data.select -> viscy_data.select - Rewire viscy.data.typing -> viscy_data._typing - Add missing docstrings for ruff D compliance Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(07-01): complete core data module migration plan - Add 07-01-SUMMARY.md documenting migration of 5 core modules - Update STATE.md: phase 7 plan 1 of 4, decisions, metrics Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(07-03): migrate mmap_cache.py and ctmc_v1.py to viscy-data - Rewire all imports from viscy.data to viscy_data prefix - Add lazy import for tensordict with clear error message - Add docstrings for ruff D compliance Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(07-03): migrate livecell.py with lazy optional dependency imports - Rewire imports from viscy.data to viscy_data prefix - Add lazy imports for pycocotools, tifffile, torchvision - Add import guards in LiveCellDataset and LiveCellTestDataset __init__ - Add docstrings for ruff D compliance Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(07-03): migrate combined.py as-is with import rewiring - Rewire viscy.data.distributed to viscy_data.distributed - Rewire viscy.data.hcs._collate_samples to viscy_data._utils._collate_samples - Preserve all 6 public classes without structural changes - Add docstrings for ruff D compliance Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(07-02): migrate cell_classification.py and cell_division_triplet.py - Rewire imports from viscy.data to viscy_data prefix - Add lazy import for pandas in cell_classification.py with clear error message - Import _transform_channel_wise from viscy_data._utils (not triplet.py) - Import INDEX_COLUMNS and AnnotationColumns from viscy_data._typing - Add docstrings for ruff D compliance * docs(07-03): complete optional dependency module migration plan Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(07-02): complete specialized module migration plan - Add 07-02-SUMMARY.md documenting triplet, classification, and cell division module migration - Update STATE.md with position, decisions, and metrics Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(07-04): add complete public API exports to viscy_data __init__.py - Export all 45 public names (17 types, 2 utilities, 26 DataModules/Datasets/enums) - Eager imports from all 13 modules (lazy guards handled internally by each module) - Comprehensive __all__ list for IDE autocompletion and star-import support - Ruff-sorted import ordering passes all lint checks Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(07-04): complete public API exports plan - phase 7 fully done - 07-04-SUMMARY.md documenting 45 public exports and full package verification - STATE.md updated: phase 7 complete (4/4 plans), 12 total plans done Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(phase-7): complete code migration execution * docs(08-test-migration-and-validation): create phase plan * test(08-01): add conftest.py with HCS OME-Zarr fixtures for viscy-data - Copy all 6 fixtures and _build_hcs helper from main branch conftest - Replace legacy np.random.rand with np.random.default_rng (NPY002) - No viscy import changes needed (only uses third-party libs) - Provides preprocessed_hcs_dataset, small_hcs_dataset, tracks fixtures Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(08-02): complete smoke tests plan - phase 8 test migration done - Created 08-02-SUMMARY.md documenting 52 smoke tests for viscy_data - Updated STATE.md: phase 8 complete, 14 total plans executed - DATA-TST-02 satisfied: import, __all__, optional dep messages, no legacy namespace Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(08-01): migrate test_hcs, test_triplet, test_select to viscy-data package - Update imports from viscy.data.X to viscy_data - Add BatchedCenterSpatialCropd to _utils.py (fixes batch dim handling) - Fix triplet.py to use BatchedCenterSpatialCropd instead of CenterSpatialCropd - Add tensorstore to test dependency group for triplet tests - All 19 tests pass across 3 test files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(08-01): complete data test migration plan summary - Create 08-01-SUMMARY.md documenting test migration and bug fixes - Update STATE.md with BatchedCenterSpatialCropd decision revision Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(phase-8): complete test migration and validation * docs(09-ci-integration): create phase plan * feat(09-01): add viscy-data CI test jobs to GitHub Actions workflow - Add test-data job with 3x3 matrix (3 OS x 3 Python) for viscy-data - Add test-data-extras job (ubuntu-latest, Python 3.13) for extras validation - Update check job needs to aggregate all test jobs: test, test-data, test-data-extras Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(09-01): complete CI integration plan - Add 09-01-SUMMARY.md documenting viscy-data CI jobs - Update STATE.md: phase 9 complete, v1.0 milestone complete Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(phase-9): complete CI integration - milestone v1.1 done * chore: complete v1.1 milestone — Extract viscy-data Delivered: viscy-data package with 15 modules, 45 public exports, optional dependency groups, 71 tests, and tiered CI. Archives: - milestones/v1.1-ROADMAP.md - milestones/v1.1-REQUIREMENTS.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * harmonize the planning between the modular-data and modular-models * viscy-utils package * add applications/dynaclr * update the monorepo uv * moving files around * update planning * docs: start milestone v2.1 DynaCLR Integration Validation * docs: define milestone v2.1 requirements * docs: create milestone v2.1 roadmap (2 phases) * docs(18-training-validation): create phase plan * feat(18-01): add training integration tests for ContrastiveModule - Add fast_dev_run tests for TripletMarginLoss and NTXentLoss code paths - Add parametrized config class_path resolution tests for fit.yml and predict.yml - Add tensorboard as test dependency for TensorBoardLogger in integration tests - Fix workspace exclude to skip non-package application directories - Use 2D-compatible synthetic data shapes (1,1,4,4) for render_images compatibility Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(18-01): complete training integration tests plan Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-18): complete phase execution * docs(19-inference-reproducibility): create phase plan * chore(19-01): add anndata test dependency and HPC conftest fixtures - Add anndata to dynacrl test dependency group - Create conftest.py with HPC path constants, skip markers, and fixtures - Update uv.lock Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(19-01): add inference reproducibility integration tests - Create test_inference_reproducibility.py with 2 HPC integration tests - test_checkpoint_loads_into_modular_contrastive_module (INFER-01) - test_predict_embeddings_and_exact_match (INFER-02 + INFER-03) - Fix lazy imports in EmbeddingWriter to avoid unconditional umap import - Fix anndata nullable string compatibility in write_embedding_dataset - Tests skip gracefully when HPC paths or GPU unavailable Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(19-01): complete inference reproducibility plan - Add 19-01-SUMMARY.md with execution results and deviation documentation - Update STATE.md: Phase 19 complete, v2.1 milestone finished Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add seed_everything(42) to all integration tests Ensures reproducibility by seeding all tests consistently. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(phase-19): complete phase execution * restructure the examples folder and ruff * update readme.me hallucination * update the readmes * - Add `viscy` console script in viscy-utils pointing to viscy_utils.cli:main - Add jsonargparse[signatures] dependency for LightningCLI - Add 4 CLI smoke tests (help, subcommands, fit --help, predict --help) - Replace conda/anaconda with uv in SLURM scripts - Update SLURM scripts to use `viscy fit/predict` instead of old monolith * add the CLI for running training and prediction * default embedding writer to None * import within the function * ruff * dynaclr typo * rename folder to dynaclr * add the classifiers here * docs: start milestone v2.2 Composable Sampling Framework * docs: define milestone v2.2 requirements * docs: create milestone v2.2 roadmap (6 phases) * docs(20): capture phase context * docs(20): create phase plan for experiment configuration * test(20-01): add failing tests for ExperimentConfig and ExperimentRegistry - 19 test cases covering config creation, defaults, channel maps, validation errors, YAML loading, tau-range conversion, and lookups - All tests fail with ModuleNotFoundError (module not yet implemented) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(20-01): implement ExperimentConfig and ExperimentRegistry - ExperimentConfig dataclass with all fields and defaults - ExperimentRegistry with fail-fast validation at __post_init__: empty check, duplicate names, source_channel membership, channel count consistency, interval_minutes positivity, condition_wells non-empty, data_path existence, zarr channel match - channel_maps: per-experiment source position -> zarr index mapping - from_yaml classmethod for YAML config loading - tau_range_frames for hours-to-frames conversion with warning - get_experiment lookup by name with KeyError - All 19 tests pass Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(20-01): clean up imports and exclude stale dynacrl workspace member - Fix ruff I001 (import sorting) and F401 (unused import) in test file - Exclude applications/dynacrl (typo) from uv workspace to unblock builds Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(20-01): complete ExperimentConfig/ExperimentRegistry plan - SUMMARY.md with TDD execution results, self-check passed - STATE.md updated with position, decisions, session continuity Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(20-02): add explicit deps and top-level experiment API exports - Add iohub>=0.3a2 and pyyaml as explicit dependencies in dynaclr pyproject.toml - Re-export ExperimentConfig and ExperimentRegistry from dynaclr __init__.py - Both classes now importable via `from dynaclr import ExperimentConfig, ExperimentRegistry` Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(20-02): add example multi-experiment YAML configuration - Demonstrate positional channel alignment across 2 experiments - SEC61 (30min interval, ER) and TOMM20 (15min interval, mito) - Show condition_wells with infected/uninfected/mock conditions - Include comments explaining channel alignment and tau_range conversion Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(20-02): complete package wiring and example config plan - SUMMARY.md with execution results and self-check - STATE.md updated: Phase 20 complete, 20/25 phases (80%) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-20): complete phase execution Phase 20 Experiment Configuration verified (11/11 must-haves). ExperimentConfig + ExperimentRegistry with TDD, package wiring, example YAML. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(21): create phase plan for Cell Index & Lineage * test(21-01): add failing tests for MultiExperimentIndex - 17 test cases covering CELL-01 (unified tracks), CELL-02 (lineage), CELL-03 (border clamping) - All fail with ModuleNotFoundError (dynaclr.index not yet implemented) - Test fixtures create mini OME-Zarr stores with tracking CSVs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(21-01): implement MultiExperimentIndex with lineage and border clamping - Unified tracks DataFrame from all experiments with enriched columns - Lineage reconstruction linking daughters to root ancestor via parent_track_id - Border clamping: retains border cells with shifted patch origins instead of exclusion - All 23 tests pass Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(21-01): fix lint issues and export MultiExperimentIndex - Remove unused variable (F841) in test_global_track_id_unique_across_experiments - Use .to_numpy() instead of .values (PD011) in test_exclude_fovs_filter - Export MultiExperimentIndex from dynaclr __init__.py Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(21-01): complete MultiExperimentIndex plan summary and state update - 21-01-SUMMARY.md with full execution documentation - STATE.md updated for 21-01 completion, decisions, session continuity Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(21-02): add failing tests for valid anchors, properties, and summary - 8 tests for valid_anchors: basic validity, subset check, end-of-track exclusion, lineage continuity, different tau ranges, empty tracks, gap handling, self-exclusion - 9 tests for properties/summary: experiment_groups, condition_groups, summary() - All 17 new tests fail with TypeError (tau_range_hours not yet accepted) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(21-02): implement valid_anchors, experiment_groups, condition_groups, summary - Add tau_range_hours parameter to MultiExperimentIndex.__init__ - _compute_valid_anchors: per-experiment tau conversion, lineage-based lookup - experiment_groups/condition_groups properties returning index arrays - summary() with experiment counts, observation counts, per-experiment breakdowns - All 40 tests pass (23 existing + 17 new) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(21-02): complete valid anchors plan - SUMMARY.md with self-check passed - STATE.md updated: Phase 21 complete, ready for Phase 22 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(22): research batch sampling phase domain * docs(22): create phase plan for batch sampling * test(22-01): add failing tests for FlexibleBatchSampler - Experiment-aware batching: single-experiment restriction, all experiments appear - Condition balancing: 2-condition and 3-condition proportional tests - Leaky mixing: zero leak, 20% leak injection, no-effect when not experiment-aware - Small group fallback: no crash, warning emission - Determinism: same seed/epoch reproduces, set_epoch changes sequence - Sampler protocol: yields list[int], correct __len__ - DDP partitioning: disjoint interleaved batches across ranks Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(22-01): implement FlexibleBatchSampler with experiment-aware, condition-balanced, leaky mixing - FlexibleBatchSampler(Sampler[list[int]]) with cascade batch construction - experiment_aware=True restricts each batch to a single experiment - condition_balanced=True balances condition representation per batch - leaky > 0.0 injects cross-experiment samples into restricted batches - Deterministic via np.random.default_rng(seed + epoch) - DDP support via interleaved batch partitioning across ranks - Small group fallback to replacement sampling with logged warning - Pre-computed group indices at __init__ for O(1) lookup - Fix lint issues in test file (import sorting, .values -> .to_numpy(), nunique -> len(unique)) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(22-01): export FlexibleBatchSampler from viscy_data package - Add FlexibleBatchSampler to viscy_data.__init__.py public API - Place import in alphabetically correct position for ruff isort compliance - Add to __all__ exports under Utilities section Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(22-01): complete FlexibleBatchSampler core plan - Create 22-01-SUMMARY.md with TDD execution results - Update STATE.md: plan 01/02 complete, decisions, session continuity Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(22-02): add failing tests for temporal enrichment, DDP coverage, validation - 6 temporal enrichment tests (focal concentration, global_fraction edge cases, validation) - 5 DDP disjoint coverage tests (interleaving, coverage, epoch reproducibility) - 3 validation guard tests (missing experiment/condition/hpi columns) - 2 package import tests (import, __all__) - All 9 new feature tests fail as expected (RED) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(22-02): implement temporal enrichment, validation guards, DDP coverage - Add temporal_enrichment, temporal_window_hours, temporal_global_fraction params - Implement _enrich_temporal: focal/global sampling from experiment pool - Add column validation guards for experiment/condition/hpi columns - Conditional precomputation: only groupby columns when feature enabled - Fix stale smoke test __all__ count (45 -> 46) from Plan 01 - All 35 sampler tests pass, 107 total viscy-data tests pass Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(22-02): complete temporal enrichment + DDP plan - 22-02-SUMMARY.md with all metrics, decisions, deviations - STATE.md advanced to Phase 23, progress 22/25 (88%) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-22): complete batch sampling phase execution Phase 22 verified: FlexibleBatchSampler with all 5 SAMP requirements. 5/5 must-haves passed. 35 tests, 107 full suite pass. No regressions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(23): create phase plan for Loss & Augmentation * test(23-01): add failing tests for NTXentHCL - 12 test cases covering subclass, beta=0 equivalence, hard negatives, gradients, temperature effect, edge cases, defaults, and CUDA - All fail with ModuleNotFoundError (dynaclr.loss not yet created) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(23-02): add failing tests for ChannelDropout and variable tau sampling - 11 tests for ChannelDropout: zeros, probability bounds, eval mode, per-sample, dtype, input safety, multi-channel, CUDA - 7 tests for sample_tau: range, exponential decay, uniform, single value, determinism, return type * feat(23-02): implement ChannelDropout and variable tau sampling - ChannelDropout nn.Module: per-sample channel zeroing on (B,C,Z,Y,X) tensors - sample_tau: exponential decay weighted sampling for temporal offsets * feat(23-01): implement NTXentHCL with hard-negative concentration - NTXentHCL subclasses NTXentLoss from pytorch_metric_learning - beta=0.0 delegates to parent for exact numerical equivalence - beta>0 applies exp(beta*sim) reweighting on negatives in denominator - Normalized weights preserve loss magnitude across beta values - All 11 tests pass (1 CUDA test skipped on macOS) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(23-02): add ChannelDropout and sample_tau to package exports - Export ChannelDropout from viscy_data top-level - Export sample_tau from dynaclr top-level - Include NTXentHCL export added by linter * docs(23-02): complete ChannelDropout and tau sampling plan - Summary with TDD metrics, decisions, self-check - STATE.md updated for Phase 23 completion Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(23-01): complete NTXentHCL loss plan - Created 23-01-SUMMARY.md with TDD execution results - Updated STATE.md with HCL implementation decisions - Self-check passed: all artifacts and commits verified Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-23): complete loss & augmentation phase execution Phase 23 verified: NTXentHCL (3/3 LOSS reqs), ChannelDropout (AUG-01), sample_tau (AUG-03). AUG-02 wiring deferred to Phase 24 by design. 30 tests pass across loss, channel_dropout, tau_sampling. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(24): create phase plan * test(24-01): add failing tests for MultiExperimentTripletDataset - 7 test cases covering __getitems__ return format, norm_meta, lineage-aware positive sampling, division event traversal, channel remapping, predict mode, and dataset length - All tests fail with ModuleNotFoundError (RED phase) * feat(24-01): implement MultiExperimentTripletDataset with lineage-aware sampling - __getitems__ returns batch dicts with anchor/positive Tensors (B,C,Z,Y,X) - Lineage-aware positive sampling via pre-built (experiment, lineage_id) lookup - Division events traversed naturally via shared lineage_id - Per-experiment channel remapping using registry.channel_maps - Tensorstore I/O with SLURM-aware context and per-FOV caching - Predict mode returns anchor + TrackingIndex dicts - Exponential decay tau sampling with fallback to full range scan * refactor(24-01): add MultiExperimentTripletDataset to package exports - Export from dynaclr.__init__ for public API access * docs(24-01): complete MultiExperimentTripletDataset plan - SUMMARY.md with TDD commits, decisions, self-check - STATE.md updated: position 24-01, decisions, session continuity * update uv * test(24-02): add failing tests for MultiExperimentDataModule - 6 test cases covering hyperparameter exposure, experiment-level split, FlexibleBatchSampler wiring, val dataloader, transforms, ChannelDropout - RED phase: all tests fail with ModuleNotFoundError Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(24-02): implement MultiExperimentDataModule with experiment-level split - MultiExperimentDataModule composes FlexibleBatchSampler + Dataset + ChannelDropout + ThreadDataLoader with collate_fn=lambda x: x - Train/val split by whole experiments via val_experiments parameter - All sampling, augmentation, and loss hyperparameters exposed as __init__ params - on_after_batch_transfer applies normalizations + augmentations + final crop + ChannelDropout with proper norm_meta handling for all-None case - 6 TDD tests passing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(24-02): add MultiExperimentDataModule to dynaclr package exports - Import MultiExperimentDataModule from dynaclr.datamodule - Add to __all__ for top-level importability Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(24-02): complete MultiExperimentDataModule plan - Summary with TDD commits, decisions, and deviation documentation - STATE.md updated: Phase 24 complete, 96% progress, ready for Phase 25 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-24): complete dataset & datamodule phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(25): create phase plan * docs(phase-25): complete integration phase plan Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(25-01): add end-to-end multi-experiment integration tests - Create test_multi_experiment_fast_dev_run: 2 experiments with different channel sets (GFP vs RFP), fast_dev_run with NTXentHCL loss - Create test_multi_experiment_fast_dev_run_with_all_sampling_axes: experiment_aware + condition_balanced + temporal_enrichment enabled - Synthetic data helpers for multi-channel HCS OME-Zarr creation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(25-01): add multi-experiment YAML config and class_path validation test - Create multi_experiment_fit.yml with MultiExperimentDataModule, NTXentHCL loss, all sampling axes, generic channel names (ch_0/ch_1) - Add test_multi_experiment_config_class_paths_resolve validating all class_path entries in the config resolve to importable Python classes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(25-01): complete integration plan - milestone v2.2 complete - Add 25-01-SUMMARY.md documenting end-to-end integration validation - Update STATE.md: phase 25/25 complete, progress 100%, milestone v2.2 done Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-25): complete integration phase execution — v2.2 milestone shipped Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * add the smoothness and dynamic range comparison * add the applications/qc * bug qc metrics exposing the device * add batch predict * adding cli for reduce dimensionality composable * add example configs for model comparision and smoothness * add the biological annotations to the zattrs * adding airtable logic * harmonize and remove duplication between airtable and qc. moving most things to airtable * cleanup readme for airtable * add callback to store embeddings every n epochs and store metadata to the anndata.uns * fix the apply-linear classifiers to make sure we use the model and version. * Exclude untracked applications/dynacell from uv workspace Local debris directory (hydra outputs, pycache) has no pyproject.toml and breaks uv lock when matched by applications/* glob. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(26): capture phase context * docs(state): record phase 26 context session * docs(26): create phase plans * feat(26-01): extract HCSPredictionWriter to viscy-utils callbacks - Create prediction_writer.py with HCSPredictionWriter, _pad_shape, _resize_image, _blend_in - Use TYPE_CHECKING guard for viscy_data imports (HCSDataModule, Sample) - Add numpy-style docstrings to all functions and class - Re-export HCSPredictionWriter from callbacks __init__.py - Fix pre-existing INDEX_COLUMNS -> ULTRACK_INDEX_COLUMNS in embedding_snapshot.py and embedding_writer.py - Add missing docstrings to embedding_snapshot.py public methods Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(26-01): extract MixedLoss to viscy-utils losses submodule - Create losses/ submodule with mixed_loss.py containing MixedLoss class - Uses ms_ssim_25d from viscy_utils.evaluation.metrics internally - Convert docstrings to numpy-style - Re-export MixedLoss from losses __init__.py Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(26-01): create translation application scaffold with workspace registration - Create applications/translation/ with src layout following dynaclr pattern - Add pyproject.toml with hatchling build, uv-dynamic-versioning, and workspace deps - Add README.md required by hatchling readme field - Add __main__.py delegating to viscy_utils.cli.main for LightningCLI entry point - Add example YAML configs (fit.yml, predict.yml) with HCSPredictionWriter callback - Create empty tests/__init__.py - Register viscy-translation in root pyproject.toml workspace sources - Update uv.lock with new workspace member Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(26-01): complete shared infra extraction + app scaffold plan - Create 26-01-SUMMARY.md with execution results - Update STATE.md with plan progress, decisions, session info - Update ROADMAP.md marking 26-01 as complete Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(26-02): migrate translation engine and evaluation modules - Copy engine.py with VSUNet, FcmaeUNet, AugmentedPredictionVSUNet, MaskedMSELoss - Copy evaluation.py with SegmentationMetrics2D - Update all imports to new package paths (viscy_data, viscy_models, viscy_utils) - Remove MixedLoss class from engine.py (now imported from viscy_utils.losses) - Update __init__.py with top-level re-exports Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(26-02): add translation engine test suite - Import tests for all public exports (VSUNet, FcmaeUNet, etc.) - VSUNet init and forward pass smoke tests with synthetic data - State dict key regression test for checkpoint compatibility - MixedLoss integration test (from viscy_utils.losses) - FcmaeUNet init test - No old import paths grep test Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(26-02): complete engine migration plan - SUMMARY.md with 2 task commits, 2 auto-fixed deviations - STATE.md updated: phase 26 complete, 20/25 phases (80%) - ROADMAP.md updated with plan progress Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(26): complete UAT - 8 passed, 0 issues * add the pseudotime evals * re-structure pseudotime folder * add the linear classifier evals and restructure folder path * add evaluations to dynaclr package * cli and linear classifier init * fix(translation): address engine and evaluation bugs - Fix operator precedence bug in evaluation.py boolean condition - Fi…
1 parent b5c6ed9 commit 85f0b88

149 files changed

Lines changed: 9424 additions & 2460 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
---
2+
name: hf-dynacell
3+
description: >-
4+
Develop, deploy, and maintain the DynaCell virtual-staining HuggingFace demo
5+
hosted at biohub/dynacell (ZeroGPU). Covers hosting in the biohub Dynacell
6+
resource group, the private-artifacts + HF_TOKEN access model, ZeroGPU
7+
configuration, and the deploy/smoke-test workflow. Use when the user asks to
8+
"push the dynacell space", "update the dynacell demo", "upload checkpoints",
9+
"deploy the hf space", mentions biohub/dynacell, or works in
10+
applications/dynacell/examples/hf_demo/.
11+
---
12+
13+
# DynaCell HuggingFace Demo — maintenance
14+
15+
Operational reference for the demo. Content + file layout live in
16+
`applications/dynacell/examples/hf_demo/AGENT.md`; this skill holds hosting,
17+
access, ZeroGPU, and deploy knowledge.
18+
19+
## Hosting — biohub org, Dynacell resource group
20+
21+
Three repos under **`biohub`**, all in the **Dynacell** resource group
22+
(`id 6a234bb4507cbbbb04456767`):
23+
24+
| Repo | Type | Visibility |
25+
| --- | --- | --- |
26+
| `biohub/dynacell` | Space (Gradio, ZeroGPU) | private (→ public at launch) |
27+
| `biohub/dynacell-checkpoints` | model | **private** |
28+
| `biohub/dynacell-demo-data` | dataset | **private** |
29+
30+
RG members who can push: `edyoshikun`, `shalinmehta` (admin); `alxndrkalinin`,
31+
`dihan-zheng` (write). Migrated from `dihan-zheng/*` (byte-for-byte mirror).
32+
33+
Moving an *existing* repo into a resource group needs org-admin. As org-write
34+
members, create repos directly in the group with
35+
`create_repo(..., resource_group_id="6a234bb4507cbbbb04456767")` — the upload
36+
scripts do this.
37+
38+
## Access model — public demo, private artifacts
39+
40+
Decision (2026-06): checkpoints and demo data stay **private** even when the
41+
Space is public. The Space downloads them server-side via an `HF_TOKEN`
42+
**secret** (Space → Settings → Variables and secrets) — a **fine-grained,
43+
read-only** token scoped to exactly those two repos. Public visitors get only
44+
outputs (plots, PCC); raw artifacts stay gated. If the token is revoked or
45+
loses RG read access, runtime downloads break (rotate via "Replace").
46+
47+
## ZeroGPU configuration (validated 2026-06)
48+
49+
- Hardware `zero-a10g` (RTX Pro 6000 Blackwell, 48 GB). Enterprise quota
50+
60 GPU-min/day, shared.
51+
- **`python_version: "3.12"` required** in the Space README — the default base
52+
image is 3.10, but VisCy needs `>=3.12` (else
53+
`BUILD_ERROR: viscy-data requires ... not in '>=3.12'`). ZeroGPU supports
54+
Python 3.12.12 / 3.10.13, PyTorch 2.8→latest, Gradio SDK only.
55+
- `requirements.txt` includes `spaces`; `@spaces.GPU(duration=120)` decorates
56+
`run_prediction` and `compute_trajectory` (`predict_runner.py`).
57+
- A subprocess inside `@spaces.GPU` inherits the GPU, so the regression
58+
`dynacell predict` subprocess works on ZeroGPU. `gr.Progress` args work under
59+
the decorator. Keep `import torch` lazy (no CUDA at import).
60+
- Tuning: CELL-Diff at 50–100 ODE steps can approach the 120 s ceiling on a cold
61+
call; raise `duration` or use a dynamic-duration callable if timeouts appear.
62+
63+
## Code pointers
64+
65+
`CHECKPOINT_REPO` (`predict_runner.py`), `_DEMO_REPO` (`app.py`), and the README
66+
`models:`/`datasets:` fields all point at `biohub/*`.
67+
68+
## Deploy
69+
70+
1. `hf auth login` (token with write to the Dynacell RG).
71+
2. Edit `hf_space/`.
72+
3. Push: `uv run python applications/dynacell/examples/hf_demo/upload_hf_space.py`.
73+
4. Watch: `HfApi().get_space_runtime("biohub/dynacell").stage`. Build/run logs at
74+
`GET /api/spaces/biohub/dynacell/logs/{build,run}` are **SSE streams** — read
75+
with `stream=True` + a line cap; a plain GET hangs.
76+
77+
Re-publish checkpoints from HPC: edit the path map in `upload_checkpoints.py`
78+
and run it where `/hpc/projects/...` is mounted.
79+
80+
## Smoke test
81+
82+
Drive endpoints with `gradio_client` (`Client("biohub/dynacell", token=get_token())`;
83+
token optional now the Space is public). `/load_demo_data` renders the data view.
84+
The run endpoints `/run_regression` and `/run_generative` read data from `gr.State`,
85+
which the client resets per call — they can't be driven headlessly, so exercise
86+
them in the live UI. Last validated GPU compute: regression VSCyto3D → Spectral
87+
PCC ~0.95 (~35 s); CELL-Diff trajectory (10 steps) ~74 s.
88+
89+
## State
90+
91+
- Space `biohub/dynacell` is **public** (set 2026-06); checkpoints + dataset
92+
repos stay **private**, so the `HF_TOKEN` secret must remain.
93+
94+
## Launch checklist
95+
96+
- Fill citation/DOI TODOs in `cards/*.md` and the uploaded READMEs.
97+
- Re-pin `requirements.txt` VisCy refs off `@dynacell-models` once merged/renamed.
98+
- Only if the artifact repos are ever made public: the `HF_TOKEN` secret can then
99+
be removed.

0 commit comments

Comments
 (0)