Commit 85f0b88
feat(dynacell): HF demo + eval metrics/perf refactor + pix2pix3d leaves + dataset notebook (#449)
* VisCy Modular: Transforms and Monorepo Skeleton (#356)
* refactor: restructure viscy into uv workspace monorepo with viscy-transforms subpackage
BREAKING CHANGE: - Import path changed: from viscy.transforms import X → from viscy_transforms import X
- Removed legacy modules: viscy.utils, viscy.cli, viscy.unet, viscy.evaluation
- Removed applications/, examples/, and docs/ directories
Signed-off-by: Sricharan Reddy Varra <sricharan.varra@biohub.org>
* docs: updated readme with citations / examples from main
Signed-off-by: Sricharan Reddy Varra <sricharan.varra@biohub.org>
* docs: added symlinking uv cache on hpc systems
Signed-off-by: Sricharan Reddy Varra <sricharan.varra@biohub.org>
* add the jupyter and ipykernel to a optional visual group
* re organize the transforms
* add jupyternotebook back
* build: updated some dep groups
Signed-off-by: Sricharan Reddy Varra <sricharan.varra@biohub.org>
* build: added matplotlib back in
Signed-off-by: Sricharan Reddy Varra <sricharan.varra@biohub.org>
* build: add ruff and prek to the dev dep group
* docs: correct the docstring for the transform that doesn't exist lol
Signed-off-by: Sricharan Reddy Varra <sricharan.varra@biohub.org>
---------
Signed-off-by: Sricharan Reddy Varra <sricharan.varra@biohub.org>
Co-authored-by: Sricharan Reddy Varra <sricharan.varra@biohub.org>
Co-authored-by: Eduardo Hirata-Miyasaki <edhiratam@gmail.com>
Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com>
* docs: map existing codebase
* Modularizing the model's folder (#365)
* add planning roadmap
* docs: start milestone v1.1 Models
* docs: define milestone v1.1 requirements
* docs: create milestone v1.1 roadmap (5 phases)
* docs(package-scaffold-shared-components): research phase domain
* docs(06): create phase plan - package scaffold and shared components
* feat(06-01): create viscy-models package scaffold
- Add pyproject.toml with hatchling build, torch/timm/monai/numpy deps
- Create src layout with _components, unet, contrastive, vae subpackages
- Add PEP 561 py.typed marker
- Add test scaffolding with device fixture
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(06-01): register viscy-models in workspace
- Add viscy-models to root dependencies and uv sources
- Update lockfile with timm and viscy-models dependencies
- Verified: uv sync, import, and pytest collection all succeed
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(06-01): complete package scaffold plan
- Add 06-01-SUMMARY.md with execution results
- Update STATE.md with position, metrics, and decisions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(06-02): extract shared components into _components/ module
- stems.py: UNeXt2Stem, StemDepthtoChannels from v0.3.3 unext2.py
- heads.py: PixelToVoxelHead, UnsqueezeHead, PixelToVoxelShuffleHead
- blocks.py: icnr_init, _get_convnext_stage, UNeXt2UpStage, UNeXt2Decoder
- __init__.py: re-exports all 8 public components
- Zero imports from unet/, vae/, or contrastive/
- All attribute names preserved for state dict compatibility
* feat(06-03): migrate ConvBlock2D and ConvBlock3D to unet/_layers/
- Copy ConvBlock2D from v0.3.3 source to snake_case file
- Copy ConvBlock3D from v0.3.3 source to snake_case file
- Preserve register_modules/add_module pattern for state dict key compatibility
- Update _layers/__init__.py with public re-exports
- Fix docstring formatting for ruff D-series compliance
* test(06-03): add tests for ConvBlock2D and ConvBlock3D
- 6 tests for ConvBlock2D: forward pass, state dict keys, residual, filter steps, instance norm
- 4 tests for ConvBlock3D: forward pass, state dict keys, dropout registration, layer order
- All 10 tests verify shape, naming patterns, and module registration
* test(06-02): add forward-pass tests for all _components
- test_stems.py: UNeXt2Stem shape, StemDepthtoChannels shape + mismatch error
- test_heads.py: PixelToVoxelHead, UnsqueezeHead, PixelToVoxelShuffleHead shapes
- test_blocks.py: icnr_init, _get_convnext_stage, UNeXt2UpStage, UNeXt2Decoder
- 10 tests total, all passing on CPU
* docs(06-03): complete UNet ConvBlock layers plan
- SUMMARY.md with migration details and self-check
- STATE.md updated: phase 6 plan 3/3, decisions recorded
* docs(06-02): complete shared components extraction plan
- SUMMARY.md documents 8 extracted components with 10 tests
- STATE.md updated with decisions from 06-02 execution
* docs(phase-6): complete phase execution and verification
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(07-core-unet-models): research phase domain
* docs(07): create phase plan for core UNet models
* feat(07-01): migrate UNeXt2 model class to viscy-models
- Copy UNeXt2 class (~70 lines) from monolithic unext2.py
- Update imports to use viscy_models._components (stems, heads, blocks)
- Preserve all attribute names for state dict compatibility
- Export UNeXt2 from viscy_models.unet public API
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(07-01): add 6 UNeXt2 forward-pass tests and fix deconv tuple bug
- Add tests: default, small backbone, multichannel, diff stack depths, deconv, stem validation
- Fix deconv decoder tuple bug in UNeXt2UpStage (trailing comma created tuple not module)
- Mark deconv test xfail: original code has channel mismatch in deconv forward path
- All 26 tests pass (25 passed, 1 xfailed) with no regressions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(07-01): complete UNeXt2 migration plan
- Add 07-01-SUMMARY.md with execution results and deviation documentation
- Update STATE.md: phase 7, plan 1/2 complete, new decisions logged
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(07-02): migrate FullyConvolutionalMAE to viscy-models
- Copy FCMAE and all helper classes/functions to unet/fcmae.py
- Replace old viscy imports with viscy_models._components imports
- Remove duplicated PixelToVoxelShuffleHead (import from _components.heads)
- Fix mutable list defaults to tuples (encoder_blocks, dims)
- Export both UNeXt2 and FullyConvolutionalMAE from unet/__init__.py
* test(07-02): migrate 11 FCMAE tests to viscy-models
- Copy all 11 test functions with zero logic changes
- Update imports from viscy.unet.networks.fcmae to viscy_models.unet.fcmae
- Import PixelToVoxelShuffleHead from viscy_models._components.heads
- All 37 tests pass across full suite (no regressions)
* docs(07-02): complete FCMAE migration plan (Phase 7 complete)
- Add 07-02-SUMMARY.md with execution results
- Update STATE.md: Phase 7 complete, 12 plans total, decisions logged
* docs(phase-7): complete phase execution and verification
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(phase-8): research representation models migration
* docs(08): create phase plan for representation models
* feat(08-02): migrate BetaVae25D and BetaVaeMonai to viscy-models
- Add BetaVae25D with VaeUpStage, VaeEncoder, VaeDecoder helpers
- Add BetaVaeMonai wrapping MONAI VarAutoEncoder
- Fix VaeDecoder mutable list defaults to tuples (COMPAT-02)
- Change VaeEncoder pretrained default to False
- Preserve all attribute names for state dict compatibility
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(08-01): migrate ContrastiveEncoder and ResNet3dEncoder to viscy-models
- Add ContrastiveEncoder with convnext/resnet50 backbone support via timm
- Add ResNet3dEncoder with MONAI ResNetFeatures backend
- Fix ResNet50 bug: use encoder.num_features instead of encoder.head.fc.in_features
- Add pretrained parameter (default False) for pure nn.Module semantics
- Preserve state dict attribute names (stem, encoder, projection)
- Share projection_mlp utility between both encoder classes
* test(08-01): add 5 forward-pass tests for contrastive models
- 3 tests for ContrastiveEncoder: convnext_tiny, resnet50, custom stem
- 2 tests for ResNet3dEncoder: resnet18, resnet10
- Verify embedding and projection output shapes
- ResNet50 test uses in_stack_depth=10 for valid stem channel alignment
* test(08-02): add forward-pass tests for BetaVae25D and BetaVaeMonai
- 2 BetaVae25D tests: resnet50 and convnext_tiny backbones
- 2 BetaVaeMonai tests: 2D and 3D spatial configurations
- Verify SimpleNamespace output with recon_x, mean, logvar, z
- Fix ResNet50 expected spatial dims (64x64 not 128x128)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(08-01): complete contrastive model migration plan
- Add 08-01-SUMMARY.md with execution results
- Update STATE.md to Phase 8, plan 1/2
* docs(08-02): complete VAE migration plan (Phase 8 complete)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(phase-8): complete phase execution and verification
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(09): research legacy UNet models migration
* docs(09): create phase plan for legacy UNet models
* feat(09-01): migrate Unet2d and Unet25d to viscy-models
- Copy Unet2d from v0.3.3 with import path update to viscy_models.unet._layers
- Copy Unet25d from v0.3.3 with import path update to viscy_models.unet._layers
- Fix mutable default num_filters=[] to num_filters=() in both models
- Add module docstrings and __all__ exports
- Update unet/__init__.py to export all 4 models (UNeXt2, FCMAE, Unet2d, Unet25d)
- Preserve register_modules/add_module pattern for state dict compatibility
- All 45 existing tests still pass
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(09-01): add pytest tests for Unet2d and Unet25d
- 12 tests for Unet2d: default forward, variable depth, multichannel, residual,
task mode, dropout, state dict keys, custom num_filters
- 11 tests for Unet25d: default Z-compression, preserved depth, variable depth,
multichannel, residual, task mode, state dict keys with skip_conv_layer, custom filters
- Fix list(num_filters) conversion in both models for tuple default compatibility
- Total test suite: 68 passed, 1 xfailed, 0 failures
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(09-01): complete legacy UNet migration plan
- SUMMARY.md with task commits, deviations, and self-check
- STATE.md updated to Phase 9 complete (15 plans total)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(phase-9): complete phase execution and verification
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(10): create phase plan for public API and CI integration
* feat(10-01): add top-level re-exports for all 8 model classes
- Import UNeXt2, FullyConvolutionalMAE, Unet2d, Unet25d from unet subpackage
- Import ContrastiveEncoder, ResNet3dEncoder from contrastive subpackage
- Import BetaVae25D, BetaVaeMonai from vae subpackage
- Update __all__ with all 8 classes in alphabetical order
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(10-01): add state dict key compatibility regression tests
- 24 tests covering all 8 migrated model architectures
- Each model tested for parameter count, top-level prefixes, and sentinel keys
- Guards COMPAT-01: state dict keys must match for checkpoint loading
- Tests import from top-level viscy_models package (validates public API)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(10-01): add viscy-models to CI test matrix
- Add package dimension to test matrix (viscy-transforms, viscy-models)
- Use cross-platform --cov=src/ instead of named package coverage
- Matrix now produces 18 jobs (3 OS x 3 Python x 2 packages)
- check job automatically aggregates all test results via alls-green
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(10-01): complete public API & CI integration plan (v1.1 milestone complete)
- Add 10-01-SUMMARY.md with execution results
- Update STATE.md: phase 10 complete, v1.1 milestone done, 100% progress
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(phase-10): complete phase execution and verification (v1.1 milestone complete)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: consolidate ConvBlock2D/3D into _components
Move conv_block_2d.py and conv_block_3d.py from unet/_layers/ to
_components/ alongside all other shared building blocks. All reusable
layers now live in one place. unet/_layers/ retained as backward-
compatible re-export shim.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* remove the _layers
* update the readme
* update the main readme
* fix description in toml
* changing ruff formatting , dosctrings and imports
* renaming folder to components
* updatet planning docs
* update to components
* numpy docstring
* add claude.md and contributing.md
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* modularizing data package (#366)
* add planning roadmap
* docs: start milestone v1.1 Extract viscy-data
* docs: complete viscy-data project research
* docs: define milestone v1.1 requirements
* docs: create milestone v1.1 roadmap (4 phases)
* docs(06-package-scaffolding-and-foundation): create phase plan
* feat(06-01): create viscy-data package directory structure with pyproject.toml
- Add pyproject.toml with hatchling build, uv-dynamic-versioning, all base deps
- Declare optional dependency groups: triplet, livecell, mmap, all
- Add PEP 561 py.typed marker and tests/__init__.py
- Configure pattern-prefix for independent versioning
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(06-01): add type definitions and package init with re-exports
- Copy all type definitions from viscy/data/typing.py into _typing.py
- Add INDEX_COLUMNS from viscy/data/triplet.py for shared access
- Update typing_extensions.NotRequired to typing.NotRequired (Python >=3.11)
- Create __init__.py with full re-export of all public types
- Add README.md required by hatchling build
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(06-01): integrate viscy-data as workspace dependency in root pyproject.toml
- Add viscy-data to root dependencies list
- Register viscy-data as workspace source in [tool.uv.sources]
- Verified editable install and full import chain works
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(06-01): complete package scaffolding plan with summary and state update
- Add 06-01-SUMMARY.md documenting viscy-data package creation
- Update STATE.md with plan position, metrics, and decisions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(06-02): extract shared utility functions into _utils.py
- Extract _ensure_channel_list, _search_int_in_str, _collate_samples, _read_norm_meta from hcs.py
- Extract _scatter_channels, _gather_channels, _transform_channel_wise from triplet.py
- Update imports to use viscy_data._typing instead of viscy.data.typing
- Add __all__ listing all 7 utility functions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(06-02): complete utility module extraction plan
- Add 06-02-SUMMARY.md documenting utility extraction
- Update STATE.md: Phase 6 complete, progress 80%
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(phase-6): complete phase execution
* docs(07-code-migration): create phase plan
* feat(07-01): migrate select.py, distributed.py, segmentation.py to viscy-data
- Copy select.py with well/FOV filtering utilities (no internal viscy imports)
- Copy distributed.py with ShardedDistributedSampler (no internal viscy imports)
- Copy segmentation.py with viscy.data.typing -> viscy_data._typing import update
- Add missing docstrings to satisfy ruff D rules
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(07-01): migrate hcs.py to viscy-data with utility import rewiring
- Copy HCSDataModule, SlidingWindowDataset, MaskTestDataset from main
- Replace viscy.data.typing imports with viscy_data._typing
- Remove 4 utility function definitions (now in _utils.py)
- Add import from viscy_data._utils for shared utilities
- Remove unused re and collate_meta_tensor imports
- Add missing docstrings for ruff D compliance
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(07-01): migrate gpu_aug.py to viscy-data with dependency rewiring
- Copy GPUTransformDataModule, CachedOmeZarrDataset, CachedOmeZarrDataModule
- Rewire viscy.data.distributed -> viscy_data.distributed
- Rewire viscy.data.hcs utility imports -> viscy_data._utils
- Rewire viscy.data.select -> viscy_data.select
- Rewire viscy.data.typing -> viscy_data._typing
- Add missing docstrings for ruff D compliance
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(07-01): complete core data module migration plan
- Add 07-01-SUMMARY.md documenting migration of 5 core modules
- Update STATE.md: phase 7 plan 1 of 4, decisions, metrics
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(07-03): migrate mmap_cache.py and ctmc_v1.py to viscy-data
- Rewire all imports from viscy.data to viscy_data prefix
- Add lazy import for tensordict with clear error message
- Add docstrings for ruff D compliance
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(07-03): migrate livecell.py with lazy optional dependency imports
- Rewire imports from viscy.data to viscy_data prefix
- Add lazy imports for pycocotools, tifffile, torchvision
- Add import guards in LiveCellDataset and LiveCellTestDataset __init__
- Add docstrings for ruff D compliance
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(07-03): migrate combined.py as-is with import rewiring
- Rewire viscy.data.distributed to viscy_data.distributed
- Rewire viscy.data.hcs._collate_samples to viscy_data._utils._collate_samples
- Preserve all 6 public classes without structural changes
- Add docstrings for ruff D compliance
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(07-02): migrate cell_classification.py and cell_division_triplet.py
- Rewire imports from viscy.data to viscy_data prefix
- Add lazy import for pandas in cell_classification.py with clear error message
- Import _transform_channel_wise from viscy_data._utils (not triplet.py)
- Import INDEX_COLUMNS and AnnotationColumns from viscy_data._typing
- Add docstrings for ruff D compliance
* docs(07-03): complete optional dependency module migration plan
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(07-02): complete specialized module migration plan
- Add 07-02-SUMMARY.md documenting triplet, classification, and cell division module migration
- Update STATE.md with position, decisions, and metrics
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(07-04): add complete public API exports to viscy_data __init__.py
- Export all 45 public names (17 types, 2 utilities, 26 DataModules/Datasets/enums)
- Eager imports from all 13 modules (lazy guards handled internally by each module)
- Comprehensive __all__ list for IDE autocompletion and star-import support
- Ruff-sorted import ordering passes all lint checks
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(07-04): complete public API exports plan - phase 7 fully done
- 07-04-SUMMARY.md documenting 45 public exports and full package verification
- STATE.md updated: phase 7 complete (4/4 plans), 12 total plans done
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(phase-7): complete code migration execution
* docs(08-test-migration-and-validation): create phase plan
* test(08-01): add conftest.py with HCS OME-Zarr fixtures for viscy-data
- Copy all 6 fixtures and _build_hcs helper from main branch conftest
- Replace legacy np.random.rand with np.random.default_rng (NPY002)
- No viscy import changes needed (only uses third-party libs)
- Provides preprocessed_hcs_dataset, small_hcs_dataset, tracks fixtures
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(08-02): complete smoke tests plan - phase 8 test migration done
- Created 08-02-SUMMARY.md documenting 52 smoke tests for viscy_data
- Updated STATE.md: phase 8 complete, 14 total plans executed
- DATA-TST-02 satisfied: import, __all__, optional dep messages, no legacy namespace
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(08-01): migrate test_hcs, test_triplet, test_select to viscy-data package
- Update imports from viscy.data.X to viscy_data
- Add BatchedCenterSpatialCropd to _utils.py (fixes batch dim handling)
- Fix triplet.py to use BatchedCenterSpatialCropd instead of CenterSpatialCropd
- Add tensorstore to test dependency group for triplet tests
- All 19 tests pass across 3 test files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(08-01): complete data test migration plan summary
- Create 08-01-SUMMARY.md documenting test migration and bug fixes
- Update STATE.md with BatchedCenterSpatialCropd decision revision
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(phase-8): complete test migration and validation
* docs(09-ci-integration): create phase plan
* feat(09-01): add viscy-data CI test jobs to GitHub Actions workflow
- Add test-data job with 3x3 matrix (3 OS x 3 Python) for viscy-data
- Add test-data-extras job (ubuntu-latest, Python 3.13) for extras validation
- Update check job needs to aggregate all test jobs: test, test-data, test-data-extras
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(09-01): complete CI integration plan
- Add 09-01-SUMMARY.md documenting viscy-data CI jobs
- Update STATE.md: phase 9 complete, v1.0 milestone complete
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(phase-9): complete CI integration - milestone v1.1 done
* chore: complete v1.1 milestone — Extract viscy-data
Delivered: viscy-data package with 15 modules, 45 public exports,
optional dependency groups, 71 tests, and tiered CI.
Archives:
- milestones/v1.1-ROADMAP.md
- milestones/v1.1-REQUIREMENTS.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: add missing pandas guards, restore conftest fixture, remove no-op CI filter
- Add `if pd is None` guard in ClassificationDataModule.setup() and
TripletDataModule._align_tracks_tables_with_positions() to raise
helpful ImportError instead of AttributeError when pandas is absent
- Fix ClassificationDataset error message to suggest `pip install pandas`
instead of `pip install 'viscy-data[triplet]'` (classification doesn't
need tensorstore)
- Restore `num_timepoints` parameter on `_build_hcs()` and add
`temporal_hcs_dataset` fixture from upstream commit 44b25b9
- Remove no-op `-m "not slow"` from test-data-extras CI job (no tests
use @pytest.mark.slow)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: add CLAUDE.md and update CONTRIBUTING.md
Add CLAUDE.md with project-specific instructions for Claude Code sessions.
Update CONTRIBUTING.md with ruff config centralization warning and numpy
docstring convention note. Synced from 71009b5.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: update uv.lock after rebase onto modular-viscy-staging
Regenerate lockfile to include viscy-data workspace dependencies
alongside viscy-models from the updated base branch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* port changes from tests-zarrv3 branch
* ruff
* add additional test for the main dataloaders
* redundant tets 3. I think test 2 alreaady takes care of this.
* rename INDEX_COLUMNS
* remove unused LABEL classes
* fix(livecell): assign transform result and avoid mutable defaults
LiveCellTestDataset.__getitem__ discarded the return value of
self.transform(sample), so MONAI transforms had no effect.
Also replace mutable default lists in LiveCellDataModule.__init__
with None to prevent cross-instance state sharing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cell_classification): raise ValueError, tighten val_fovs type, fix mutable default
- `raise (f"Unknown stage: {stage}")` raised a string instead of an
exception — use `ValueError`.
- `val_fovs: list[str] | None` was unconditionally indexed in
`setup()` — remove the `None` option since it's always required.
- `_subset(..., exclude_timepoints=[])` used a mutable default —
replace with `None`.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(gpu_aug): remove _filter_fit_fovs override so exclude_fovs is applied
CachedOmeZarrDataModule accepted exclude_fovs but its local
_filter_fit_fovs override only filtered wells, silently ignoring
excluded FOVs. Remove the override so the SelectWell mixin's
implementation (which filters both wells and FOVs) is used.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: rename select.py to _select.py (private module)
The module contains mostly private helpers (_filter_wells,
_filter_fovs) and a mixin dataclass (SelectWell). Renaming to
_select.py signals it is internal implementation detail.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: update package descriptions to "AI x Imaging" and CLAUDE.md
Update viscy-data description from "virtual staining microscopy"
to "AI x Imaging tasks" in README, pyproject.toml, and __init__.py.
Add viscy-models test example to CLAUDE.md.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ctmc_v1): add missing prefetch_factor attribute
CTMCv1DataModule.__init__ did not set self.prefetch_factor, causing
an AttributeError when train_dataloader() or val_dataloader() was
called (inherited from GPUTransformDataModule). Set it to None.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: add functional tests for ctmc, livecell, segmentation, classification
Add unit tests for the four data modules that previously only had
import smoke tests:
- test_ctmc_v1.py: setup, val subsample ratio, batch shape
- test_livecell.py: dataset/datamodule with mock TIFF + COCO data
- test_segmentation.py: paired pred/target datasets, z-slice
- test_cell_classification.py: annotation CSV, FOV split, timepoint exclusion
Also adds shared fixtures to conftest.py (single_channel_hcs_pair,
segmentation_hcs_pair, classification_hcs_dataset).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(livecell): add missing prefetch_factor; add dataloader iteration tests
Add prefetch_factor attribute to LiveCellDataModule (same fix as 97455eb
for CTMC). Add batch iteration + shape validation to classification and
livecell datamodule tests to match coverage patterns in test_hcs/test_triplet.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: delete leftover select.py after rename to _select.py
Commit 5b132f9 renamed select.py to _select.py but did not remove
the original file. Nothing imports from it.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use explicit positions[0] instead of leaked loop variable
CachedOmeZarrDataset and MmappedDataset both built self.channels
using the loop variable `position` after iterating, implicitly
depending on the last element. Use positions[0] to be explicit
and avoid UnboundLocalError on empty input.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(livecell): handle empty annotations and correct return type
Guard torch.stack against empty annotation lists in
LiveCellTestDataset.__getitem__ when load_labels=True, returning
properly shaped empty tensors instead of crashing.
Fix _parse_image_names return type annotation: list[Path] -> list[str].
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(segmentation): defer open_ome_zarr from __init__ to setup()
Store only paths in __init__ and open OME-Zarr stores in setup("test"),
consistent with other DataModules in the package and Lightning
conventions for resource lifecycle.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* print to logger
* refactor(triplet): remove BatchedCenterSpatialCropd from viscy-data
The transform already exists in viscy-transforms and viscy-data should
not depend on it. Replace the final crop with a shape validation check
in on_after_batch_transfer() and require initial_yx_patch_size to match
the desired output size.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(docs): convert Sphinx-style docstrings to numpy style in _utils.py
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(init): lazy-load submodules via PEP 562 __getattr__
Replace eager imports of all DataModule/Dataset submodules with
on-demand loading. Modules with optional dependencies (triplet,
livecell, mmap_cache) are no longer imported at `import viscy_data`
time. Add __init__.pyi stub for type-checker/IDE support.
Also split CI test-data job to run without --all-extras so the base
package is validated independently from optional dependencies.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: guard None dereferences and replace mutable default arguments
- cell_classification.py: guard _read_norm_meta() returning None
- hcs.py: guard MaskTestDataset with ground_truth_masks=None, add
missing array_key parameter, replace mutable default [] with None
- triplet.py, cell_division_triplet.py: replace mutable default []
with None for normalizations/augmentations parameters
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: correct return type annotations in _utils.py
_gather_channels and _transform_channel_wise return Tensor, not
list[Tensor].
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(combined): check stage before calling dm.setup()
Move the unsupported-stage guard to the top of setup() in
ConcatDataModule and CachedConcatDataModule so constituent data
modules are not set up for stages that will be rejected.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: add tests for cell_division_triplet, mmap_cache, and HCS test stage
- test_cell_division_triplet.py: 11 smoke tests for dataset and datamodule
- test_mmap_cache.py: 5 smoke tests (skipped when tensordict missing)
- test_hcs.py: add setup("test") coverage for MaskTestDataset
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Eduardo Hirata-Miyasaki <edhiratam@gmail.com>
Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Applications: Cytoland refactor (#379)
* test(10-01): add state dict key compatibility regression tests
- 24 tests covering all 8 migrated model architectures
- Each model tested for parameter count, top-level prefixes, and sentinel keys
- Guards COMPAT-01: state dict keys must match for checkpoint loading
- Tests import from top-level viscy_models package (validates public API)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(10-01): add viscy-models to CI test matrix
- Add package dimension to test matrix (viscy-transforms, viscy-models)
- Use cross-platform --cov=src/ instead of named package coverage
- Matrix now produces 18 jobs (3 OS x 3 Python x 2 packages)
- check job automatically aggregates all test results via alls-green
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(10-01): complete public API & CI integration plan (v1.1 milestone complete)
- Add 10-01-SUMMARY.md with execution results
- Update STATE.md: phase 10 complete, v1.1 milestone done, 100% progress
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(phase-10): complete phase execution and verification (v1.1 milestone complete)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: consolidate ConvBlock2D/3D into _components
Move conv_block_2d.py and conv_block_3d.py from unet/_layers/ to
_components/ alongside all other shared building blocks. All reusable
layers now live in one place. unet/_layers/ retained as backward-
compatible re-export shim.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* remove the _layers
* update the readme
* docs: start milestone v1.1 Extract viscy-data
* update the main readme
* docs: complete viscy-data project research
* docs: define milestone v1.1 requirements
* docs: create milestone v1.1 roadmap (4 phases)
* docs(06-package-scaffolding-and-foundation): create phase plan
* feat(06-01): create viscy-data package directory structure with pyproject.toml
- Add pyproject.toml with hatchling build, uv-dynamic-versioning, all base deps
- Declare optional dependency groups: triplet, livecell, mmap, all
- Add PEP 561 py.typed marker and tests/__init__.py
- Configure pattern-prefix for independent versioning
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(06-01): add type definitions and package init with re-exports
- Copy all type definitions from viscy/data/typing.py into _typing.py
- Add INDEX_COLUMNS from viscy/data/triplet.py for shared access
- Update typing_extensions.NotRequired to typing.NotRequired (Python >=3.11)
- Create __init__.py with full re-export of all public types
- Add README.md required by hatchling build
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(06-01): integrate viscy-data as workspace dependency in root pyproject.toml
- Add viscy-data to root dependencies list
- Register viscy-data as workspace source in [tool.uv.sources]
- Verified editable install and full import chain works
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(06-01): complete package scaffolding plan with summary and state update
- Add 06-01-SUMMARY.md documenting viscy-data package creation
- Update STATE.md with plan position, metrics, and decisions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(06-02): extract shared utility functions into _utils.py
- Extract _ensure_channel_list, _search_int_in_str, _collate_samples, _read_norm_meta from hcs.py
- Extract _scatter_channels, _gather_channels, _transform_channel_wise from triplet.py
- Update imports to use viscy_data._typing instead of viscy.data.typing
- Add __all__ listing all 7 utility functions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(06-02): complete utility module extraction plan
- Add 06-02-SUMMARY.md documenting utility extraction
- Update STATE.md: Phase 6 complete, progress 80%
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(phase-6): complete phase execution
* docs(07-code-migration): create phase plan
* feat(07-01): migrate select.py, distributed.py, segmentation.py to viscy-data
- Copy select.py with well/FOV filtering utilities (no internal viscy imports)
- Copy distributed.py with ShardedDistributedSampler (no internal viscy imports)
- Copy segmentation.py with viscy.data.typing -> viscy_data._typing import update
- Add missing docstrings to satisfy ruff D rules
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(07-01): migrate hcs.py to viscy-data with utility import rewiring
- Copy HCSDataModule, SlidingWindowDataset, MaskTestDataset from main
- Replace viscy.data.typing imports with viscy_data._typing
- Remove 4 utility function definitions (now in _utils.py)
- Add import from viscy_data._utils for shared utilities
- Remove unused re and collate_meta_tensor imports
- Add missing docstrings for ruff D compliance
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(07-01): migrate gpu_aug.py to viscy-data with dependency rewiring
- Copy GPUTransformDataModule, CachedOmeZarrDataset, CachedOmeZarrDataModule
- Rewire viscy.data.distributed -> viscy_data.distributed
- Rewire viscy.data.hcs utility imports -> viscy_data._utils
- Rewire viscy.data.select -> viscy_data.select
- Rewire viscy.data.typing -> viscy_data._typing
- Add missing docstrings for ruff D compliance
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(07-01): complete core data module migration plan
- Add 07-01-SUMMARY.md documenting migration of 5 core modules
- Update STATE.md: phase 7 plan 1 of 4, decisions, metrics
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(07-03): migrate mmap_cache.py and ctmc_v1.py to viscy-data
- Rewire all imports from viscy.data to viscy_data prefix
- Add lazy import for tensordict with clear error message
- Add docstrings for ruff D compliance
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(07-03): migrate livecell.py with lazy optional dependency imports
- Rewire imports from viscy.data to viscy_data prefix
- Add lazy imports for pycocotools, tifffile, torchvision
- Add import guards in LiveCellDataset and LiveCellTestDataset __init__
- Add docstrings for ruff D compliance
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(07-03): migrate combined.py as-is with import rewiring
- Rewire viscy.data.distributed to viscy_data.distributed
- Rewire viscy.data.hcs._collate_samples to viscy_data._utils._collate_samples
- Preserve all 6 public classes without structural changes
- Add docstrings for ruff D compliance
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(07-02): migrate cell_classification.py and cell_division_triplet.py
- Rewire imports from viscy.data to viscy_data prefix
- Add lazy import for pandas in cell_classification.py with clear error message
- Import _transform_channel_wise from viscy_data._utils (not triplet.py)
- Import INDEX_COLUMNS and AnnotationColumns from viscy_data._typing
- Add docstrings for ruff D compliance
* docs(07-03): complete optional dependency module migration plan
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(07-02): complete specialized module migration plan
- Add 07-02-SUMMARY.md documenting triplet, classification, and cell division module migration
- Update STATE.md with position, decisions, and metrics
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(07-04): add complete public API exports to viscy_data __init__.py
- Export all 45 public names (17 types, 2 utilities, 26 DataModules/Datasets/enums)
- Eager imports from all 13 modules (lazy guards handled internally by each module)
- Comprehensive __all__ list for IDE autocompletion and star-import support
- Ruff-sorted import ordering passes all lint checks
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(07-04): complete public API exports plan - phase 7 fully done
- 07-04-SUMMARY.md documenting 45 public exports and full package verification
- STATE.md updated: phase 7 complete (4/4 plans), 12 total plans done
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(phase-7): complete code migration execution
* docs(08-test-migration-and-validation): create phase plan
* test(08-01): add conftest.py with HCS OME-Zarr fixtures for viscy-data
- Copy all 6 fixtures and _build_hcs helper from main branch conftest
- Replace legacy np.random.rand with np.random.default_rng (NPY002)
- No viscy import changes needed (only uses third-party libs)
- Provides preprocessed_hcs_dataset, small_hcs_dataset, tracks fixtures
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(08-02): complete smoke tests plan - phase 8 test migration done
- Created 08-02-SUMMARY.md documenting 52 smoke tests for viscy_data
- Updated STATE.md: phase 8 complete, 14 total plans executed
- DATA-TST-02 satisfied: import, __all__, optional dep messages, no legacy namespace
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(08-01): migrate test_hcs, test_triplet, test_select to viscy-data package
- Update imports from viscy.data.X to viscy_data
- Add BatchedCenterSpatialCropd to _utils.py (fixes batch dim handling)
- Fix triplet.py to use BatchedCenterSpatialCropd instead of CenterSpatialCropd
- Add tensorstore to test dependency group for triplet tests
- All 19 tests pass across 3 test files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(08-01): complete data test migration plan summary
- Create 08-01-SUMMARY.md documenting test migration and bug fixes
- Update STATE.md with BatchedCenterSpatialCropd decision revision
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(phase-8): complete test migration and validation
* docs(09-ci-integration): create phase plan
* feat(09-01): add viscy-data CI test jobs to GitHub Actions workflow
- Add test-data job with 3x3 matrix (3 OS x 3 Python) for viscy-data
- Add test-data-extras job (ubuntu-latest, Python 3.13) for extras validation
- Update check job needs to aggregate all test jobs: test, test-data, test-data-extras
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(09-01): complete CI integration plan
- Add 09-01-SUMMARY.md documenting viscy-data CI jobs
- Update STATE.md: phase 9 complete, v1.0 milestone complete
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(phase-9): complete CI integration - milestone v1.1 done
* chore: complete v1.1 milestone — Extract viscy-data
Delivered: viscy-data package with 15 modules, 45 public exports,
optional dependency groups, 71 tests, and tiered CI.
Archives:
- milestones/v1.1-ROADMAP.md
- milestones/v1.1-REQUIREMENTS.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* harmonize the planning between the modular-data and modular-models
* viscy-utils package
* add applications/dynaclr
* update the monorepo uv
* moving files around
* update planning
* docs: start milestone v2.1 DynaCLR Integration Validation
* docs: define milestone v2.1 requirements
* docs: create milestone v2.1 roadmap (2 phases)
* docs(18-training-validation): create phase plan
* feat(18-01): add training integration tests for ContrastiveModule
- Add fast_dev_run tests for TripletMarginLoss and NTXentLoss code paths
- Add parametrized config class_path resolution tests for fit.yml and predict.yml
- Add tensorboard as test dependency for TensorBoardLogger in integration tests
- Fix workspace exclude to skip non-package application directories
- Use 2D-compatible synthetic data shapes (1,1,4,4) for render_images compatibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(18-01): complete training integration tests plan
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(phase-18): complete phase execution
* docs(19-inference-reproducibility): create phase plan
* chore(19-01): add anndata test dependency and HPC conftest fixtures
- Add anndata to dynacrl test dependency group
- Create conftest.py with HPC path constants, skip markers, and fixtures
- Update uv.lock
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(19-01): add inference reproducibility integration tests
- Create test_inference_reproducibility.py with 2 HPC integration tests
- test_checkpoint_loads_into_modular_contrastive_module (INFER-01)
- test_predict_embeddings_and_exact_match (INFER-02 + INFER-03)
- Fix lazy imports in EmbeddingWriter to avoid unconditional umap import
- Fix anndata nullable string compatibility in write_embedding_dataset
- Tests skip gracefully when HPC paths or GPU unavailable
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(19-01): complete inference reproducibility plan
- Add 19-01-SUMMARY.md with execution results and deviation documentation
- Update STATE.md: Phase 19 complete, v2.1 milestone finished
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: add seed_everything(42) to all integration tests
Ensures reproducibility by seeding all tests consistently.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(phase-19): complete phase execution
* restructure the examples folder and ruff
* update readme.me hallucination
* update the readmes
* - Add `viscy` console script in viscy-utils pointing to
viscy_utils.cli:main
- Add jsonargparse[signatures] dependency for LightningCLI
- Add 4 CLI smoke tests (help, subcommands, fit --help, predict
--help)
- Replace conda/anaconda with uv in SLURM scripts
- Update SLURM scripts to use `viscy fit/predict` instead of old
monolith
* add the CLI for running training and prediction
* default embedding writer to None
* import within the function
* ruff
* dynaclr typo
* rename folder to dynaclr
* add the classifiers here
* docs: start milestone v2.2 Composable Sampling Framework
* docs: define milestone v2.2 requirements
* docs: create milestone v2.2 roadmap (6 phases)
* docs(20): capture phase context
* docs(20): create phase plan for experiment configuration
* test(20-01): add failing tests for ExperimentConfig and ExperimentRegistry
- 19 test cases covering config creation, defaults, channel maps,
validation errors, YAML loading, tau-range conversion, and lookups
- All tests fail with ModuleNotFoundError (module not yet implemented)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(20-01): implement ExperimentConfig and ExperimentRegistry
- ExperimentConfig dataclass with all fields and defaults
- ExperimentRegistry with fail-fast validation at __post_init__:
empty check, duplicate names, source_channel membership,
channel count consistency, interval_minutes positivity,
condition_wells non-empty, data_path existence, zarr channel match
- channel_maps: per-experiment source position -> zarr index mapping
- from_yaml classmethod for YAML config loading
- tau_range_frames for hours-to-frames conversion with warning
- get_experiment lookup by name with KeyError
- All 19 tests pass
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(20-01): clean up imports and exclude stale dynacrl workspace member
- Fix ruff I001 (import sorting) and F401 (unused import) in test file
- Exclude applications/dynacrl (typo) from uv workspace to unblock builds
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(20-01): complete ExperimentConfig/ExperimentRegistry plan
- SUMMARY.md with TDD execution results, self-check passed
- STATE.md updated with position, decisions, session continuity
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(20-02): add explicit deps and top-level experiment API exports
- Add iohub>=0.3a2 and pyyaml as explicit dependencies in dynaclr pyproject.toml
- Re-export ExperimentConfig and ExperimentRegistry from dynaclr __init__.py
- Both classes now importable via `from dynaclr import ExperimentConfig, ExperimentRegistry`
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(20-02): add example multi-experiment YAML configuration
- Demonstrate positional channel alignment across 2 experiments
- SEC61 (30min interval, ER) and TOMM20 (15min interval, mito)
- Show condition_wells with infected/uninfected/mock conditions
- Include comments explaining channel alignment and tau_range conversion
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(20-02): complete package wiring and example config plan
- SUMMARY.md with execution results and self-check
- STATE.md updated: Phase 20 complete, 20/25 phases (80%)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(phase-20): complete phase execution
Phase 20 Experiment Configuration verified (11/11 must-haves).
ExperimentConfig + ExperimentRegistry with TDD, package wiring, example YAML.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(21): create phase plan for Cell Index & Lineage
* test(21-01): add failing tests for MultiExperimentIndex
- 17 test cases covering CELL-01 (unified tracks), CELL-02 (lineage), CELL-03 (border clamping)
- All fail with ModuleNotFoundError (dynaclr.index not yet implemented)
- Test fixtures create mini OME-Zarr stores with tracking CSVs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(21-01): implement MultiExperimentIndex with lineage and border clamping
- Unified tracks DataFrame from all experiments with enriched columns
- Lineage reconstruction linking daughters to root ancestor via parent_track_id
- Border clamping: retains border cells with shifted patch origins instead of exclusion
- All 23 tests pass
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(21-01): fix lint issues and export MultiExperimentIndex
- Remove unused variable (F841) in test_global_track_id_unique_across_experiments
- Use .to_numpy() instead of .values (PD011) in test_exclude_fovs_filter
- Export MultiExperimentIndex from dynaclr __init__.py
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(21-01): complete MultiExperimentIndex plan summary and state update
- 21-01-SUMMARY.md with full execution documentation
- STATE.md updated for 21-01 completion, decisions, session continuity
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test(21-02): add failing tests for valid anchors, properties, and summary
- 8 tests for valid_anchors: basic validity, subset check, end-of-track exclusion,
lineage continuity, different tau ranges, empty tracks, gap handling, self-exclusion
- 9 tests for properties/summary: experiment_groups, condition_groups, summary()
- All 17 new tests fail with TypeError (tau_range_hours not yet accepted)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(21-02): implement valid_anchors, experiment_groups, condition_groups, summary
- Add tau_range_hours parameter to MultiExperimentIndex.__init__
- _compute_valid_anchors: per-experiment tau conversion, lineage-based lookup
- experiment_groups/condition_groups properties returning index arrays
- summary() with experiment counts, observation counts, per-experiment breakdowns
- All 40 tests pass (23 existing + 17 new)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(21-02): complete valid anchors plan
- SUMMARY.md with self-check passed
- STATE.md updated: Phase 21 complete, ready for Phase 22
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(22): research batch sampling phase domain
* docs(22): create phase plan for batch sampling
* test(22-01): add failing tests for FlexibleBatchSampler
- Experiment-aware batching: single-experiment restriction, all experiments appear
- Condition balancing: 2-condition and 3-condition proportional tests
- Leaky mixing: zero leak, 20% leak injection, no-effect when not experiment-aware
- Small group fallback: no crash, warning emission
- Determinism: same seed/epoch reproduces, set_epoch changes sequence
- Sampler protocol: yields list[int], correct __len__
- DDP partitioning: disjoint interleaved batches across ranks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(22-01): implement FlexibleBatchSampler with experiment-aware, condition-balanced, leaky mixing
- FlexibleBatchSampler(Sampler[list[int]]) with cascade batch construction
- experiment_aware=True restricts each batch to a single experiment
- condition_balanced=True balances condition representation per batch
- leaky > 0.0 injects cross-experiment samples into restricted batches
- Deterministic via np.random.default_rng(seed + epoch)
- DDP support via interleaved batch partitioning across ranks
- Small group fallback to replacement sampling with logged warning
- Pre-computed group indices at __init__ for O(1) lookup
- Fix lint issues in test file (import sorting, .values -> .to_numpy(), nunique -> len(unique))
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(22-01): export FlexibleBatchSampler from viscy_data package
- Add FlexibleBatchSampler to viscy_data.__init__.py public API
- Place import in alphabetically correct position for ruff isort compliance
- Add to __all__ exports under Utilities section
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(22-01): complete FlexibleBatchSampler core plan
- Create 22-01-SUMMARY.md with TDD execution results
- Update STATE.md: plan 01/02 complete, decisions, session continuity
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test(22-02): add failing tests for temporal enrichment, DDP coverage, validation
- 6 temporal enrichment tests (focal concentration, global_fraction edge cases, validation)
- 5 DDP disjoint coverage tests (interleaving, coverage, epoch reproducibility)
- 3 validation guard tests (missing experiment/condition/hpi columns)
- 2 package import tests (import, __all__)
- All 9 new feature tests fail as expected (RED)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(22-02): implement temporal enrichment, validation guards, DDP coverage
- Add temporal_enrichment, temporal_window_hours, temporal_global_fraction params
- Implement _enrich_temporal: focal/global sampling from experiment pool
- Add column validation guards for experiment/condition/hpi columns
- Conditional precomputation: only groupby columns when feature enabled
- Fix stale smoke test __all__ count (45 -> 46) from Plan 01
- All 35 sampler tests pass, 107 total viscy-data tests pass
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(22-02): complete temporal enrichment + DDP plan
- 22-02-SUMMARY.md with all metrics, decisions, deviations
- STATE.md advanced to Phase 23, progress 22/25 (88%)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(phase-22): complete batch sampling phase execution
Phase 22 verified: FlexibleBatchSampler with all 5 SAMP requirements.
5/5 must-haves passed. 35 tests, 107 full suite pass. No regressions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(23): create phase plan for Loss & Augmentation
* test(23-01): add failing tests for NTXentHCL
- 12 test cases covering subclass, beta=0 equivalence, hard negatives,
gradients, temperature effect, edge cases, defaults, and CUDA
- All fail with ModuleNotFoundError (dynaclr.loss not yet created)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test(23-02): add failing tests for ChannelDropout and variable tau sampling
- 11 tests for ChannelDropout: zeros, probability bounds, eval mode, per-sample, dtype, input safety, multi-channel, CUDA
- 7 tests for sample_tau: range, exponential decay, uniform, single value, determinism, return type
* feat(23-02): implement ChannelDropout and variable tau sampling
- ChannelDropout nn.Module: per-sample channel zeroing on (B,C,Z,Y,X) tensors
- sample_tau: exponential decay weighted sampling for temporal offsets
* feat(23-01): implement NTXentHCL with hard-negative concentration
- NTXentHCL subclasses NTXentLoss from pytorch_metric_learning
- beta=0.0 delegates to parent for exact numerical equivalence
- beta>0 applies exp(beta*sim) reweighting on negatives in denominator
- Normalized weights preserve loss magnitude across beta values
- All 11 tests pass (1 CUDA test skipped on macOS)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(23-02): add ChannelDropout and sample_tau to package exports
- Export ChannelDropout from viscy_data top-level
- Export sample_tau from dynaclr top-level
- Include NTXentHCL export added by linter
* docs(23-02): complete ChannelDropout and tau sampling plan
- Summary with TDD metrics, decisions, self-check
- STATE.md updated for Phase 23 completion
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(23-01): complete NTXentHCL loss plan
- Created 23-01-SUMMARY.md with TDD execution results
- Updated STATE.md with HCL implementation decisions
- Self-check passed: all artifacts and commits verified
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(phase-23): complete loss & augmentation phase execution
Phase 23 verified: NTXentHCL (3/3 LOSS reqs), ChannelDropout (AUG-01),
sample_tau (AUG-03). AUG-02 wiring deferred to Phase 24 by design.
30 tests pass across loss, channel_dropout, tau_sampling.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(24): create phase plan
* test(24-01): add failing tests for MultiExperimentTripletDataset
- 7 test cases covering __getitems__ return format, norm_meta, lineage-aware
positive sampling, division event traversal, channel remapping, predict mode,
and dataset length
- All tests fail with ModuleNotFoundError (RED phase)
* feat(24-01): implement MultiExperimentTripletDataset with lineage-aware sampling
- __getitems__ returns batch dicts with anchor/positive Tensors (B,C,Z,Y,X)
- Lineage-aware positive sampling via pre-built (experiment, lineage_id) lookup
- Division events traversed naturally via shared lineage_id
- Per-experiment channel remapping using registry.channel_maps
- Tensorstore I/O with SLURM-aware context and per-FOV caching
- Predict mode returns anchor + TrackingIndex dicts
- Exponential decay tau sampling with fallback to full range scan
* refactor(24-01): add MultiExperimentTripletDataset to package exports
- Export from dynaclr.__init__ for public API access
* docs(24-01): complete MultiExperimentTripletDataset plan
- SUMMARY.md with TDD commits, decisions, self-check
- STATE.md updated: position 24-01, decisions, session continuity
* update uv
* test(24-02): add failing tests for MultiExperimentDataModule
- 6 test cases covering hyperparameter exposure, experiment-level split,
FlexibleBatchSampler wiring, val dataloader, transforms, ChannelDropout
- RED phase: all tests fail with ModuleNotFoundError
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(24-02): implement MultiExperimentDataModule with experiment-level split
- MultiExperimentDataModule composes FlexibleBatchSampler + Dataset +
ChannelDropout + ThreadDataLoader with collate_fn=lambda x: x
- Train/val split by whole experiments via val_experiments parameter
- All sampling, augmentation, and loss hyperparameters exposed as __init__ params
- on_after_batch_transfer applies normalizations + augmentations + final crop
+ ChannelDropout with proper norm_meta handling for all-None case
- 6 TDD tests passing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(24-02): add MultiExperimentDataModule to dynaclr package exports
- Import MultiExperimentDataModule from dynaclr.datamodule
- Add to __all__ for top-level importability
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(24-02): complete MultiExperimentDataModule plan
- Summary with TDD commits, decisions, and deviation documentation
- STATE.md updated: Phase 24 complete, 96% progress, ready for Phase 25
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(phase-24): complete dataset & datamodule phase execution
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(25): create phase plan
* docs(phase-25): complete integration phase plan
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(25-01): add end-to-end multi-experiment integration tests
- Create test_multi_experiment_fast_dev_run: 2 experiments with different
channel sets (GFP vs RFP), fast_dev_run with NTXentHCL loss
- Create test_multi_experiment_fast_dev_run_with_all_sampling_axes:
experiment_aware + condition_balanced + temporal_enrichment enabled
- Synthetic data helpers for multi-channel HCS OME-Zarr creation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(25-01): add multi-experiment YAML config and class_path validation test
- Create multi_experiment_fit.yml with MultiExperimentDataModule,
NTXentHCL loss, all sampling axes, generic channel names (ch_0/ch_1)
- Add test_multi_experiment_config_class_paths_resolve validating all
class_path entries in the config resolve to importable Python classes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(25-01): complete integration plan - milestone v2.2 complete
- Add 25-01-SUMMARY.md documenting end-to-end integration validation
- Update STATE.md: phase 25/25 complete, progress 100%, milestone v2.2 done
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(phase-25): complete integration phase execution — v2.2 milestone shipped
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* add the smoothness and dynamic range comparison
* add the applications/qc
* bug qc metrics exposing the device
* add batch predict
* adding cli for reduce dimensionality composable
* add example configs for model comparision and smoothness
* add the biological annotations to the zattrs
* adding airtable logic
* harmonize and remove duplication between airtable and qc. moving most things to airtable
* cleanup readme for airtable
* add callback to store embeddings every n epochs and store metadata to the anndata.uns
* fix the apply-linear classifiers to make sure we use the model and version.
* Exclude untracked applications/dynacell from uv workspace
Local debris directory (hydra outputs, pycache) has no pyproject.toml
and breaks uv lock when matched by applications/* glob.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(26): capture phase context
* docs(state): record phase 26 context session
* docs(26): create phase plans
* feat(26-01): extract HCSPredictionWriter to viscy-utils callbacks
- Create prediction_writer.py with HCSPredictionWriter, _pad_shape, _resize_image, _blend_in
- Use TYPE_CHECKING guard for viscy_data imports (HCSDataModule, Sample)
- Add numpy-style docstrings to all functions and class
- Re-export HCSPredictionWriter from callbacks __init__.py
- Fix pre-existing INDEX_COLUMNS -> ULTRACK_INDEX_COLUMNS in embedding_snapshot.py and embedding_writer.py
- Add missing docstrings to embedding_snapshot.py public methods
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(26-01): extract MixedLoss to viscy-utils losses submodule
- Create losses/ submodule with mixed_loss.py containing MixedLoss class
- Uses ms_ssim_25d from viscy_utils.evaluation.metrics internally
- Convert docstrings to numpy-style
- Re-export MixedLoss from losses __init__.py
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(26-01): create translation application scaffold with workspace registration
- Create applications/translation/ with src layout following dynaclr pattern
- Add pyproject.toml with hatchling build, uv-dynamic-versioning, and workspace deps
- Add README.md required by hatchling readme field
- Add __main__.py delegating to viscy_utils.cli.main for LightningCLI entry point
- Add example YAML configs (fit.yml, predict.yml) with HCSPredictionWriter callback
- Create empty tests/__init__.py
- Register viscy-translation in root pyproject.toml workspace sources
- Update uv.lock with new workspace member
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(26-01): complete shared infra extraction + app scaffold plan
- Create 26-01-SUMMARY.md with execution results
- Update STATE.md with plan progress, decisions, session info
- Update ROADMAP.md marking 26-01 as complete
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(26-02): migrate translation engine and evaluation modules
- Copy engine.py with VSUNet, FcmaeUNet, AugmentedPredictionVSUNet, MaskedMSELoss
- Copy evaluation.py with SegmentationMetrics2D
- Update all imports to new package paths (viscy_data, viscy_models, viscy_utils)
- Remove MixedLoss class from engine.py (now imported from viscy_utils.losses)
- Update __init__.py with top-level re-exports
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(26-02): add translation engine test suite
- Import tests for all public exports (VSUNet, FcmaeUNet, etc.)
- VSUNet init and forward pass smoke tests with synthetic data
- State dict key regression test for checkpoint compatibility
- MixedLoss integration test (from viscy_utils.losses)
- FcmaeUNet init test
- No old import paths grep test
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(26-02): complete engine migration plan
- SUMMARY.md with 2 task commits, 2 auto-fixed deviations
- STATE.md updated: phase 26 complete, 20/25 phases (80%)
- ROADMAP.md updated with plan progress
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(26): complete UAT - 8 passed, 0 issues
* add the pseudotime evals
* re-structure pseudotime folder
* add the linear classifier evals and restructure folder path
* add evaluations to dynaclr package
* cli and linear classifier init
* fix(translation): address engine and evaluation bugs
- Fix operator precedence bug in evaluation.py boolean condition
- Fi…1 parent b5c6ed9 commit 85f0b88
149 files changed
Lines changed: 9424 additions & 2460 deletions
File tree
- .claude/skills/hf-dynacell
- applications/dynacell
- configs/benchmarks
- virtual_staining
- _internal/leaf
- er/fcmae_vscyto3d_pretrained_randinit/randinit
- grouped
- er_a549_trained
- er_ipsc_trained
- er_joint
- membrane_a549_trained_carved
- membrane_a549_trained
- membrane_ipsc_trained_carved
- membrane_ipsc_trained
- membrane_joint_carved
- membrane_joint
- mitochondria_a549_trained
- mitochondria_ipsc_trained
- mitochondria_joint
- nucleus_a549_trained
- nucleus_ipsc_trained
- nucleus_joint
- instance_ap
- membrane_a549
- membrane_ipsc
- nucleus_a549
- nucleus_ipsc
- membrane
- fcmae_vscyto3d_pretrained_cytoland/cytoland
- fcmae_vscyto3d_pretrained_infectionft/infectionft
- fcmae_vscyto3d_pretrained_randinit/randinit
- vscyto3d_cytolandft
- a549_mantis
- ipsc_confocal
- vscyto3d_infectionft_dynacellft
- a549_mantis
- ipsc_confocal
- mito/fcmae_vscyto3d_pretrained_randinit/randinit
- nucleus
- fcmae_vscyto3d_pretrained_cytoland/cytoland
- fcmae_vscyto3d_pretrained_infectionft/infectionft
- fcmae_vscyto3d_pretrained_randinit/randinit
- vscyto3d_cytolandft
- a549_mantis
- ipsc_confocal
- vscyto3d_infectionft_dynacellft
- a549_mantis
- ipsc_confocal
- er/pix2pix3d_unetvit/ipsc_confocal
- membrane/pix2pix3d_unetvit
- a549_mantis
- ipsc_confocal
- joint_ipsc_confocal_a549_mantis
- nucleus
- fcmae_vscyto3d_pretrained/joint_ipsc_confocal_a549_mantis
- pix2pix3d_unetvit
- a549_mantis
- ipsc_confocal
- joint_ipsc_confocal_a549_mantis
- examples
- hf_demo
- cards
- hf_space
- config_templates
- notebooks
- src/dynacell
- evaluation
- _configs
- tests
- tools
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
0 commit comments