Harmonize planning doc

.planning/MILESTONES.md

@@ -0,0 +1,61 @@
+# Milestones: VisCy Modularization
+
+## v1.0 — Transforms & Monorepo Skeleton
+
+**Shipped:** 2026-01-29
+**Phases:** 1-5 (4 executed, 1 deferred)
+
+**What shipped:**
+- uv workspace scaffolding with `packages/` directory
+- `viscy-transforms` package extracted with src layout
+- Clean import paths: `from viscy_transforms import X`
+- hatchling + uv-dynamic-versioning build system
+- All 16 transform modules migrated with full test suite
+- CI/CD: 9-job test matrix (3 OS x 3 Python) + lint workflow
+- prek hooks with ruff formatting
+
+**Deferred:**
+- Phase 4: Documentation (Zensical + GitHub Pages)
+
+**Last phase:** 5
+
+---
+
+## v1.1 — Extract viscy-data
+
+**Shipped:** 2026-02-14
+**Phases:** 6-9 (4 phases, 9 plans)
+
+**What shipped:**
+- `viscy-data` package extracted with src layout (15 modules, 4015 LOC)
+- All 13 data modules migrated with clean import paths (45 public exports)
+- No cross-package dependency on viscy-transforms
+- Optional dependency groups: `[triplet]`, `[livecell]`, `[mmap]`, `[all]`
+- Shared utilities extracted from hcs.py into _utils.py
+- Lazy import pattern for optional deps with clear error messages
+- All existing data tests passing (71 tests)
+- Tiered CI: viscy-data 3x3 base + 1x1 extras
+
+**Last phase:** 9
+
+---
+
+## v1.2 — Extract viscy-models
+
+**Shipped:** 2026-02-13
+**Phases:** 10-14 (5 phases, 9 plans)
+
+**What shipped:**
+- `viscy-models` package with 8 architectures organized by function (unet/, vae/, contrastive/)
+- Shared components extracted to `_components/` (stems, heads, blocks)
+- Full test coverage: migrated existing + new forward-pass tests for all models
+- Pure nn.Module — no Lightning/Hydra coupling
+- State dict key compatibility preserved for checkpoint loading
+- CI includes viscy-models in test matrix
+
+**Pending todo from v1.2:**
+- Fix deconv decoder channel mismatch in UNeXt2UpStage (pre-existing bug, xfailed test)
+
+**Last phase:** 14
+
+---

.planning/PROJECT.md

@@ -0,0 +1,105 @@
+# VisCy Modularization
+
+## What This Is
+
+Restructuring VisCy from a monolithic package into a uv workspace monorepo. This enables reusing transforms, dataloaders, and models in downstream projects without requiring the entire VisCy package as a dependency. Three subpackages have been extracted: `viscy-transforms` (v1.0), `viscy-data` (v1.1), and `viscy-models` (v1.2).
+
+## Core Value
+
+**Independent, reusable subpackages with clean import paths.** Users can `pip install viscy-transforms`, `pip install viscy-data`, or `pip install viscy-models` and use clean imports without pulling in the entire VisCy ecosystem.
+
+## Current Milestone: v2.0 Applications & Airtable
+
+**Goal:** Extract application-level LightningModules (DynaCLR, Cytoland) and the Airtable abstraction into independent packages, composing the extracted model and data subpackages.
+
+**Target features:**
+- `applications/DynaCLR` with ContrastiveModule LightningModule
+- `applications/Cytoland` with VSUNet/FcmaeUNet LightningModules
+- `viscy-airtable` package abstracted from current Airtable integration
+- Hydra configuration infrastructure (viscy-hydra or integrated)
+
+## Requirements
+
+### Validated
+
+- uv workspace scaffolding with `packages/` directory structure — v1.0
+- `viscy-transforms` package extracted with src layout (16 modules, 44 exports) — v1.0
+- Import path: `from viscy_transforms import X` (clean break) — v1.0
+- hatchling build backend with uv-dynamic-versioning — v1.0
+- All existing transform tests passing in new structure — v1.0
+- CI for monorepo (test matrix + lint) — v1.0
+- `viscy-data` package extracted with src layout (15 modules, 4015 LOC) — v1.1
+- All 13 data modules migrated with clean import paths — v1.1
+- Import path: `from viscy_data import X` (45 public exports) — v1.1
+- No dependency on viscy-transforms (BatchedCenterSpatialCropd in _utils.py) — v1.1
+- Optional dependency groups: `[triplet]`, `[livecell]`, `[mmap]`, `[all]` — v1.1
+- Shared utilities extracted from hcs.py into _utils.py — v1.1
+- All existing data tests passing (71 tests) — v1.1
+- Tiered CI for viscy-data (3x3 base + 1x1 extras) — v1.1
+- `viscy-models` package with src layout and function-based organization — v1.2
+- All 8 architectures migrated (UNeXt2, FCMAE, ContrastiveEncoder, ResNet3dEncoder, BetaVae25D, BetaVaeMonai, Unet2d, Unet25d) — v1.2
+- Shared components in `_components/` (stems, heads, blocks) — v1.2
+- Full test coverage for all models (existing + new forward-pass tests) — v1.2
+- Import path: `from viscy_models import UNeXt2` (clean break) — v1.2
+- State dict key compatibility preserved — v1.2
+- CI includes viscy-models in test matrix — v1.2
+
+### Active
+
+(Pending v2.0 milestone definition)
+
+### Out of Scope
+
+- Meta-package with re-exports — decided against, clean break approach
+- Backward-compatible imports — not maintaining
+- Zensical documentation / GitHub Pages — deferred
+- ONNX/TorchScript export — VAE models return SimpleNamespace, incompatible
+
+## Context
+
+**Design doc:** https://github.com/mehta-lab/VisCy/issues/353
+
+**Current state (after v1.2):**
+- uv workspace monorepo with 3 extracted packages:
+  - `packages/viscy-transforms/` — 16 transform modules, 44 exports
+  - `packages/viscy-data/` — 15 data modules, 45 exports, 4015 LOC source + 671 LOC tests
+  - `packages/viscy-models/` — 8 architectures in unet/, vae/, contrastive/ with shared _components/
+- CI: test.yml (viscy-transforms 3x3, viscy-data 3x3 + extras 1x1, viscy-models 3x3) + lint.yml
+- Python >=3.11, hatchling + uv-dynamic-versioning
+- Original code on `main` branch for reference
+
+**Architecture reference:**
+- `viscy/data/README.md` documents module inventory, class hierarchy, training pipeline mapping
+- `.planning/codebase/` contains architecture, conventions, structure analysis
+
+## Constraints
+
+- **Package naming**: hyphen for package name, underscore for import
+- **Python version**: >=3.11
+- **Build system**: hatchling with uv-dynamic-versioning
+- **Layout**: src layout (`packages/*/src/*/`)
+- **Tooling**: uv only
+- **No cross-package dependencies between data, transforms, and models**
+
+## Key Decisions
+
+| Decision | Rationale | Outcome |
+|----------|-----------|---------|
+| Clean break on imports | Simpler architecture, no re-export complexity | Good |
+| hatchling over setuptools | Modern, faster, better uv integration | Good |
+| src layout | Prevents import confusion during development | Good |
+| Tests inside packages | Isolated testing, `uv run --package` workflow | Good |
+| No viscy-transforms dep in data | Transforms separate from data | Good |
+| No viscy-transforms/data dep in models | Keep packages loosely coupled | Good |
+| Optional dependency groups for data | Heavy deps as extras, lean base install | Good |
+| Extract shared utils from hcs.py | Prevent dual-role module anti-pattern | Good |
+| BatchedCenterSpatialCropd in _utils.py | CenterSpatialCropd can't handle batch dim | Good |
+| Lazy imports for optional deps | try/except at module level, guard in __init__ | Good |
+| Flat public API | MONAI pattern, consistent across packages | Good |
+| combined.py preserved as-is | No split per REF-02 deferral | Good |
+| Pure nn.Module in viscy-models | No Lightning/Hydra coupling; maximum reusability | Good |
+| Function-based grouping (unet/, vae/, contrastive/) | Clean organization for 8+ models with shared components | Good |
+| State dict key compatibility | Non-negotiable for checkpoint loading | Good |
+
+---
+*Last updated: 2026-02-16 after harmonizing modular-data and modular-models branches*

.planning/REQUIREMENTS.md

@@ -0,0 +1,189 @@
+# Requirements: VisCy Modularization
+
+**Defined:** 2025-01-27
+**Core Value:** Independent, reusable subpackages with clean import paths
+
+## v1.0 Requirements (Complete)
+
+### Workspace Foundation
+
+- [x] **WORK-00**: Clean slate setup — wipe repo keeping only LICENSE, CITATION.cff, .gitignore
+- [x] **WORK-01**: Virtual workspace root with `[tool.uv.workspace]` and `members = ["packages/*"]`
+- [x] **WORK-02**: Shared lockfile (`uv.lock`) at repository root
+- [x] **WORK-03**: Python version floor (>=3.11) enforced in root pyproject.toml
+- [x] **WORK-04**: Pre-commit hooks configured (ruff, prek) for local development
+- [x] **WORK-05**: Shared pytest configuration in root pyproject.toml
+
+### Package Structure (viscy-transforms)
+
+- [x] **PKG-01**: src layout for viscy-transforms (`packages/viscy-transforms/src/viscy_transforms/`)
+- [x] **PKG-02**: Package pyproject.toml with hatchling build backend
+- [x] **PKG-03**: uv-dynamic-versioning configured for git-based versioning
+- [x] **PKG-04**: Package README.md with installation and usage instructions
+
+### Code Migration (viscy-transforms)
+
+- [x] **MIG-01**: All transform modules migrated from `viscy/transforms/` to package
+- [x] **MIG-02**: All transform tests migrated to `packages/viscy-transforms/tests/`
+- [x] **MIG-03**: Import path updated to `from viscy_transforms import X`
+- [x] **MIG-04**: All migrated tests passing with `uv run --package viscy-transforms pytest`
+- [x] **MIG-05**: Original `viscy/transforms/` directory removed
+
+### CI/CD
+
+- [x] **CI-01**: GitHub Actions workflow for testing viscy-transforms package
+- [x] **CI-03**: Matrix testing across Python 3.11, 3.12, 3.13 on 3 OSes
+- [x] **CI-04**: Linting via prek (uvx prek) in CI workflows
+
+## v1.1 Requirements (Complete)
+
+### Data — Package Structure
+
+- [x] **DATA-PKG-01**: viscy-data package at `packages/viscy-data/src/viscy_data/` with hatchling + uv-dynamic-versioning
+- [x] **DATA-PKG-02**: Optional dependency groups `[triplet]`, `[livecell]`, `[mmap]`, `[all]` in pyproject.toml
+- [x] **DATA-PKG-03**: No dependency on viscy-transforms; BatchedCenterSpatialCropd in _utils.py
+- [x] **DATA-PKG-04**: Shared utilities extracted from hcs.py and triplet.py into `_utils.py`
+
+### Data — Code Migration
+
+- [x] **DATA-MIG-01**: All 13 data modules migrated with updated import paths
+- [x] **DATA-MIG-02**: Flat top-level exports in `__init__.py` (45 public exports)
+- [x] **DATA-MIG-03**: Lazy imports for optional dependencies with clear error messages
+- [x] **DATA-MIG-04**: Internal imports use absolute `viscy_data.` prefix
+
+### Data — Testing
+
+- [x] **DATA-TST-01**: All existing data tests passing under new import paths
+- [x] **DATA-TST-02**: Smoke tests for import without extras and correct error messages
+
+### Data — CI/CD
+
+- [x] **DATA-CI-01**: GitHub Actions test workflow extended with viscy-data jobs
+- [x] **DATA-CI-02**: Tiered CI matrix: base deps (3x3) + full extras (1x1)
+
+## v1.2 Requirements (Complete)
+
+### Models — Package Infrastructure
+
+- [x] **MPKG-01**: Package directory `packages/viscy-models/` with src layout
+- [x] **MPKG-02**: pyproject.toml with hatchling, uv-dynamic-versioning, torch/timm/monai/numpy deps
+- [x] **MPKG-03**: `uv sync --package viscy-models` succeeds in workspace
+- [x] **MPKG-04**: `_components/` module with stems.py, heads.py, blocks.py
+
+### Models — UNet Architectures
+
+- [x] **UNET-01**: UNeXt2 migrated to `unet/unext2.py`
+- [x] **UNET-02**: FullyConvolutionalMAE migrated to `unet/fcmae.py`
+- [x] **UNET-03**: Unet2d migrated to `unet/unet2d.py`
+- [x] **UNET-04**: Unet25d migrated to `unet/unet25d.py`
+- [x] **UNET-05**: ConvBlock2D/3D migrated to `unet/_layers/`
+- [x] **UNET-06**: Forward-pass tests for UNeXt2
+- [x] **UNET-07**: FCMAE tests migrated from existing test suite
+- [x] **UNET-08**: Unet2d/Unet25d tests migrated and converted to pytest
+
+### Models — Variational Autoencoders
+
+- [x] **VAE-01**: BetaVae25D migrated to `vae/beta_vae_25d.py`
+- [x] **VAE-02**: BetaVaeMonai migrated to `vae/beta_vae_monai.py`
+- [x] **VAE-03**: Forward-pass tests for both VAE models
+
+### Models — Contrastive Learning
+
+- [x] **CONT-01**: ContrastiveEncoder migrated to `contrastive/encoder.py`
+- [x] **CONT-02**: ResNet3dEncoder migrated to `contrastive/resnet3d.py`
+- [x] **CONT-03**: Forward-pass tests for contrastive models
+
+### Models — Public API & CI
+
+- [x] **API-01**: `from viscy_models import UNeXt2` works for all 8 model classes
+- [x] **API-02**: `uv run --package viscy-models pytest` passes all tests
+- [x] **API-03**: CI test matrix updated to include viscy-models
+- [x] **API-04**: Root pyproject.toml updated with viscy-models workspace dependency
+
+### Models — Compatibility
+
+- [x] **COMPAT-01**: State dict keys preserved identically for all migrated models
+- [x] **COMPAT-02**: Mutable default arguments fixed to tuples in model constructors
+
+## Future Requirements
+
+Deferred to v2.0+ milestones. Tracked but not in current roadmap.
+
+### Applications (v2.0)
+
+- **APP-01**: applications/DynaCLR with ContrastiveModule LightningModule
+- **APP-02**: applications/Cytoland with VSUNet/FcmaeUNet LightningModules
+- **APP-03**: viscy-airtable package abstracted from current Airtable integration
+
+### Hydra Integration (future viscy-hydra package)
+
+- **HYDRA-01**: BaseModelMeta metaclass or `__init_subclass__` registry for model discovery
+- **HYDRA-02**: BaseModel(LightningModule) base class with auto Hydra instantiation
+- **HYDRA-03**: Hydra ConfigStore integration (optional dependency)
+- **HYDRA-04**: `get_model("unext2")` factory function for name-based lookup
+
+### Documentation (deferred from v1.0)
+
+- **DOC-01**: Zensical configuration (`zensical.toml`) at repository root
+- **DOC-02**: API reference auto-generated from docstrings
+- **DOC-03**: Documentation site structure with navigation
+- **DOC-04**: GitHub Pages deployment
+
+### Refactoring
+
+- **REF-01**: GPU transform protocol/mixin (GPUTransformMixin) for interface standardization
+- **REF-02**: Split combined.py into combined.py + concat.py
+- **REF-03**: Abstract cache interface across Manager.dict, tensorstore, MemoryMappedTensor
+
+## Out of Scope
+
+| Feature | Reason |
+|---------|--------|
+| Backward-compatible imports | Clean break established in v1.0 |
+| Meta-package with re-exports | Decided against, clean break approach |
+| Hydra/BaseModel registry in viscy-models | Deferred to separate viscy-hydra package |
+| LightningModule wrappers in model package | Training logic stays in applications/ |
+| ONNX/TorchScript export | VAE models return SimpleNamespace, incompatible |
+| Unified batch structure across pipelines | Different pipelines have fundamentally different batch semantics |
+| Split into multiple data packages | Over-fragmentation for 13 modules |
+| Zensical documentation | Deferred |
+
+## Traceability
+
+### v1.0 (18/18 complete)
+
+| Requirement | Phase | Status |
+|-------------|-------|--------|
+| WORK-00 through WORK-05 | Phase 1 | Complete |
+| PKG-01 through PKG-04 | Phase 2 | Complete |
+| MIG-01 through MIG-05 | Phase 3 | Complete |
+| CI-01, CI-03, CI-04 | Phase 5 | Complete |
+
+### v1.1 (12/12 complete)
+
+| Requirement | Phase | Status |
+|-------------|-------|--------|
+| DATA-PKG-01, DATA-PKG-02, DATA-PKG-04 | Phase 6 | Complete |
+| DATA-PKG-03, DATA-MIG-01 through DATA-MIG-04 | Phase 7 | Complete |
+| DATA-TST-01, DATA-TST-02 | Phase 8 | Complete |
+| DATA-CI-01, DATA-CI-02 | Phase 9 | Complete |
+
+### v1.2 (24/24 complete)
+
+| Requirement | Phase | Status |
+|-------------|-------|--------|
+| MPKG-01 through MPKG-04, UNET-05 | Phase 10 | Complete |
+| UNET-01, UNET-02, UNET-06, UNET-07 | Phase 11 | Complete |
+| CONT-01 through CONT-03, VAE-01 through VAE-03 | Phase 12 | Complete |
+| UNET-03, UNET-04, UNET-08 | Phase 13 | Complete |
+| API-01 through API-04, COMPAT-01, COMPAT-02 | Phase 14 | Complete |
+
+**Coverage:**
+- v1.0: 18 requirements, 18 complete
+- v1.1: 12 requirements, 12 complete
+- v1.2: 24 requirements, 24 complete
+- **Total: 54 requirements shipped**
+
+---
+*Requirements defined: 2025-01-27*
+*Harmonized from modular-data + modular-models branches: 2026-02-16*