DynaCLR gut-cell training: per-cell Z centering, cell split, gut configs#481
Open
edyoshikun wants to merge 7 commits into
Open
DynaCLR gut-cell training: per-cell Z centering, cell split, gut configs#481edyoshikun wants to merge 7 commits into
edyoshikun wants to merge 7 commits into
Conversation
…orm fix
Add two opt-in MultiExperimentDataModule params and fix a latent
normalization bug, motivated by static single-FOV datasets (one position
per store, per-cell focal plane, multi-channel input):
- z_center_source ('fov_center' default | 'cell_z'): 'cell_z' centers the
Z extraction window on each cell's parquet z column instead of the
per-FOV focus plane. Applies z_focus_offset and clamps into the image.
- split_mode ('fov' default | 'cell'): 'cell' splits train/val by cell_id
within each experiment, needed when every experiment is a single FOV
(FOV split would leave zero validation FOVs).
- _build_norm_meta: guard the parquet norm fast-path to bag-of-channels
(from_index) mode only. All-channels/fixed modes now fall through to the
zarr .zattrs path, which returns every channel's stats and all levels.
Previously multi-channel per-channel NormalizeSampled KeyError'd because
a flat-parquet row carries only its own channel's timepoint stats.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add DynaCLR configs for Zuben's zebrafish gut cell data (static, single-timepoint, 4 channels: nuclear/septate/brush_border/SuH). Phase 1 bag-of-channels + Phase 2 4-channel configs and launchers, plus data-prep scripts: verify parquet<->zarr, generate v2->v3 biahub-concatenate configs, generate viscy preprocess jobs, and conform the collaborator parquet to CELL_INDEX_SCHEMA. Launchers require WORKSPACE_DIR set to the runner's clone; the parquet path points at the shared collab dataset. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…olumn
Replace the z_center_source flag with one Z-window focus resolution, and
consolidate focus onto a dedicated parquet column, mirroring how norm stats
already prefer the parquet over zattrs (I/O: no per-sample zarr open).
- Rename CELL_INDEX_SCHEMA column z_focus_mean -> z_focus. z_focus is the
plane the Z window centers on; the tracked per-cell z column is now left
distinct and untouched by preprocessing.
- dataset.py._slice_patch: center on the per-sample z_focus column when
populated, else the per-experiment z_range center (registry-derived from
zattrs focus_slice, else mid-stack). Removes z_center_source.
- preprocess-cell-index: add --focus-level {fov, per_timepoint} choosing
which focus_slice level feeds z_focus. No longer overwrites z.
- conform_parquet.py (gut): end-to-end — conform, fill norm_*, then seed
z_focus = bbox-center z last (gut has no focus_slice zattrs).
- Gut configs drop z_center_source.
No regression: existing parquets have no z_focus column -> reads all-null ->
fallback computes the identical z_start as before. Verified against the
shipped 3D parquet (z=8-40 tracked != z_focus_mean~23 per-FOV). Both gut
phases pass fast_dev_run. DAG training.md updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared trainer recipe logs to the computational_imaging entity. Add an optional WANDB_ENTITY env hook to both gut launchers so a collaborator can log to their own entity without editing the shared recipe (EXTRA_ARGS overrides the recipe value). Unset = unchanged default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tate Set the contrastive scheme for the imbalanced 6-state gut data: - SimCLR self-positives (anchor == positive crop) — trivially same-marker and no supervised same-state pull that would fight same-state negatives. - Phase 1 bag-of-channels: batch_group_by=marker so every negative shares the anchor's marker → the model can't shortcut on channel identity and must learn cell-state structure; stratify_by=perturbation balances the 6 states within each marker batch (dataset is ~3.5x imbalanced, stage_0 1379 vs stage_5 390). - Phase 2 multi-channel: no marker shortcut (all channels stacked), so just self-positives + stratify_by=perturbation. Supervised same-state (SupCon) was considered and rejected for the first pass: same-state cells co-occurring in a batch become negatives, fighting the pull. Deferred to a later phase (consecutive-state positives for temporal regularization). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
W&B logs sample-image grids by writing PNGs to a temp dir under $TMPDIR before upload. On some SLURM nodes the default /tmp is per-job and gets swept mid-run, so validation image logging crashed with FileNotFoundError: .../wandb-media/*.png (training itself was fine — the run reached epoch 3). Pin TMPDIR and WANDB_DIR to persistent paths under MODEL_ROOT that the launcher creates, so the media temp dir can't vanish. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Override log_embeddings_every_n_epochs (base recipe default 10) to 5 in both gut configs to watch the 6 developmental states separate earlier in training. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds DynaCLR training support for Zuben's zebrafish gut-cell data (jacobo group) — static, single-timepoint, 4-channel (nuclear / septate / brush_border / SuH) — plus a small, general datamodule change that consolidates Z-window focus onto the parquet.
Datamodule changes (
applications/dynaclr/src/dynaclr/data/+packages/viscy-data/)z_focusparquet column when populated; otherwise it falls back to the per-experimentz_range(registry-derived from zattrsfocus_slice, else mid-stack). This mirrors hownorm_*stats already prefer the parquet over zattrs (no per-sample zarr open). Columnz_focus_meanis renamed toz_focus; the tracked per-cellzcolumn is now a distinct quantity, untouched by preprocessing.preprocess-cell-index --focus-level {fov, per_timepoint}chooses whichfocus_slicelevel feedsz_focus(with--focus-channel).split_mode(fovdefault |cell):cellsplits train/val bycell_idwithin each experiment — needed when every experiment is a single FOV (FOV split leaves zero validation FOVs)._build_norm_metafix: the parquet norm fast-path is guarded to bag-of-channels (from_index) mode; all-channels / fixed modes fall through to the zarr.zattrspath so per-channelNormalizeSampledgets every channel's stats.No regression for existing models. Existing parquets have no
z_focuscolumn → read as all-null → the fallback computes the identicalz_startas before (verified against the shipped 3D parquet, where trackedz≠ per-FOV focus). New params default to current behavior.Configs + scripts
DynaCLR-3D-Gut-BagOfChannels.{yml,sh}— Phase 1: one channel per sample, SimCLR self-positives, cell-level split, Z window centered onz_focus.DynaCLR-3D-Gut-MultiChannel.{yml,sh}— Phase 2: all 4 channels stacked, per-channelfov_statisticsnormalization.configs/collections/zuben_gut/— data-prep scripts: verify parquet↔zarr, generate v2→v3biahub concatenateconfigs, generateviscy preprocessjobs, and conform the collaborator parquet end-to-end (schema + norm + seedz_focus = z).Launchers require
WORKSPACE_DIRset to the runner's clone; the parquet path points at the shared collab dataset (/hpc/projects/jacobo_group/collab/ed/dynaclr).Docs
applications/dynaclr/docs/DAGs/training.mdupdated:z_focuscolumn,--focus-level, and the parquet-first centering behavior.Testing
Both configs pass
fast_dev_runon an A40 before and after the consolidation, verified from a fresh worktree offmain.🤖 Generated with Claude Code