feat(nextflow): integrate virtual-stain into mantis-v2 pipeline#271
Merged
Conversation
Recreate the Nextflow virtual-stain integration against the current in-process cytoland CLI (post-#267), replacing the #259 approach which was built on the old `viscy predict` + `--copy` temp-zarr flow. - New `nextflow/modules/virtual_stain.nf`: init → preprocess → fan-out per-position GPU prediction. Per-position work is a single `biahub virtual-stain --cluster debug` call (no temp zarr, no --copy). - `viscy preprocess` runs over the whole input plate to compute the normalization statistics the model reads via read_norm_meta. - Both `biahub virtual-stain` and `viscy` run under biahub's `stain` extra (cytoland → viscy-utils provides the `viscy` console script). - Add a `gpu` process label (queue=gpu) to nextflow.config. - Wire virtual_stain_wf after reconstruct in mantis-v2.nf. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
4 tasks
Every step CLI (deskew, flat-field, apply-inv-tf, virtual-stain) now emits
its per-position resource request through a single shared helper,
biahub.cli.utils.echo_resources, as a JSON payload:
RESOURCES:{"cpus": 16, "mem_gb": 64, "time_min": 480}
The same (cpus, total mem_gb, time_min) values feed both the CLI's own
slurm_* submission args and, via parse_resources in nextflow, the Nextflow
per-position task directives — so the SLURM fan-out and the Nextflow fan-out
can no longer request different resources.
- Add echo_resources() and parse the JSON in common.nf::parse_resources,
which now returns time_min alongside cpus/mem_gb.
- Emit memory as a TOTAL (mem_gb) and request it via slurm_mem rather than
slurm_mem_per_cpu, fixing virtual-stain's per-cpu over-request.
- Wire each CLI's slurm_time from time_min; drop the hardcoded `time`
literals in the run_* processes for `meta.time_min * task.attempt`.
- virtual-stain computes time_min before the init_only return so --init
emits it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Avoids confusion with "minimum time"; renames the resource payload key and Nextflow meta field consistently across both languages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aofei-liu
added a commit
that referenced
this pull request
Jul 9, 2026
Adopt merged-PR decisions: JSON echo_resources protocol (cpus/mem_gb/ time_minutes) migrated across stabilize/track/concatenate/estimate-crop/ nf init-resources; take main's deskew/flat-field/reconstruct/virtual-stain (#255/#257/#258/#267/#271); nextflow.config preempted-partition hardening (#275); iohub>=0.3.8 (#272). Remove obsolete virtual-stain CLI test; move example_pipeline_steps.yml to nextflow/. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tayllatheodoro
added a commit
that referenced
this pull request
Jul 9, 2026
Bring PR #260 (track --init/--cluster + rename-channels + cellpose) up to date with main. main had already merged the deskew/flat-field/reconstruct (#255/#257/ #258), cytoland virtual-stain (#267/#271), and stitch-vendor (#238) refactors, which the refactor-* stack implemented in parallel; those subsystems are resolved in favour of main. Conflict resolution: - Took main's versions of deskew.py, flat_field_correction.py, apply_inverse_transfer_function.py, virtual_stain.py (cytoland rewrite, supersedes the stack's VisCy --init/--copy), cli/utils.py, and the deskew/flat_field/reconstruct/virtual_stain Nextflow modules + tests. - Took main's nextflow.config (adds the gpu label/queue that tracking.nf's run_track needs) and common.nf (JSON resources contract), and mantis-v2.nf. - pyproject.toml/uv.lock resolved to main (identical after taking iohub>=0.3.8). - Adopted main's JSON resources contract in track.py: --init now calls echo_resources(num_cpus, mem_gb, time_minutes) instead of the old space-separated 'RESOURCES:' line, so parse_resources (common.nf) works. - Preserved PR #260's unique work (tracking.nf, rename_channels.nf, the refactored/cellpose track.py, example_track_settings.yml).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Recreates the Nextflow virtual-staining integration against the current
biahub virtual-stainCLI (post-#267, which rewrote prediction to run in-process via cytoland). Replaces #259, which was built on the oldviscy predict→ temp-zarr →biahub virtual-stain --copyflow that no longer exists.The new CLI makes this much simpler than #259: per-position work is a single in-process
biahub virtual-stain --cluster debugcall — identical in shape torun_deskew/run_apply_inv_tf. No temp per-position zarr, no--copymerge.Changes
nextflow/modules/virtual_stain.nf— three-phase subworkflow mirroringreconstruct.nf:init_virtual_stain(cpu_local):biahub virtual-stain --initcreates the output plate, emitsRESOURCES:run_virtual_stain_preprocess(cpu):viscy preprocessover the whole input plate — computes the normalization statisticsvirtual_stain_positionreads viaviscy_data.read_norm_meta(errors if missing). Note: mutates the input store by adding norm metadata.run_virtual_stain(gpu,--gres=gpu:1): per-positionbiahub virtual-stain --cluster debugbiahub virtual-stainandviscyrun via astain_cmd()helper that adds--extra stain(cytoland → viscy-utils provides theviscyconsole script).nextflow.config: add agpuprocess label (queue = 'gpu'in the slurm profile).mantis-v2.nf: addparams.virtual_stain_config, include + wirevirtual_stain_wfafter reconstruct, uncomment3-virtual-staininDIRECTORY_LAYOUT.Test plan
biahub virtual-stain --help→ shows--init/--clusteruv run --extra stain viscy preprocess --help→ resolves; flags (--data_path/--channel_names/--num_workers/--block_size) match the module exactlymantis-v2.nf -profile slurmagainst a small reconstruct output (--max_positions 2), passing--virtual_stain_config settings/example_virtual_stain_settings.ymland--biahub_project <path>. Expectinit_virtual_stain (1/1) → run_virtual_stain_preprocess (1/1) → run_virtual_stain (2/2)and an output plate at3-virtual-stain/<ds>.zarrwith the predicted channels and input geometry.Closes #259.
🤖 Generated with Claude Code