Skip to content

Commit c56cfe7

Browse files
KumoLiupre-commit-ci[bot]ericspod
authored
8110 update highlights page for v1.4 (#8111)
Part of #8110 ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: YunLiu <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Kerfoot <[email protected]>
1 parent 3fd73f1 commit c56cfe7

File tree

6 files changed

+70
-1
lines changed

6 files changed

+70
-1
lines changed

docs/images/maisi_train.png

183 KB
Loading

docs/images/vista2d.png

445 KB
Loading

docs/images/vista3d.png

85 KB
Loading

docs/source/whatsnew.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ What's New
66
.. toctree::
77
:maxdepth: 1
88

9+
whatsnew_1_4.md
910
whatsnew_1_3.md
1011
whatsnew_1_2.md
1112
whatsnew_1_1.md

docs/source/whatsnew_1_3.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# What's new in 1.3 🎉🎉
1+
# What's new in 1.3
22

33
- Bundle usability enhancements
44
- Integrating MONAI Generative into MONAI core

docs/source/whatsnew_1_4.md

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# What's new in 1.4 🎉🎉
2+
3+
- MAISI: state-of-the-art 3D Latent Diffusion Model
4+
- VISTA-3D: interactive foundation model for segmenting and anotating human anatomies
5+
- VISTA-2D: cell segmentation pipeline
6+
- Integrating MONAI Generative into MONAI core
7+
- Lazy TensorRT export via `trt_compile`
8+
- Geometric Data Support
9+
10+
11+
## MAISI: state-of-the-art 3D Latent Diffusion Model
12+
13+
![maisi](../images/maisi_train.png)
14+
15+
MAISI (Medical AI for Synthetic Imaging) is a state-of-the-art three-dimensional (3D) Latent Diffusion Model designed for generating high-quality synthetic CT images with or without anatomical annotations. This AI model excels in data augmentation and creating realistic medical imaging data to supplement limited datasets due to privacy concerns or rare conditions. It can also significantly enhance the performance of other medical imaging AI models by generating diverse and realistic training data.
16+
17+
A tutorial for generating large CT images accompanied by corresponding segmentation masks using MAISI is provided within
18+
[`project-monai/tutorials`](https://github.com/Project-MONAI/tutorials/blob/main/generation/maisi).
19+
It contains the following features:
20+
- A foundation Variational Auto-Encoder (VAE) model for latent feature compression that works for both CT and MRI with flexible volume size and voxel size
21+
- A foundation Diffusion model that can generate large CT volumes up to 512 × 512 × 768 size, with flexible volume size and voxel size
22+
- A ControlNet to generate image/mask pairs that can improve downstream tasks, with controllable organ/tumor size
23+
24+
## VISTA-3D: state-of-the-art 3D Latent Diffusion Model
25+
26+
![vista-3d](../images/vista3d.png)
27+
28+
VISTA-3D is a specialized interactive foundation model for 3D medical imaging. It excels in providing accurate and adaptable segmentation analysis across anatomies and modalities. Utilizing a multi-head architecture, VISTA-3D adapts to varying conditions and anatomical areas, helping guide users' annotation workflow.
29+
30+
A tutorial showing how to finetune VISTA-3D on spleen dataset is provided within
31+
[`project-monai/tutorials`](https://github.com/Project-MONAI/tutorials/blob/main/vista_3d).
32+
It supports three core workflows:
33+
- Segment everything: Enables whole body exploration, crucial for understanding complex diseases affecting multiple organs and for holistic treatment planning.
34+
- Segment using class: Provides detailed sectional views based on specific classes, essential for targeted disease analysis or organ mapping, such as tumor identification in critical organs.
35+
- Segment point prompts: Enhances segmentation precision through user-directed, click-based selection. This interactive approach accelerates the creation of accurate ground-truth data, essential in medical imaging analysis.
36+
37+
## VISTA-2D: cell segmentation pipeline
38+
39+
![vista-2d](../images/vista2d.png)
40+
41+
VISTA-2D is a comprehensive training and inference pipeline for cell segmentation in imaging applications. For more information, refer to this [Blog](https://developer.nvidia.com/blog/advancing-cell-segmentation-and-morphology-analysis-with-nvidia-ai-foundation-model-vista-2d/)
42+
43+
Key features of the model include:
44+
- A robust deep learning algorithm utilizing transformers
45+
- Foundational model as compared to specialist models
46+
- Supports a wide variety of datasets and file formats
47+
- Capable of handling multiple imaging modalities
48+
- Multi-GPU and multinode training support
49+
50+
A tutorial demonstrating how to train a cell segmentation model using the MONAI framework on the Cellpose dataset can be found in [`project-monai/tutorials`](https://github.com/Project-MONAI/tutorials/blob/main/vista_2d).
51+
52+
## Integrating MONAI Generative into MONAI Core
53+
54+
Key modules originally developed in the [MONAI GenerativeModels](https://github.com/Project-MONAI/GenerativeModels) repository have been integrated into the core MONAI codebase. This integration ensures consistent maintenance and streamlined release of essential components for generative AI. In this version, all utilities, networks, diffusion schedulers, inferers, and engines have been migrated into the core codebase. Special care has been taken to ensure saved weights from models trained using GenerativeModels can be loaded into those now integrated into core.
55+
56+
Additionally, several tutorials have been ported and are available within [`project-monai/tutorials`](https://github.com/Project-MONAI/tutorials/blob/main/generation)
57+
58+
## Lazy TensorRT export via `trt_compile`
59+
This release expands TensorRT optimization options for MONAI bundles with `trt_compile` API.
60+
The existing `trt_export` API requires the user to run a separate export script to prepare a TensorRT engine-based TorchScript model.
61+
`trt_compile` builds and saves a TensorRT engine the first time a bundle is run and provides limited dependency support.
62+
It also allows partial TensorRT export where only a certain submodule is being optimized, which improves usability.
63+
A few bundles in the MONAI model zoo, like the new [VISTA-3D](https://github.com/Project-MONAI/model-zoo/tree/dev/models/vista3d)
64+
and [VISTA-2D](https://github.com/Project-MONAI/model-zoo/tree/dev/models/vista2d) bundles, already come with `trt_inference.json` config files which use `trt_compile`.
65+
66+
## Geometric Data Support
67+
68+
MONAI introduces support for geometric data transformations as a key feature. As a starting point, ApplyTransformToPoints transform is added to facilitate matrix operations on points, enabling flexible and efficient handling of geometric transformations. Alongside this, the framework now supports conversions between boxes and points, providing seamless interoperability within detection pipelines. These updates have been integrated into existing pipelines, such as the [detection tutorial](https://github.com/Project-MONAI/tutorials/blob/main/detection) and the [3D registration workflow](https://github.com/Project-MONAI/tutorials/blob/main/3d_registration/learn2reg_nlst_paired_lung_ct.ipynb), leveraging the latest APIs for improved functionality.

0 commit comments

Comments
 (0)