Pixel2Cancer

Subscribe us: https://groups.google.com/u/2/g/bodymaps

This repository provides the code and checkpoints for our novel tumor synthesis approach, Pixel2Cancer, which can simulate tumor development within organs with realistic texture, shape, and interactions with other tissues.

Paper

From Pixel to Cancer: Cellular Automata in Computed Tomography
Yuxiang Lai^1,2, Xiaoxi Chen³, Angtian Wang¹, Alan L. Yuille¹, and Zongwei Zhou^1,*
1 Johns Hopkins University
² Southeast University,
³ University of Illinois Urbana-Champaign
International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2024; Early Accept)
paper | code

We have summarized publications related to tumor synthesis in Awesome Synthetic Tumors .

Model

You can download the trained model files from the table below

Organ	Tumor	Model	Pre-trained?	Download
liver	real	unet	no	link
liver	real	swin_unetrv2_base	no	link
liver	synt	unet	no	link
liver	synt	swin_unetrv2_base	no	link
pancreas	real	unet	no	link
pancreas	real	swin_unetrv2_base	no	link
pancreas	synt	unet	no	link
pancreas	synt	swin_unetrv2_base	no	link
kidney	real	unet	no	link
kidney	real	swin_unetrv2_base	no	link
kidney	synt	unet	no	link
kidney	synt	swin_unetrv2_base	no	link

Where to put those model files?

It is suggested that you put those into a folder named runs/your_model_name, which will be the same path to save the trained model after training. For example, runs/synt.no_pretrain.unet/model.pt for the non-pretrained U-Net model trained with synthetic tumors.

You can download other materials from these links:

All other checkpoints: link

Data Download

(1). 🚍 Public datasets:

Below are the public dataset download links for different organs used in medical image analysis tasks:

🧠 Organ	🔗 Download Link	📖 Dataset Name
Liver	Download LiTS Dataset	04_LiTS
Kidney	Download KiTS Dataset	05_KiTS
Pancreas	Download Pancreas Dataset	Task07_Pancreas

(2). 🚗 Our private 9k data of AbdominalAtlas1.1:

The release of AbdomenAtlas 1.0 can be found at This Link.

(3). Where to put those data files:

It is suggested that you put your download dataset folder under the folder your_data_path. For example, put the Liver dataset 04_LiTS under the path Pixel2Cancer/your_data_path/04_LiTS.

0. Installation

git clone https://github.com/MrGiovanni/Pixel2Cancer.git
cd Pixel2Cancer/

# download pre-trained models
wget https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/model_swinvit.pt

🚨 See detailed installation instructions to create an environment and obtain requirements.

1. Train segmentation models using synthetic tumors

datapath= your_data_path

# UNET (no.pretrain)
CUDA_VISIBLE_DEVICES=0,1,2,3 python -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=unet --val_every=200 --max_epochs=2000 --save_checkpoint --workers=0 --noamp --distributed --dist-url=tcp://127.0.0.1:12235 --cache_num=200 --val_overlap=0.5 --syn --logdir="runs/synt.no_pretrain.unet" --train_dir $datapath --val_dir $datapath --json_dir datafolds/healthy.json

# Swin-UNETR-Base (pretrain)
CUDA_VISIBLE_DEVICES=0,1,2,3 python -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=swin_unetrv2 --swin_type=base --val_every=200 --max_epochs=2000 --save_checkpoint --workers=0 --noamp --distributed --dist-url=tcp://127.0.0.1:12231 --cache_num=200 --val_overlap=0.5 --syn --logdir="runs/synt.pretrain.swin_unetrv2_base" --train_dir $datapath --val_dir $datapath --json_dir datafolds/healthy.json --use_pretrained

# Swin-UNETR-Base (no.pretrain)
CUDA_VISIBLE_DEVICES=0,1,2,3 python -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=swin_unetrv2 --swin_type=base --val_every=200 --max_epochs=2000 --save_checkpoint --workers=0 --noamp --distributed --dist-url=tcp://127.0.0.1:12231 --cache_num=200 --val_overlap=0.5 --syn --logdir="runs/synt.no_pretrain.swin_unetrv2_base" --train_dir $datapath --val_dir $datapath --json_dir datafolds/healthy.json

# Swin-UNETR-Small (no.pretrain)
CUDA_VISIBLE_DEVICES=0,1,2,3 python -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=swin_unetrv2 --swin_type=small --val_every=200 --max_epochs=2000 --save_checkpoint --workers=0 --noamp --distributed --dist-url=tcp://127.0.0.1:12233 --cache_num=200 --val_overlap=0.5 --syn --logdir="runs/synt.no_pretrain.swin_unetrv2_small" --train_dir $datapath --val_dir $datapath --json_dir datafolds/healthy.json

# Swin-UNETR-Tiny (no.pretrain)
CUDA_VISIBLE_DEVICES=0,1,2,3 python -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=swin_unetrv2 --swin_type=tiny --val_every=200 --max_epochs=2000 --save_checkpoint --workers=0 --noamp --distributed --dist-url=tcp://127.0.0.1:12234 --cache_num=200 --val_overlap=0.5 --syn --logdir="runs/synt.no_pretrain.swin_unetrv2_tiny" --train_dir $datapath --val_dir $datapath --json_dir datafolds/healthy.json

⚠️ You may have to modify:

--logdir: the path to save trained model pickle
--json_dir: the path for .json file to load the dataset

🚨 [ IMPORTANT! ] About .json file location:

The .json file here is the configuration file for loading the data. For example, I want to train with mixed pancrea dataset that has both synt and original tumors, therefore, configured as: --json_dir datafolds/mix_pancreas.json. And the corrsponding dataset path will be: your_data_path\10_Decathlon\Task07_Pancreas, the same as defined in the .json file.

💡💡💡 Please check your .json file before training

2. Train segmentation models using real tumors (for comparison)

datapath= your_data_path

# UNET (no.pretrain)
CUDA_VISIBLE_DEVICES=0,1,2,3 python -W ignore -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=unet --val_every=200 --val_overlap=0.5 --max_epochs=2000 --save_checkpoint --workers=2 --noamp --distributed --dist-url=tcp://127.0.0.1:12235 --cache_num=200 --logdir="runs/real.no_pretrain.unet" --train_dir $datapath --val_dir $datapath --json_dir datafolds/lits.json

# Swin-UNETR-Base (pretrain)
CUDA_VISIBLE_DEVICES=0,1,2,3 python -W ignore -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=swin_unetrv2 --swin_type=base --val_every=200 --val_overlap=0.5 --max_epochs=2000 --save_checkpoint --workers=2 --noamp --distributed --dist-url=tcp://127.0.0.1:12231 --cache_num=200 --logdir="runs/real.pretrain.swin_unetrv2_base" --train_dir $datapath --val_dir $datapath --json_dir datafolds/lits.json --use_pretrained

# Swin-UNETR-Base (no.pretrain)
CUDA_VISIBLE_DEVICES=0,1,2,3 python -W ignore -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=swin_unetrv2 --swin_type=base --val_every=200 --val_overlap=0.5 --max_epochs=2000 --save_checkpoint --workers=2 --noamp --distributed --dist-url=tcp://127.0.0.1:12232 --cache_num=200 --logdir="runs/real.no_pretrain.swin_unetrv2_base" --train_dir $datapath --val_dir $datapath --json_dir datafolds/lits.json

# Swin-UNETR-Small (no.pretrain)
CUDA_VISIBLE_DEVICES=0,1,2,3 python -W ignore -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=swin_unetrv2 --swin_type=small --val_every=200 --val_overlap=0.5 --max_epochs=2000 --save_checkpoint --workers=2 --noamp --distributed --dist-url=tcp://127.0.0.1:12233 --cache_num=200 --logdir="runs/real.no_pretrain.swin_unetrv2_small" --train_dir $datapath --val_dir $datapath --json_dir datafolds/lits.json

# Swin-UNETR-Tiny (no.pretrain)
CUDA_VISIBLE_DEVICES=0,1,2,3 python -W ignore -W ignore main.py --optim_lr=4e-4 --batch_size=2 --lrschedule=warmup_cosine --optim_name=adamw --model_name=swin_unetrv2 --swin_type=tiny --val_every=200 --val_overlap=0.5 --max_epochs=2000 --save_checkpoint --workers=2 --noamp --distributed --dist-url=tcp://127.0.0.1:12234 --cache_num=200 --logdir="runs/real.no_pretrain.swin_unetrv2_tiny" --train_dir $datapath --val_dir $datapath --json_dir datafolds/lits.json

3. Evaluation

AI model trained by synthetic tumors

datapath= your_data_path

# UNET (no.pretrain)
CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=unet --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/synt.no_pretrain.unet --save_dir out
# Swin-UNETR-Base (pretrain)
CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=swin_unetrv2 --swin_type=base --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/synt.pretrain.swin_unetrv2_base --save_dir out
# Swin-UNETR-Base (no.pretrain)
CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=swin_unetrv2 --swin_type=base --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/synt.no_pretrain.swin_unetrv2_base --save_dir out
# Swin-UNETR-Small (no.pretrain)
CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=swin_unetrv2 --swin_type=small --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/synt.no_pretrain.swin_unetrv2_small --save_dir out
# Swin-UNETR-Tiny (no.pretrain)
CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=swin_unetrv2 --swin_type=tiny --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/synt.no_pretrain.swin_unetrv2_tiny --save_dir out

AI model trained by real tumors

datapath= your_data_path

# UNET (no.pretrain)
CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=unet --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/real.no_pretrain.unet --save_dir out
# Swin-UNETR-Base (pretrain)
CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=swin_unetrv2 --swin_type=base --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/real.pretrain.swin_unetrv2_base --save_dir out
# Swin-UNETR-Base (no.pretrain)
CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=swin_unetrv2 --swin_type=base --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/real.no_pretrain.swin_unetrv2_base --save_dir out
# Swin-UNETR-Small (no.pretrain)
CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=swin_unetrv2 --swin_type=small --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/real.no_pretrain.swin_unetrv2_small --save_dir out
# Swin-UNETR-Tiny (no.pretrain)
CUDA_VISIBLE_DEVICES=0 python -W ignore validation.py --model=swin_unetrv2 --swin_type=tiny --val_overlap=0.75 --val_dir $datapath --json_dir datafolds/lits.json --log_dir runs/real.no_pretrain.swin_unetrv2_tiny --save_dir out

Outputs

All the prediction outpus can be found at --save_dir out folder. There will be the prediction form of .nii.gz files as well as metrics.csv for recording metric performance.

Data Setting

Below are some .json file settings for training and validation:

Train on private data:

# AbdominalAtlas1.1 training data list
# Liver 
--json_dir /datafolds/Bodymap/Atlas9k_liver/Atlas9k_liver_fold0.json
--json_dir /datafolds/Bodymap/Atlas9k_liver/Atlas9k_liver_fold1.json
--json_dir /datafolds/Bodymap/Atlas9k_liver/Atlas9k_liver_fold2.json
--json_dir /datafolds/Bodymap/Atlas9k_liver/Atlas9k_liver_fold3.json
--json_dir /datafolds/Bodymap/Atlas9k_liver/Atlas9k_liver_fold4.json
#Pancreas
--json_dir /datafolds/Bodymap/Atlas9k_liver/Atlas9k_pancreas_fold0.json
--json_dir /datafolds/Bodymap/Atlas9k_liver/Atlas9k_pancreas_fold1.json
--json_dir /datafolds/Bodymap/Atlas9k_liver/Atlas9k_pancreas_fold2.json
--json_dir /datafolds/Bodymap/Atlas9k_liver/Atlas9k_pancreas_fold3.json
--json_dir /datafolds/Bodymap/Atlas9k_liver/Atlas9k_pancreas_fold4.json
#Kidney
--json_dir /datafolds/Bodymap/Atlas9k_liver/Atlas9k_kidney_fold0.json
--json_dir /datafolds/Bodymap/Atlas9k_liver/Atlas9k_kidney_fold1.json
--json_dir /datafolds/Bodymap/Atlas9k_liver/Atlas9k_kidney_fold2.json
--json_dir /datafolds/Bodymap/Atlas9k_liver/Atlas9k_kidney_fold3.json
--json_dir /datafolds/Bodymap/Atlas9k_liver/Atlas9k_kidney_fold4.json

Train on public data & Intern experiments:

# Public training data list
# Liver
--json_dir /datafolds/5_fold/liver/liver_tumor_0.json
--json_dir /datafolds/5_fold/liver/liver_tumor_1.json
--json_dir /datafolds/5_fold/liver/liver_tumor_2.json
--json_dir /datafolds/5_fold/liver/liver_tumor_3.json
--json_dir /datafolds/5_fold/liver/liver_tumor_4.json
# Pancreas
--json_dir /datafolds/5_fold/pancreas/pancreas_tumor_0.json
--json_dir /datafolds/5_fold/pancreas/pancreas_tumor_1.json
--json_dir /datafolds/5_fold/pancreas/pancreas_tumor_2.json
--json_dir /datafolds/5_fold/pancreas/pancreas_tumor_3.json
--json_dir /datafolds/5_fold/pancreas/pancreas_tumor_4.json
# Kidney
--json_dir /datafolds/5_fold/kidney/kidney_tumor_0.json
--json_dir /datafolds/5_fold/kidney/kidney_tumor_1.json
--json_dir /datafolds/5_fold/kidney/kidney_tumor_2.json
--json_dir /datafolds/5_fold/kidney/kidney_tumor_3.json
--json_dir /datafolds/5_fold/kidney/kidney_tumor_4.json

Acknowledgement

This work was supported by the Lustgarten Foundation for Pancreatic Cancer Research and the McGovern Foundation. The segmentation backbone is based on Swin UNETR; we appreciate the effort of the MONAI Team to provide and maintain open-source code to the community.