Investigating text-guided cross-region feature alignment for multimodal disease localization in Chest X-Ray images

Investigating text-guided cross-region feature alignment for multimodal disease localization in Chest X-Ray images

Sourya Potti

Installation

Setup environment

conda create --name cxrcodet python=3.8 -y
conda activate cxrcodet
pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116

# Under working directory 
git clone https://github.com/souryatech/TGCRFA-CXR.git
pip install ninja
pip install -v -U git+https://github.com/facebookresearch/xformers.git@7e05e2caaaf8060c1c6baadc2b04db02d5458a94
git clone https://github.com/NVIDIA/apex && cd apex
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./ && cd ..

#Install additional dependencies
cd TGCRFA-CXR/third_party/detectron2
pip install -e .
cd ../..
pip install -r requirements.txt

Dataset Preparation

Please download the train images from the VinDr dataset and move the images to the images folder, placing the data in the following way:

datasets/
        vindr/
            zero
            annotations/
            images/
                4d..92.jpg,
                ...
                

Model Download

To download the cxr-codet model trained on VinDr, please download the config and the model

Train & Inference

To train a config on the vindr dataset, please enter the following lines in your shell:

python train_net.py \
    --config-file $CONFIG_PATH \
    MODEL.ROI_HEADS.NUM_CLASSES 15 \

If you would like to change the pretrained model weights, please do so by editing MODEL_WEIGHTS in the corresponding config file.

To test a model on the vindr dataset, please entire the following lines in your shell:

python train_net.py --num-gpus $NUM_GPUS --config-file $CONFIG_PATH --eval-only MODEL.WEIGHTS $PRETRAINED_MODEL_WEIGHTS_PATH MODEL.ROI_HEADS.NUM_CLASSES 15

Acknowledgements

This work's model is built on the amazing works: CoDet and BiomedCLIP