ixrt

CLIP (ixRT)

Model Description

CLIP (Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3.

Supported Environments

GPU	IXUCA SDK	Release	Branch
MR-V100	4.4.0	26.03	release/26.03
MR-V100	4.3.0	25.12	release/25.12

Note: 请切换到与您的 SDK 版本对应的 Release 分支进行测试。请勿直接在 master 分支上运行测试，因为 master 分支可能包含与您的本地 SDK 版本不兼容的最新更改。

切换分支命令示例：git checkout release/26.03

Model Preparation

Prepare Resources

Pretrained model: https://huggingface.co/openai/clip-vit-base-patch32

Dataset: https://www.image-net.org/download.php to download the validation dataset.

git lfs install
git clone https://huggingface.co/openai/clip-vit-base-patch32 clip-vit-base-patch32

Install Dependencies

Contact the Iluvatar administrator to get the missing packages:

• ixrt-*.whl

pip3 install -r requirements.txt

Model Conversion

mkdir -p checkpoints/clip
python3 export.py --output checkpoints/clip/clip.onnx

Model Inference

git clone https://gitee.com/deep-spark/iluvatar-corex-ixrt.git --depth=1
export OPTIMIER_FILE=./iluvatar-corex-ixrt/tools/optimizer/optimizer.py
export DATASETS_DIR=/path/to/imagenet_val/
export PROJ_DIR=./
export CHECKPOINTS_DIR=./checkpoints/clip
export RUN_DIR=../../ixrt_common
export CONFIG_DIR=../../ixrt_common/config/CLIP_CONFIG

FP16

# Accuracy
bash scripts/infer_clip_fp16_accuracy.sh
# Performance
bash scripts/infer_clip_fp16_performance.sh

Model Results

Model	BatchSize	Precision	FPS	Top-1(%)	Top-5(%)
CLIP	32	FP16	350.94	59.68	86.14