AICAttack: Adversarial Image Captioning Attack with Attention-Based Optimization

Overview

This is a Python implementation of `AICAttack: Adversarial Image Captioning Attack with Attention-Based Optimization''. DOI: 10.1007/s11633-024-1535-z. A GPU environment is required for running the code.

Requirements:

The code is written in Python 3.8, with:

rouge-score==0.1.2

tensorflow==2.7.4

torch==2.0.1

Other Python packages can be installed by running the following command from the command line.

pip install -r requirements.txt The checkpoints you need to download: (https://drive.google.com/file/d/1HkhMC_SBRz57puNTN9MqCz-Vut5OzvSA/view?usp=sharing)

Datasets and victim models download

First, you need to download victim models from the following links:

SAT: https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Image-Captioning

BLIP: https://github.com/salesforce/BLIP

Secondly, the datasets are available at:

COCO -- https://cocodataset.org/#home

Flickr8k -- https://www.kaggle.com/datasets/adityajn105/flickr8k

Move data and model to the root directory.

Evaluation

To attack an image, run the attack_sat.py or attack_blip_test.py according to the victim model. For example, to attack SAT:

python3 attack_sat.py

Test

To test AICAttack performance, you should run the "seperate.py".

python3 seperate.py

The results are saved in a .json file in outputs/jsons/........json.

You can customize your test run by changing the following arguments:

• --name, dataset name;
• --num_input, number of input data;
• --pixels, number of pixels to attack;
• --max_iter, max iteration of DE algorithm;
• --pop_size, population size;
• --keywords, number of keywords;
• --F, mutation scale factor ($\lambda$ in paper);
• --image_size, input image size;
• --metric, metrics used for comparing attack performance in DE algorithm;
• --separate, word-attack or sentence-attack;
• --attention, Whether use attention;
• --save_img, Save the perturbed image or not.