Questions about VL-Gender evaluation dataset and missing file

[jiyunBae007]

Hi, thank you for releasing this great project.

I have two questions:

1. I would like to reproduce the VL-Gender evaluation dataset used in the paper. I followed the instructions in the README:

Data
Run setup_data.sh. This will download and process all data. Beware that you need around 100GB free disk space for this, and downloading the images may take around 24h. To increase efficiency, modify scripts under ./setup/ accordingly.

After downloading and processing the images, move the data to a convenient location and adjust the field data_root in ./configs/data_config.yaml.

However, it seems that this script downloads the entire dataset, not just the 5,000-image VL-Gender evaluation subset described in the paper. Is there a way to obtain or extract the exact subset used for VL-Gender evaluation?

2. I tried running setup_prompts.py to create prompts, but encountered a missing file error for occupation_probabilities.json. Would it be possible to share this file?

Thank you in advance!

[LGirrbach]

Hi, thank you for your interest in our work!

Regarding your questions:

1. There is no functionality to only download the images used in our experiments, but BiasPromptIterator in bias_eval_utils.py uses the first 500 male/female-gendered images per dataset (after filtering for occupation-related information), as extracted by pd.DataFrame.head(). You can check the function select_images in the same file for details. So, after downloading and processing the datasets, the subset can be extracted by using these images from the annotations.csv files. I also validated once on my local system that the ordering of files after downloading is deterministic.
2. I added the missing file in the ./data/ folder.

I hope this is helpful to you. Let me know if you have further questions!

[lussssep]

Hello, I'd also like to know if there's a direct link to access the dataset of 5,000 images you used for fine-tuning. thank you.

[LGirrbach]

Hi,

Thank you for your interest in our work and for the question!

Unfortunately, we cannot share the images publicly because of the different licenses of the original datasets. However, we include detailed instructions and scripts to reconstruct our dataset here.

Are you mainly interested in all images, to use for debiasing or mainly in the 5000 images we used for the benchmark?

[lussssep]

Thanks for your reply. I think I am mainly interested in the data you used to debias each model and the data used to evaluate the effect of dibias.

[LGirrbach]

Hi, if you want to train the models, I would recommend downloading the full dataset, then you can freely choose the number of images you're using during training.

For the benchmark images, you can contact me privately. Thank you.