This is the official implementation of paper FETA-Pro. FETA-Pro proposes a spatial-frequency curriculum, which leverages frequency domain features to capture global structures and textures of images, complementing spatial domain features. To address the challenges of learning caused by heterogeneity between spatial and frequency domain features, FETA-Pro extracts frequency domain features using the Fourier Transformer under DP, and introduces an auxiliary generator to produce images aligned the noisy features. Then, DP synthesizers use synthetic images for warm-up.
We conduct experiments to show that across five sensitive image datasets, FETA-Pro achieves, on average, 24.5% higher fidelity and 2.8% greater utility compared to the state-of-the-art method under a privacy budget of
The framework of FETA-Pro. During warm-up, FETA-Pro extracts spatial features to train the synthesizer. Then, FETA-Pro introduces auxiliary generators to generate images aligning the noisy frequency features, and training synthesizers on these synthetic images. Then, we train warm-up synthesizers on sensitive images using DP-SGD.
We list baselines as follows.
We list the studied datasets as follows in our paper, which include five sensitive datasets.
| Usage | Dataset |
|---|---|
| Sensitive dataset | MNIST, FashionMNIST, CIFAR-10, CelebA, Camelyon |
This repository is built upon DPImageBench. Users can find more details about its functions in their Repository.
Below is the directory structure of the DPImageBench project, which organizes its two core functionalities within the models/ and evaluation/ directories. To enhance user understanding and showcase the toolkit's ease of use, we offer a variety of example scripts located in the scripts/ directory.
DPImageBench/
├── configs/ # Configuration files for various DP image synthesis algorithms
├── data/ # Data Preparation for Our Benchmark
├── dataset/ # Datasets studied in the project
├── docker/ # Docker file
├── exp/ # The output of the training process and evaluation results.
├── evaluation/ # Evaluation module of DPImageBench, including utility and fidelity
│ ├── classifier/ # Downstream tasks classification training algorithms
│ │ ├── densenet.py
│ │ ├── resnet.py
│ │ ├── resnext.py
│ │ └── wrn.py
│ ├── ema.py
│ └── evaluator.py
├── models/ # Implementation framework for DP image synthesis algorithms
├── opacus/ # Implementation of DPSGD
├── results_demo/ # Demo of experimental results
├── plot/ # Figures and plots in our paper
│ ├── ablation.py # Plotting for Figure 5 in RQ2
│ ├── fid_curve.py # Plotting for Figure 4 in RQ1
│ ├── plot_eps_changes.py # Plotting for Figure 7 in RQ3
│ ├── plot_heatmap.py # Plotting for Figure 6 in RQ3
│ ├── visualization_tf.py # Plotting for Figure 2
│ └── visualization.py # Plotting for Figure 3 in RQ1
├── scripts/ # Scripts for using DPImageBench
│ └── script-feta-pro.sh
├── utils/ # Helper classes and functions supporting various operations
│ └── utils.py
├── README.md # Main project documentation
├── cal_privacy.py # Calculate the privacy budget ratio in RDP
└── requirements.txt # Dependencies required for the project
Clone repo and setup the environment:
git clone https://github.com/2019ChenGong/Feta-Pro.git
cd Feta-Pro
bash install.sh
We also provide the Docker file.
conda activate dpimagebench
bash data_preparation.sh
After running, we can found the folder dataset:
dataset/
├── camelyon/
├── celeba/
├── cifar10/
...
If you want to quickly test your implementation or perform preliminary experiments, you can run:
bash data_preparation_quick.shThis will only prepare MNIST and Fashion-MNIST, allowing you to validate your pipeline, hyperparameters, or DP configurations in minutes rather than hours.
The training and evaluatin codes are run.py and eval.py.
The core codes of run.py are present as follows.
We provide an example of training a synthesizer using the DP-FETA-Pro method with 4 GPUs.
python run.py setup.n_gpus_per_node=4 --method DP-FETA-Pro --data_name mnist_28 -e 1.0 eval.mode=val
def main(config):
initialize_environment(config)
model, config = load_model(config)
sensitive_train_loader, sensitive_val_loader, sensitive_test_loader, public_train_loader, config = load_data(config)
model.pretrain(public_train_loader, config.pretrain)
model.train(sensitive_train_loader, config.train)
syn_data, syn_labels = model.generate(config.gen)
evaluator = Evaluator(config)
evaluator.eval(syn_data, syn_labels, sensitive_train_loader, sensitive_val_loader, sensitive_test_loader)We list the key hyper-parameters below, including their explanations and available options.
--data_name(-dn): means the sensitive dataset; the option is [mnist_28,fmnist_28,cifar10_32,celeba_male_32,camelyon_32].--method(-m): the method to train the DP image synthesizers; the option is [DP-NTK,DP-Kernel,DP-MERF,DPGAN,DP-LDM-SD,DP-LDM,DP-LORA,DPDM,PE,GS-WGAN,PDP-Diffusion,PrivImage,DP-FETA,DP-FETA-Pro].--epsilon(-e): the privacy budget 10.0; the option is [1.0,10.0].--exp_description(-ed): the notes for the name of result folders.setup.n_gpus_per_node: means the number of GPUs to be used for training.pretrain.cond: specifies the mode of pretraining. The options are [true,false], wheretrueindicates conditional pretraining andfalseindicates conditional pretraining.public_data.name: the name of pretraining dataset; the option is [null,imagenet], which mean that without pretraining, using ImageNet dataset as pretraining dataset, and using Places365 as pretraining dataset. It is notice that DPImageBench uses ImageNet as default pretraining dataset.eval.mode: the mode of evaluations; the option is [val,syn] which means that using part of sensitive images and directly using the synthetic images as the validation set for model selection, respectively. The default setting isval.setup.master_port: a configuration parameter specifying the port number on the master node (or primary process) that other processes or nodes use to communicate within a distributed system.pretrain.n_epochs: the number of epoch for pretraining.train.n_epochs: the number of epoch for finetuning on sensitive datasets.train.dp.n_split: the number of gradient accumulations for saving GPU memory usage.train.sigma_freq: the noise scale of frequency domain features.train.sigma_time: the noise scale of time domain features.train.accountant: the privacy budget accounting mehtods.train.accountant=prvmeans usingprvto account the privacy budget. The default setting isrdp.
Tip
Experiments such as pretraining or using DPSGD require significant computational resources. We recommend to use 4 NVIDIA GeForce A6000 Ada GPUs and 512GB of memory. Here are some tips to help users efficiently reduce computational resource usage and running time in an appropriate way:
- Reduce
pretrain.n_epochsandtrain.n_epochs: Reducing the number of pretraining and fine-tuning steps can decrease running time but may also impact the performance of synthetic images. - Increase
train.dp.n_split: Increasingtrain.dp.n_splitenables jobs to run even when GPU memory is insufficient. However, this adjustment will lead to longer running times.
Warning
It is a common issue that we can not run a distributed process under a setup.master_port=6026. If you intend to run multiple distributed processes on the same machine, please consider using a different setup.master_port, such as 6027.
Users should first activate the conda environment.
conda activate dpimagebench
We list an example as follows. Users can modify the configuration files in configs as their preference.
The results reported in Table 3 were obtained by following the instructions provided. We provide an example of training a synthesizer using the DP-FETA-Pro method with 4 GPUs.
python run.py setup.n_gpus_per_node=4 --method DP-FETA-Pro --data_name mnist_28 -e 1.0 eval.mode=val
We provide more examples in the scripts/script-feta-pro.sh, please refer to scrips.
Besides, if users want to directly evaluate the synthetic images,
python eval.py --method DP-FETA-Pro --data_name mnist_28 --epsilon 10.0 --exp_path exp/dp-feta-pro/<the-name-of-file>
The results are recorded in exp/pdp-diffusion/<the-name-of-file>/stdout.txt. Besides, these files also record the IS, Precision, Recall, and FLD of our method and seven baselines, as shown in Table 10.
For Figure3 and Figure4, please refer to plot/visualization.py and plot/fid_curve.py and change the log_files in codes.
For baselines, readers can select the options -m: [DP-NTK, DP-Kernel, DP-MERF, DPGAN, DPDM, GS-WGAN, DP-FETA, DP-FETA-Pro], refering to implementations in DPImageBench.
python run.py setup.n_gpus_per_node=4 setup.master_port=6662 eval.mode=val -m DPDM -dn mnist_28 -e 1.0 -ed dpdm
The train.accountant=prv means using prv to account the privacy budget. To obtain results in Appendix E.3, Table 13, please run the following code,
python run.py setup.n_gpus_per_node=3 setup.master_port=6662 train.accountant=prv eval.mode=val -m DP-FETA-Pro -dn mnist_28 -e 1.0 -ed val_test_prv
We also provide the checkpoints of FETA-Pro
In RQ2, to investigate the benifits of frequency features, in Figure 5, we compare the performance of DP-FETA-Pro with three invariants,
- `FETA-Pro_{ft}' denotes a synthesizer that prioritizes learning spatial domain knowledge from sensitive datasets before acquiring frequency domain knowledge.
- `FETA-Pro_{mix}' means the synthesizers simultaneously learn the spatial and frequency domain knowledge.
- `FETA-Pro_{f}' means the synthesizers solely learn the frequency domain knowledge for warm-up.
Users can set the pretrain.mode=[freq_time, mix, freq] to choose the invariants. We provide example as follows,
python run.py setup.n_gpus_per_node=4 setup.master_port=6662 eval.mode=val pretrain.mode=mix -m DP-FETA-Pro -dn mnist_28 -e 1.0 -ed mix
To investigate the benifits of auxiliary generator, in Table 5, we compare the performance of DP-FETA-Pro with two invariants,
- `FETA-Pro-No-Auxiliary' means directly warming up synthesizers on frequency domain features.
- `FETA-Pro-DM-Auxiliary' means using diffusion models as the auxiliary generator.
Users can set the pretrain.mode=[no_auxiliary, dm_auxiliary] to choose the invariants. We provide example as follows,
python run.py setup.n_gpus_per_node=4 setup.master_port=6662 eval.mode=val pretrain.mode=no_auxiliary -m DP-FETA-Pro -dn mnist_28 -e 1.0 -ed no_auxiliary
To obtain the results in Figure 6, users can set the train.sigma_freq and train.sigma_time to change the privacy budget allocation plans. The algorithm will try different noise scales for DP-SGD to obtain the corresponding privacy budget settings (controlled by -e).
train.sigma_freq: the noise scale of frequency domain features.train.sigma_time: the noise scale of time domain features.
python run.py setup.n_gpus_per_node=4 --method DP-FETA-Pro --data_name mnist_28 -e 1.0 eval.mode=val train.sigma_freq=26.6 train.sigma_time=20
For results in Figure 7, users can adjust -e, train.sigma_freq and train.sigma_time, to run DP-FETA-Pro with varying privacy budget.
For example,
python run.py setup.n_gpus_per_node=4 --method DP-FETA-Pro --data_name mnist_28 -e 10.0 eval.mode=val train.sigma_freq=7.4 train.sigma_time=5
For the results in Table 12, we can run cal_privacy.py to obtain the DP cost ratios of spatial feature / frequency feature / DP-SGD in FETA-Pro based on train.sigma_freq, train.sigma_time, and epsilon.
For example,
python cal_privacy.py setup.n_gpus_per_node=1 --method DP-FETA-Pro --data_name mnist_28 -e 1.0 eval.mode=val train.sigma_freq=26.6 train.sigma_time=20
You will get the DP cost ratios in the terminal,
RDP cost ratio of time, frequency, and dpsgd: 0.3% / 2.79% / 96.91%
Test the classification algorithm on the sensitive images without DP.
python ./scripts/test_classifier.py --data_name mnist_28 --epsilon 10.0 -ed no-dp-mnist_28
The results are recorded in exp/pdp-diffusion/<the-name-of-file>no-dp-mnist_28/stdout.txt. This process is independent of --method and uses of --epsilon.
Note
If users wish to combine warm-up training in FETA-Pro with other methods using public images, you should set the pretrain.mode=time_freq.
For example,
python run.py setup.n_gpus_per_node=3 pretrain.mode=time_freq --method PDP-Diffusion --data_name mnist_28 -e 10.0 eval.mode=val
We also support training synthesizers from the checkpoints. If users wish to finetune the synthesizers using pretrained models, they should load the pretrained synthesizers through model.ckpt. For example, the pretrained synthesizer can be sourced from other algorithms. Readers can refer to the file structure for more details about loading pretrained models like
python run.py setup.n_gpus_per_node=3 eval.mode=val \
model.ckpt=./exp/pdp-diffusion/<the-name-of-scripts>/pretrain/checkpoints/snapshot_checkpoint.pth \
--method DP-FETA-Pro --data_name fmnist_28 --epsilon 10.0 --exp_description <any-notes>
For CelebA64, run:
python run.py setup.n_gpus_per_node=3 eval.mode=val \
--method DP-FETA-Pro --data_name celeba_male_64 --epsilon 10.0 --exp_description <any-notes>
For CelebA128, run:
python run.py setup.n_gpus_per_node=3 eval.mode=val \
--method DP-FETA-Pro --data_name celeba_male_128 --epsilon 10.0 --exp_description <any-notes>
We can find the stdout.txt files in the result folder, which record the training and evaluation processes. The results for utility and fidelity evaluations are available in stdout.txt. The result folder name consists of <data_name>_eps<epsilon><notes>-<starting-time>, e.g., mnist_28_eps1.0-2024-10-25-23-09-18.
We outline the structure of the results files as follows. The training and evaluations results are recorded in the file exp. For example, if users leverage the DP-FETA-Pro method to generate synthetic images for the MNIST dataset under a privacy budget of eps=1.0, the structure of the folder is as follows:
exp/
├── dp-feta-pro/
│ └── mnist_28_eps1.0-2025-07-29-13-35-18/
│ ├── gen
│ │ ├── gen.npz
│ │ └── sample.png
│ ├── gen_freq
│ │ ├── gen.npz
│ │ └── sample.png
│ ├── pretrain_time
│ │ ├── checkpoints
│ │ │ ├── final_checkpoint.pth
│ │ │ └── snapshot_checkpoint.pth
│ │ └── samples
│ │ ├── iter_2000
│ │ └── ...
│ ├── pretrain_freq
│ │ ├── checkpoints
│ │ │ ├── final_checkpoint.pth
│ │ │ └── snapshot_checkpoint.pth
│ │ └── samples
│ │ ├── iter_2000
│ │ └── ...
│ ├── train
│ │ ├── checkooints
│ │ │ ├── final_checkpoint.pth
│ │ │ └── snapshot_checkpoint.pth
│ │ └── samples
│ │ ├── iter_2000
│ │ └── ...
│ ├── train_freq
│ │ ├── checkooints
│ │ │ ├── final_checkpoint.pth
│ │ │ └── noisy_emb.pt
│ └──stdout.txt
├── pe/
└── privimage/
We introduce the files as follows,
./gen/gen.npz: the synthetic images../gen/sample.png: the samples of synthetic images../gen_freq/gen.npz: the synthetic images from the auxiliary generator using frequency features../gen_freq/sample.png: the sample of synthetic images from the auxiliary generator../pretrain_time/checkpoints/final_checkpoint.pth: the parameters of synthsizer pretrained using the time feature at the final epochs../pretrain_time/checkpoints/snapshot_checkpoint.pth: we store the synthesizer's parameters at the current epoch after each iteration, deleting the previous parameters to manage storage efficiently../pretrain_time/samples/iter_2000: the synthetic images under 2000 iterations for pretraining on the time feature../pretrain_freq/checkpoints/final_checkpoint.pth: the parameters of synthsizer pretrained using the frequency feature at the final epochs../pretrain_freq/checkpoints/snapshot_checkpoint.pth: we store the synthesizer's parameters at the current epoch after each iteration, deleting the previous parameters to manage storage efficiently../pretrain_freq/samples/iter_2000: the synthetic images under 2000 iterations for pretraining on the frequency feature../train/checkpoints/final_checkpoint.pth: the parameters of synthsizer at the final epochs../train/checkpoints/snapshot_checkpoint.pth: we store the synthesizer's parameters at the current epoch after each iteration, deleting the previous parameters to manage storage efficiently../train/samples/iter_2000: the synthetic images under 2000 iterations for training on sensitive datasets../train_freq/checkpoints/final_checkpoint.pth: the parameters of the auxiliary generator using frequency features../train_freq/checkpoints/noisy_emb.pt: the noisy frequency feature../stdout.txt: the file used to record the training and evaluation results.
In utility evaluation, after each classifier training, we can find,
INFO - evaluator.py - 2024-11-12 05:54:26,463 - The best acc of synthetic images on sensitive val and the corresponding acc on test dataset from wrn is 64.75999999999999 and 63.99
INFO - evaluator.py - 2024-11-12 05:54:26,463 - The best acc of synthetic images on noisy sensitive val and the corresponding acc on test dataset from wrn is 64.75999999999999 and 63.87
INFO - evaluator.py - 2024-11-12 05:54:26,463 - The best acc test dataset from wrn is 64.12
These results represent the best accuracy achieved by: (1) using the sensitive validation set (63.99%), (2) adding noise to the validation results of the sensitive dataset (model.eval = val), and the accuracy is 63.87%, and (3) using the sensitive test set for classifier selection (64.12%).
The following results can be found at the end of the log file:
INFO - evaluator.py - 2024-11-13 21:19:44,813 - The best acc of accuracy (adding noise to the results on the sensitive set of validation set) of synthetic images from resnet, wrn, and resnext are [61.6, 64.36, 59.31999999999999].
INFO - evaluator.py - 2024-11-13 21:19:44,813 - The average and std of accuracy of synthetic images are 61.76 and 2.06
INFO - evaluator.py - 2024-11-13 21:50:27,195 - The FID of synthetic images is 21.644407353392182
INFO - evaluator.py - 2024-11-13 21:50:27,200 - The Inception Score of synthetic images is 7.621163845062256
INFO - evaluator.py - 2024-11-13 21:50:27,200 - The Precision and Recall of synthetic images is 0.5463906526565552 and 0.555840015411377
INFO - evaluator.py - 2024-11-13 21:50:27,200 - The FLD of synthetic images is 7.258963584899902
The first line shows the accuracy of the downstream task when noise is added to the validation results of the sensitive dataset for classifier selection (model.eval = val), across three studied classification outcomes.
The synthetic images can be found at the ./exp/<algorithm_name>/<file_name>/gen/gen.npz.
We provide the plotting codes for results visualization in the folder plot.
ablation.py: plotting for Figure 5 in RQ2.fid_curve.py: plotting for Figure 4 in RQ1.plot_eps_changes.py: plotting for Figure 7 in RQ3.plot_heatmap.py: plotting for Figure 6 in RQ3.visualization_tf.py: plotting for Figure 2.visualization.py: plotting for Figure 3 in RQ1.
If you have any question about our work or this repository, please don't hesitate to contact us by emails or open an issue under this project.
-
Chen Gong (ChenG_abc@outlook.com)
-
Kecen Li (sunameizing@gmail.com)
@article{gong2026easy,
title={From Easy to Hard++: Promoting Differentially Private Image Synthesis Through Spatial-Frequency Curriculum},
author={Gong, Chen and Li, Kecen and Lin, Zinan and Wang, Tianhao},
journal={arXiv preprint arXiv:2601.06368},
year={2026}
}
Part of code is borrowed from DPImageBench. We sincerely thank them for their contributions to the community.