weird FGSM accuracy on MNIST clean data

[chhyun]

I tried FGSM attack on MNIST clean dataset, and I got 49% accuracy,

which is too large compared to 6.4% [Madry, https://arxiv.org/pdf/1706.06083.pdf]

Am i missing something?

I'd like to ask if anyone else has done a FGSM attack against mnist, what performance you got?.

[ZhangYuef]

Hi @chhyun ,I am facing the same problem as you. I got too low accuracy in my case for FGSM (epsilon=0.1, 0.3):

# attack type: GradientSignAttack
# attack kwargs: loss_fn=CrossEntropyLoss()
#                eps=0.1
#                clip_min=0.0
#                clip_max=1.0
#                targeted=False
# data: mnist_test, 10000 samples
# model: MNIST LeNet5 standard training
# accuracy: 98.89%
# adversarial accuracy: 79.96%
# attack success rate: 20.04%

# attack type: GradientSignAttack
# attack kwargs: loss_fn=CrossEntropyLoss()
#                eps=0.3
#                clip_min=0.0
#                clip_max=1.0
#                targeted=False
# data: mnist_test, 10000 samples
# model: MNIST LeNet5 standard training
# accuracy: 98.89%
# adversarial accuracy: 0.98%
# attack success rate: 99.02%

My guess here is how the epsilon is calculated. Should we normalized epsilon as epsilon/255 ?

[chhyun]

Hi @chhyun ,I am facing the same problem as you. I got too low accuracy in my case for FGSM (epsilon=0.1, 0.3):

# attack type: GradientSignAttack
# attack kwargs: loss_fn=CrossEntropyLoss()
#                eps=0.1
#                clip_min=0.0
#                clip_max=1.0
#                targeted=False
# data: mnist_test, 10000 samples
# model: MNIST LeNet5 standard training
# accuracy: 98.89%
# adversarial accuracy: 79.96%
# attack success rate: 20.04%

# attack type: GradientSignAttack
# attack kwargs: loss_fn=CrossEntropyLoss()
#                eps=0.3
#                clip_min=0.0
#                clip_max=1.0
#                targeted=False
# data: mnist_test, 10000 samples
# model: MNIST LeNet5 standard training
# accuracy: 98.89%
# adversarial accuracy: 0.98%
# attack success rate: 99.02%

My guess here is how the epsilon is calculated. Should we normalized epsilon as epsilon/255 ?

Hi @ZhangYuef. I used 0.3 as epsilon to FGSM attack my natural trained MNIST model and got 49% adversarial accuracy.
It's somewhat strange to see such different results in two experiments using the same epsilon value.
How many epochs did you train and which checkpoint did you use for the result?

[Djmcflush]

Please dump full hyper parameters. The variance between your result and the expected is far beyond the margin of error.