notebooks/art-for-tensorflow-v2-keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ART for TensorFlow v2 - Keras API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook demonstrate applying ART with the new TensorFlow v2 using the Keras API. The code follows and extends the examples on www.tensorflow.org."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "import tensorflow as tf\n",
    "tf.compat.v1.disable_eager_execution()\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "from art.estimators.classification import KerasClassifier\n",
    "from art.attacks.evasion import FastGradientMethod, CarliniLInfMethod"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "if tf.__version__[0] != '2':\n",
    "    raise ImportError('This notebook requires TensorFlow v2.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load MNIST dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0\n",
    "\n",
    "x_test = x_test[0:100]\n",
    "y_test = y_test[0:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow with Keras API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a model using Keras API. Here we use the Keras Sequential model and add a sequence of layers. Afterwards the model is compiles with optimizer, loss function and metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.models.Sequential([\n",
    "    tf.keras.layers.InputLayer(input_shape=(28, 28)),\n",
    "    tf.keras.layers.Flatten(),\n",
    "    tf.keras.layers.Dense(128, activation='relu'),\n",
    "    tf.keras.layers.Dropout(0.2),\n",
    "    tf.keras.layers.Dense(10, activation='softmax')\n",
    "])\n",
    "\n",
    "model.compile(optimizer='adam',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy']);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit the model on training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples\n",
      "Epoch 1/3\n",
      "60000/60000 [==============================] - 3s 46us/sample - loss: 0.2968 - accuracy: 0.9131\n",
      "Epoch 2/3\n",
      "60000/60000 [==============================] - 3s 46us/sample - loss: 0.1435 - accuracy: 0.9575\n",
      "Epoch 3/3\n",
      "60000/60000 [==============================] - 3s 46us/sample - loss: 0.1102 - accuracy: 0.9664\n"
     ]
    }
   ],
   "source": [
    "model.fit(x_train, y_train, epochs=3);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate model accuracy on test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on test data: 100.00%\n"
     ]
    }
   ],
   "source": [
    "loss_test, accuracy_test = model.evaluate(x_test, y_test)\n",
    "print('Accuracy on test data: {:4.2f}%'.format(accuracy_test * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a ART Keras classifier for the TensorFlow Keras model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier = KerasClassifier(model=model, clip_values=(0, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fast Gradient Sign Method attack"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a ART Fast Gradient Sign Method attack."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "attack_fgsm = FastGradientMethod(estimator=classifier, eps=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate adversarial test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_test_adv = attack_fgsm.generate(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate accuracy on adversarial test data and calculate average perturbation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on adversarial test data: 0.00%\n",
      "Average perturbation: 0.18\n"
     ]
    }
   ],
   "source": [
    "loss_test, accuracy_test = model.evaluate(x_test_adv, y_test)\n",
    "perturbation = np.mean(np.abs((x_test_adv - x_test)))\n",
    "print('Accuracy on adversarial test data: {:4.2f}%'.format(accuracy_test * 100))\n",
    "print('Average perturbation: {:4.2f}'.format(perturbation))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualise the first adversarial test sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQEAAAECCAYAAAD+eGJTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAQB0lEQVR4nO3dfYxc5XXH8d/BODYs0NhssQ1Z10AdKQ5VTLICUreRK6uB4j8AtVS1KuqmUU0rUIFGtAhVAqWqhKqAoVXr1tRuTARUSLy2Rg2uFYVWKW5sYoHNUsyLAeOVjdmAXyqMvZz+sZdkY888d3bu3Jfd8/1IaHfvmZl79s74x515nn2uubsAxHVK3Q0AqBchAARHCADBEQJAcIQAEBwhAARXSwiY2RVm9r9m9qqZ3VZHDylmttvMXjSz7Wa2tQH9rDez/Wa2Y9y22Wa2ycx2ZV9nNay/O83snewYbjezK2vsb8DMvmdmQ2a208xuyrY34hgm+qvkGFrV8wTMbJqkVyT9uqQ9kn4oaYW7v1RpIwlmtlvSoLsfqLsXSTKzr0g6LOkBd78o2/bXkkbc/a4sSGe5+583qL87JR1292/V0dN4ZjZP0jx3f97MzpS0TdLVkn5fDTiGif5+WxUcwzrOBC6R9Kq7v+7uH0n6F0lX1dDHpOHuz0oaOWHzVZI2ZN9v0NiLphZt+msMdx929+ez7w9JGpJ0nhpyDBP9VaKOEDhP0tvjft6jCn/hDrmkZ8xsm5mtqruZNua4+7A09iKSdE7N/bRyo5m9kL1dqO3tynhmtkDSxZK2qIHH8IT+pAqOYR0hYC22NW3u8hJ3/6Kk35B0Q3a6i4lZI+lCSYslDUu6u952JDM7Q9Kjkm5294N193OiFv1VcgzrCIE9kgbG/fwZSXtr6KMtd9+bfd0v6XGNvYVpmn3Ze8lP3lPur7mfn+Hu+9x91N0/lnS/aj6GZjZdY//AHnT3x7LNjTmGrfqr6hjWEQI/lLTQzM43s09J+h1JT9XQR0tm1pd9OCMz65P0VUk70veqxVOSVmbfr5T0ZI29nOSTf1yZa1TjMTQzk7RO0pC73zOu1Ihj2K6/qo5h5aMDkpQNddwraZqk9e7+V5U30YaZXaCx//tL0qmSHqq7PzN7WNJSSf2S9km6Q9ITkh6RNF/SW5KudfdaPpxr099SjZ3GuqTdkq7/5P13Df39iqT/lPSipI+zzbdr7H137ccw0d8KVXAMawkBAM3BjEEgOEIACI4QAIIjBIDgCAEguFpDoMFTciXRX1FN7q/JvUnV9lf3mUCjnwjRX1FN7q/JvUkV9ld3CACoWaHJQmZ2haT7NDbz75/c/a7U7T9lM3ym+n7y8zEd1XTN6Hr/vTZ6dt/P/Hz8wyM6dWZfm1ufbNp7Ryb0+EVNtL88ef1PVK+f314ev1bHrujz18vj1+rYFfn9jx4e0fEPj7T64z2d2u2DZouD/J3GLQ5iZk+lFgeZqT5dasu63WXp3l/+5UL3//R3/rvUxy9bXv91K/v4FX3+yj5+RX7/lzaublsr8naAxUGAKaBICEyGxUEA5Oj67YA6XBwkG+pYJUkzdXqB3QEoQ5EzgY4WB3H3te4+6O6DTfoQEMCYIiHQ6MVBAHSm67cD7n7czG6U9F39dHGQnUWaef+6kj+dL/j4ecp+/LLl9V/38a17/3mafvzaKfKZgNz9aUlP96gXADVgxiAQHCEABEcIAMERAkBwhAAQHCEABFfpdQf6+gd80fJbKtvfRDV1HLdTk73/6Io+f6n7b/HNOugjLf+UmDMBIDhCAAiOEACCIwSA4AgBIDhCAAiOEACCK/SnxL1WdLXWouPgk30cfbL3j3pwJgAERwgAwRECQHCEABAcIQAERwgAwRECQHCNmidQdN32PPy9PcpU9PVV5jyX0Y3Pta1xJgAERwgAwRECQHCEABAcIQAERwgAwRECQHCVzhOY9t6R5Fhq0XHSMtdt70Tdj496Tdbnp1AImNluSYckjUo67u6DvWgKQHV6cSbwa+5+oAePA6AGfCYABFc0BFzSM2a2zcxW9aIhANUq+nZgibvvNbNzJG0ys5fd/dnxN8jCYZUkzdTpBXcHoNcKnQm4+97s635Jj0u6pMVt1rr7oLsPTteMIrsDUIKuQ8DM+szszE++l/RVSTt61RiAahR5OzBH0uNm9snjPOTu/96TrkpSdJy+7scve/91j3PXvd5D3fuvS9ch4O6vS/pCD3sBUAOGCIHgCAEgOEIACI4QAIIjBIDgCAEguEZdd6DpTjk9Pe35zT9dnKw/9PXVyfof/cVNyfpknwdQFPMwysGZABAcIQAERwgAwRECQHCEABAcIQAERwgAwZm7V7azvv4BX7T8lrb1useB88Zx3132UbJ+4cD+XrYzYQcO9yXrh4dmJeu7fm9Nof0vfOCPC92/bH17LVmf8zc/qKiT1sqcR/DSxtU6cuDtlgeAMwEgOEIACI4QAIIjBIDgCAEgOEIACI4QAIKrdJ7AWTbbL7Vlle2vanbx55N1/9HOQvd/95KzkvUPvvJhsn7+3PTFo1/f25+sX3BuuRefPno8vbzF8Hs/l6wvmPNesf3/7bxk/bQn/qfQ4xeVN48gNc9mi2/WQR9hngCAkxECQHCEABAcIQAERwgAwRECQHCEABDclLruQJFx1F748UXpcfxP/yh9/7x5BP059+//x3Q9zy/q7WT9lfWDyfpn/2Brof2fllO/IKe+59H0PIudX34wWb9y6LeS9dGc/dct9fof3fhc21rumYCZrTez/Wa2Y9y22Wa2ycx2ZV/Tq1UAaKxO3g58W9IVJ2y7TdJmd18oaXP2M4BJKDcE3P1ZSSMnbL5K0obs+w2Sru5xXwAq0u0Hg3PcfViSsq/n9K4lAFUq/YNBM1slaZUkzVT6gp4AqtftmcA+M5snSdnXtsvsuvtadx9098HpmtHl7gCUpdsQeErSyuz7lZKe7E07AKqWu56AmT0saamkfkn7JN0h6QlJj0iaL+ktSde6+4kfHp6k7vUE6p5H0HRlrnvfibzjf8rMmcn6R/+W/mjqjZfT6wWcuWtasj73vmLXJajz+KauO5D7mYC7r2hTmrqrgwCBMG0YCI4QAIIjBIDgCAEgOEIACI4QAIKbUusJFFV0HkHT5yHUPQ+gqDceWJisv7zoO8n65z/43WR95pb0ehBlK/P1Nc2PtK1xJgAERwgAwRECQHCEABAcIQAERwgAwRECQHCVzhMYPbtP7y9vP9ZZ9zh6UZO9/7LlHZ/jy76UrA+c3XYBq4585jfT13UoW9mvj9KuOwBgaiMEgOAIASA4QgAIjhAAgiMEgOAIASC4UOsJFB2nbfp6A01fL+CVNZck6+edfyBZPy3n8S8/d/EEO5pcirw+WE8AQFuEABAcIQAERwgAwRECQHCEABAcIQAEV+k8gWnvHUmOddY9jp73+HX+PfhUsOBJT9ZPvfVYsr7n+wPJ+ny9PeGeqlT289vt6zP3TMDM1pvZfjPbMW7bnWb2jpltz/67squ9A6hdJ28Hvi3pihbbV7v74uy/p3vbFoCq5IaAuz8raaSCXgDUoMgHgzea2QvZ24VZPesIQKW6DYE1ki6UtFjSsKS7293QzFaZ2VYz23pMR7vcHYCydBUC7r7P3Ufd/WNJ90tq++dh7r7W3QfdfXC6ZnTbJ4CSdBUCZjZv3I/XSNrR7rYAmi13noCZPSxpqaR+M9sj6Q5JS81ssSSXtFvS9SX2+BOTfRx9qq8XkPf7vXrvZcn63GPTk/X53/zBhHvqpbLnseQpsl5F6roDuSHg7itabF6Xdz8AkwPThoHgCAEgOEIACI4QAIIjBIDgCAEguFDXHUAxeePUo0u/mKxf8EvvJOt7/yO9XsAZej1Zr3seRdH9lz3PoB3OBIDgCAEgOEIACI4QAIIjBIDgCAEgOEIACM7c02vB99JZNtsvtWVd37/uceCimr6eQNH+jj6zIFn/8f+dlqzPvXooWZ/sz39RRZ6fLb5ZB33EWtU4EwCCIwSA4AgBIDhCAAiOEACCIwSA4AgBILhJtZ5AkXXXm6Dp/eV55Z+/lKy/cVF6Jfq3jh9O1q+97tYJ9zSVsJ4AgFoQAkBwhAAQHCEABEcIAMERAkBwhAAQ3KSaJzDZx9nrljcO/drdlyXrb1z+D4X2f/m6P0vWz1J1a1vgp3LPBMxswMy+Z2ZDZrbTzG7Kts82s01mtiv7Oqv8dgH0WidvB45L+oa7f07SZZJuMLNFkm6TtNndF0ranP0MYJLJDQF3H3b357PvD0kaknSepKskbchutkHS1WU1CaA8E/pg0MwWSLpY0hZJc9x9WBoLCknn9Lo5AOXrOATM7AxJj0q62d0PTuB+q8xsq5ltPaaj3fQIoEQdhYCZTddYADzo7o9lm/eZ2bysPk/S/lb3dfe17j7o7oPTNaMXPQPooU5GB0zSOklD7n7PuNJTklZm36+U9GTv2wNQtk7mCSyRdJ2kF81se7btdkl3SXrEzL4u6S1J1+Y90OjZfXp/OWP9dXnzm+lj/+qKNYUe/7PfX5msz3qdeQBF5M2T6XY9gtwQcPf/ktTyogWSur+SCIBGYNowEBwhAARHCADBEQJAcIQAEBwhAARn7tWN3Z5ls/1Saz+qyHoBxRya324kd8xLN/x9ocf/2lu/mqzvvexQss7zW5+XNq7WkQNvt3yBcCYABEcIAMERAkBwhAAQHCEABEcIAMERAkBwk+q6A3XL+3vtusfBv7biu6U+PvMAytXtegCdmOZH2tY4EwCCIwSA4AgBIDhCAAiOEACCIwSA4AgBILhK5wk0/boDZY7T9sL+JaPJ+q2zXyt1/6/ee1my3r+t1N1PeUWvK5C6/+jG59rWOBMAgiMEgOAIASA4QgAIjhAAgiMEgOAIASC43OsOmNmApAckzZX0saS17n6fmd0p6Q8lvZvd9HZ3fzr1WHnXHchT1vXZe7X/sl1609ZkfeiDucn6ps/9a6H9X37u4mS97uMz2RVdryJ1/y2+WQd9pOV1BzqZLHRc0jfc/XkzO1PSNjPblNVWu/u3OngMAA2VGwLuPixpOPv+kJkNSTqv7MYAVGNCnwmY2QJJF0vakm260cxeMLP1Zjarx70BqEDHIWBmZ0h6VNLN7n5Q0hpJF0parLEzhbvb3G+VmW01s63HdLQHLQPopY5CwMymaywAHnT3xyTJ3fe5+6i7fyzpfkmXtLqvu69190F3H5yuGb3qG0CP5IaAmZmkdZKG3P2ecdvnjbvZNZJ29L49AGXrZHRgiaTrJL1oZtuzbbdLWmFmiyW5pN2Sri+lQwClyp0n0Et9/QO+aPktbetFx/nrHqcue56C//IX0vW/HEnWX9t5brK+8E+2JOtTXZl/z9/J/fOUNU+AGYNAcIQAEBwhAARHCADBEQJAcIQAEBwhAARX6TyBvPUEio7zN/26AUBKmfMMmCcAoC1CAAiOEACCIwSA4AgBIDhCAAiOEACCq3SegJm9K+nNcZv6JR2orIGJo79imtxfk3uTet/fL7j7z7cqVBoCJ+3cbKu7D9bWQA76K6bJ/TW5N6na/ng7AARHCADB1R0Ca2vefx76K6bJ/TW5N6nC/mr9TABA/eo+EwBQM0IACI4QAIIjBIDgCAEguP8HYccW/xF2T1AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.matshow(x_test_adv[0])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Carlini&Wagner Infinity-norm attack"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a ART Carlini&Wagner Infinity-norm attack."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "attack_cw = CarliniLInfMethod(classifier=classifier, eps=0.3, max_iter=100, learning_rate=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate adversarial test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C&W L_inf: 100%|██████████| 1/1 [00:04<00:00,  4.23s/it]\n"
     ]
    }
   ],
   "source": [
    "x_test_adv = attack_cw.generate(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate accuracy on adversarial test data and calculate average perturbation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on adversarial test data: 10.00%\n",
      "Average perturbation: 0.03\n"
     ]
    }
   ],
   "source": [
    "loss_test, accuracy_test = model.evaluate(x_test_adv, y_test)\n",
    "perturbation = np.mean(np.abs((x_test_adv - x_test)))\n",
    "print('Accuracy on adversarial test data: {:4.2f}%'.format(accuracy_test * 100))\n",
    "print('Average perturbation: {:4.2f}'.format(perturbation))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualise the first adversarial test sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQEAAAECCAYAAAD+eGJTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAP1UlEQVR4nO3df4xVdXrH8c8zMIIC8kMqIGJBSzdVNx3XKUq1G4ypUpNGyNZt6ZbSjV1su6Sabtoa2kS7SVPSrLrbJssGCxVddNesuNrEVCnZLDUqFSirICpUEZFZ0B0V6GaBmXn6xxzaKd75nuGee8+5M8/7lZC5c557znk4zHw4557v/V5zdwGIq63qBgBUixAAgiMEgOAIASA4QgAIjhAAgqskBMxsoZm9YWb7zOzuKnpIMbP9Zvaqme00s20t0M86MztiZrsGLJtiZpvMbG/2dXKL9Xevmb2XHcOdZnZLhf3NMrMfmNkeM9ttZndmy1viGCb6K+UYWtnjBMxslKQ3Jf26pIOSXpa0xN1fK7WRBDPbL6nT3T+ouhdJMrPPSjou6WF3vzJb9veSut19VRakk939L1uov3slHXf3r1XR00BmNkPSDHffYWYTJG2XtEjSH6gFjmGiv8+rhGNYxZnAPEn73P0tdz8p6TuSbq2gj2HD3bdI6j5j8a2S1meP16v/h6YSg/TXMty9y913ZI+PSdojaaZa5Bgm+itFFSEwU9K7A74/qBL/wkPkkp4zs+1mtrzqZgYxzd27pP4fIkkXVtxPLSvM7JXscqGyy5WBzGy2pKskbVULHsMz+pNKOIZVhIDVWNZqY5evc/fPSPoNSV/OTndxdlZLukxSh6QuSfdV245kZuMlPSHpLnc/WnU/Z6rRXynHsIoQOChp1oDvL5Z0qII+BuXuh7KvRyQ9qf5LmFZzOLuWPH1NeaTifv4fdz/s7r3u3ifpQVV8DM2sXf2/YBvcfWO2uGWOYa3+yjqGVYTAy5LmmtkcMztH0u9IerqCPmoys3HZizMys3GSbpK0K71WJZ6WtCx7vEzSUxX28gmnf7kyi1XhMTQzk7RW0h53v39AqSWO4WD9lXUMS787IEnZrY6vSxolaZ27/23pTQzCzC5V///+kjRa0qNV92dmj0laIGmqpMOS7pH0fUmPS7pE0gFJt7l7JS/ODdLfAvWfxrqk/ZLuOH39XUF/10v6d0mvSurLFq9U/3V35ccw0d8SlXAMKwkBAK2DEYNAcIQAEBwhAARHCADBEQJAcJWGQAsPyZVEf0W1cn+t3JtUbn9Vnwm09D+E6K+oVu6vlXuTSuyv6hAAULFCg4XMbKGkb6h/5N8/ufuq1PPPsTE+VuP+9/tTOqF2jal7/81Gf8W0cn+t3JvU+P5+pv/WST9R68179YdAPZODnG9T/Bq7sa79AajfVt+so95dMwSKXA4wOQgwAhQJgeEwOQiAHKMLrDukyUGyWx3LJWmsziuwOwDNUORMYEiTg7j7GnfvdPfOVn4hBoiqSAi09OQgAIam7ssBd+8xsxWSntX/TQ6yu2GdAShFkdcE5O7PSHqmQb0AqAAjBoHgCAEgOEIACI4QAIIjBIDgCAEgOEIACI4QAIIjBIDgCAEgOEIACI4QAIIjBIDgCAEguEJvJcYwYzUnm22Ytis+laz3vbY3vYG+3mT5o6Xzk/VJj7yY3j5q4kwACI4QAIIjBIDgCAEgOEIACI4QAIIjBIDgGCcwkrSNStdz7sMX5XvfTtatPf3j5qfS25/8ne3p9Sv++w9XnAkAwRECQHCEABAcIQAERwgAwRECQHCEABAc4wRGkorvg/uJE4XWH3X++eknzJyW3v/BHyfr3YuuTNYnfful9P7d0/VhqlAImNl+Scck9UrqcffORjQFoDyNOBO4wd0/aMB2AFSA1wSA4IqGgEt6zsy2m9nyRjQEoFxFLweuc/dDZnahpE1m9rq7bxn4hCwclkvSWJ1XcHcAGq3QmYC7H8q+HpH0pKR5NZ6zxt073b2zXWOK7A5AE9QdAmY2zswmnH4s6SZJuxrVGIByFLkcmCbpSeufy360pEfd/V8b0hVC6j16NP2EvHqOyj+XIO9zHyoah1B3CLj7W5J+uYG9AKgAtwiB4AgBIDhCAAiOEACCIwSA4AgBIDjmEzgLbeelhz2/82cdyfqjtz+QrP/RX9+ZrE/ckPN+9ybLe79/7n3+6Fp0PgLOBIDgCAEgOEIACI4QAIIjBIDgCAEgOEIACM68xHuX59sUv8ZuLG1/jbb3oauT9ctmHSmpk9o+OD4uWT++Z3Kyvvf3Vxfa/9yH/7jQ+kXl9d+x6k+S9Wn/8EIj22kpW32zjnp3zQkNOBMAgiMEgOAIASA4QgAIjhAAgiMEgOAIASA4xgk00LQX0++3Pzw//X57u+qKZP39eentf/zZnyXrc6anPzz6rUNTk/VLL2ruh0+f6ElPb9H1k4nJ+uxpPym2/3+ckayf+/3/KLT9KjFOAMCgCAEgOEIACI4QAIIjBIDgCAEgOEIACI5xAhgxDj6RHmexe/6GZP2WG34rWe99Y99Z99QqCo0TMLN1ZnbEzHYNWDbFzDaZ2d7sa3q2CgAtayiXAw9JWnjGsrslbXb3uZI2Z98DGIZyQ8Ddt0jqPmPxrZLWZ4/XS1rU4L4AlKTeFwanuXuXJGVfL2xcSwDK1PQPJDWz5ZKWS9JYpT/QE0D56j0TOGxmMyQp+zroNLvuvsbdO929s11j6twdgGapNwSelrQse7xM0lONaQdA2XIvB8zsMUkLJE01s4OS7pG0StLjZna7pAOSbmtmk4AktY0dm6xPn3gsWb904x3J+oSFo9LbH8bjBFJyQ8DdlwxSYtQPMAIwbBgIjhAAgiMEgOAIASA4QgAIjhAAgmv6sGEMHx8tnV9o/UmPvNigTmp7++G5yfrrlz+SrF/x8ReS9ekrdp91TyMBZwJAcIQAEBwhAARHCADBEQJAcIQAEBwhAATHOIFAio4DsL7mfkZFz41XJ+uzLhh0AitJ0qe3/m6yfvHnYo4DyMOZABAcIQAERwgAwRECQHCEABAcIQAERwgAwTFOIJDJj+9I1j/8/GeS9YkbXmpkO59w4s8/TNbPzVn/osWvNa6ZQDgTAIIjBIDgCAEgOEIACI4QAIIjBIDgCAEgOMYJBJI3DiBP3vv9R2/enqyfXPgryfq57T9O1g/+cFayfoneTdZRW+6ZgJmtM7MjZrZrwLJ7zew9M9uZ/bmluW0CaJahXA48JGlhjeUPuHtH9ueZxrYFoCy5IeDuWyR1l9ALgAoUeWFwhZm9kl0uTG5YRwBKVW8IrJZ0maQOSV2S7hvsiWa23My2mdm2UzpR5+4ANEtdIeDuh9291937JD0oaV7iuWvcvdPdO9s1pt4+ATRJXSFgZjMGfLtY0q7BngugteWOEzCzxyQtkDTVzA5KukfSAjPrkOSS9ku6o4k9ItN3fUeyfnRO3jvui8kbB5DnwMJRyfr0U+3J+iVffaHQ/lFbbgi4+5Iai9c2oRcAFWDYMBAcIQAERwgAwRECQHCEABAcIQAEx3wCw0jb8zuTdZt9bbLuZsn61C0Hk/WeZFXqXZCer+DST7+XrB/6t/R8AeP1Vk4HqAdnAkBwhAAQHCEABEcIAMERAkBwhAAQHCEABDeixgmMuuJTyXrv7jdK6qQaE7/9UqH188YB5K6/Mj0f7YfHxifrF//dMJ8voC09X4L6esvp4yxxJgAERwgAwRECQHCEABAcIQAERwgAwRECQHAjapzASB8HULU3//nqZP3tK9Mz0R/oOZ6sf0nXJ+ttEyYk633H09uXe7qe4+PfKzZfw6RHXiy0/2bhTAAIjhAAgiMEgOAIASA4QgAIjhAAgiMEgOBG1DiB0TOmJ+vdN8xO1idvTs9r39f9UbLup04m63ms/Zymbj/Pf92Xvg/+9s3fKrT9m9f+RbJ+SdvWZL170ZXJ+qhT6XEAE76b3v7HX7gmWc8bBzBc5Z4JmNksM/uBme0xs91mdme2fIqZbTKzvdnXyc1vF0CjDeVyoEfSV9z9lyRdK+nLZna5pLslbXb3uZI2Z98DGGZyQ8Ddu9x9R/b4mKQ9kmZKulXS+uxp6yUtalaTAJrnrF4YNLPZkq6StFXSNHfvkvqDQtKFjW4OQPMNOQTMbLykJyTd5e5Hz2K95Wa2zcy2ndKJenoE0ERDCgEza1d/AGxw943Z4sNmNiOrz5B0pNa67r7G3TvdvbNdYxrRM4AGGsrdAZO0VtIed79/QOlpScuyx8skPdX49gA021DGCVwnaamkV81sZ7ZspaRVkh43s9slHZB0W3NaHDqfMjFZP/+x9H3i3pz3mx/77fR99N5z0veRL3hqd3r9o0O+yqrLO1+dn6zvW7K60PZ/8YfLkvU5f1PscwUs59+ntz3nPn7O+nmf23Dqps5kvX3T9vT+W1RuCLj785IGO7o3NrYdAGVj2DAQHCEABEcIAMERAkBwhAAQHCEABDei5hPofe3NZP2ni+cl6+dtTI8jmPDd9H3k0TMvStZ7mjwO4N2/+tVk/fU//Gah7X/xwK8l63OW/KjQ9tU2Kllu/2n6Pv+4f9mR3v6Y9IhVP5Ee1t7+3Lb09ocpzgSA4AgBIDhCAAiOEACCIwSA4AgBIDhCAAhuWI0T+Ghp+v3weZr9+fA97x1KPyFv3vqc97vn+eKSZwutn+fQtceaun319SbLJ8el/88ae83lyfrROecm60V/Pmx0+tfJe3oKbb9ZOBMAgiMEgOAIASA4QgAIjhAAgiMEgOAIASC4YTVOoNn3+Zuu4DiAN7+Vng/h2SlrCm0/z76vpz934RfuSs+3UHScxAU7upP13t1vJOuTXkjPV1BUq44DyMOZABAcIQAERwgAwRECQHCEABAcIQAERwgAweWOEzCzWZIeljRdUp+kNe7+DTO7V9KXJL2fPXWluz/TrEazZtL1gvfhW91vdv5npfvPHQeQp+C/T944gFw58xVENZTBQj2SvuLuO8xsgqTtZrYpqz3g7l9rXnsAmi03BNy9S1JX9viYme2RNLPZjQEox1m9JmBmsyVdJen053WtMLNXzGydmU1ucG8ASjDkEDCz8ZKekHSXux+VtFrSZZI61H+mcN8g6y03s21mtu2U0p/1BqB8QwoBM2tXfwBscPeNkuTuh9291937JD0oqea7W9x9jbt3untnu9IfCAmgfLkhYGYmaa2kPe5+/4DlMwY8bbGkXY1vD0CzDeXuwHWSlkp61cx2ZstWSlpiZh2SXNJ+SXc0pUMATTWUuwPPS6p1g765YwJqGeHjAPK8/qfpefX1vZcLbf/mizoKrY/hiRGDQHCEABAcIQAERwgAwRECQHCEABAcIQAEN6w+dyA6e+FHyTr3+VEPzgSA4AgBIDhCAAiOEACCIwSA4AgBIDhCAAjOvMT36JvZ+5LeGbBoqqQPSmvg7NFfMa3cXyv3JjW+v59395+rVSg1BD6xc7Nt7t5ZWQM56K+YVu6vlXuTyu2PywEgOEIACK7qEFhT8f7z0F8xrdxfK/cmldhfpa8JAKhe1WcCACpGCADBEQJAcIQAEBwhAAT3P6rRd0KXaTIVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.matshow(x_test_adv[0, :, :])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}