eth_mnist.py expected runtime on GPU?

[0xnurl]

Hi,
I'm running eth_mnist.py out of the box (in training mode) on a machine with GeForce RTX 2080 Ti.

The expected running time according to the tqdm progress bar is ~12 hours.
Is this the expected running time for this model? The running time that I get for the same script on a CPU is ~17 hours.

nvidia-smi gives this output:

+-------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce ...  Off  | 00000000:61:00.0 Off |                  N/A |
|  0%   45C    P2    64W / 250W |    248MiB / 11264MiB |     19%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

...

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     34378      C   python                           2019MiB |

[Hananel-Hazan]

Try to disable the plotting option, that can improve the training time.

[0xnurl]

Thanks so much! Indeed this lowers the runtime to ~6 hours!

This is strange though, since the only place where --plot is ever used is the condition that occurs only every update_interval here:
https://github.com/BindsNET/bindsnet/blob/master/examples/mnist/eth_mnist.py#L253

How come this has such a dramatic effect on the running time?

[0xnurl]

Also, this dramatically reduces the running time on a CPU as well, from ~17 hours to ~7 hours.
So the GPU effectiveness is still quite limited.

[Hananel-Hazan]

Yes, the default network size in eth_mnist.py is too small for the GPU to be effective. Increase the network size and you will see the difference.

Plotting in python is not very efficient, also in case of GPU, the data needed to be fetched from the GPU memory to CPU memory and needed to be converted to numpy, that takes time.

For the best GPU utilization, check out the batch_eth_mnist.py

[0xnurl]

Thank you!