Enable Choice of NVML Device

[ranocha]

Up to now, the NVML device is hard coded in

toolkitICL/src/main.cpp

Line 452 in 1160e77

nvmlDeviceGetHandleByIndex(0, &device);

and

toolkitICL/src/main.cpp

Line 481 in 1160e77

nvmlDeviceGetHandleByIndex(0, &device);

We should add some command line option to choose another device or even other devices. A somewhat simple option would be to allow logging on only one device. However, I would prefer the ability to enable logging on an arbitrary number of devices.

As described at here, it would be better to use nvmlDeviceGetHandleByUUID or nvmlDeviceGetHandleByPciBusId.

CC @Kostaszki

[ranocha]

Another option would be to log the power/temperature of all devices, similar to the approach for Intel (packages 0 and 1).

[Kostaszki]

When logging the power/temperature of all device you still need an option to correlate the used OpenCL device with the NVML device. Considering this I would prefer the command line option.

[ranocha]

In that case, one possibility might be to query the UUID via

$ nvidia-smi -L
GPU 0: GeForce GTX 1070 Ti (UUID: GPU-7350c62a-efab-c59a-a51f-f99f19ccbf6b)

Then, we can have the general calling syntax toolkitICL -d 0 -nvidia_power 100 [optional uuids] -c config.h5.

1. If no UUID is specified, we can/should log all devices. If we consider the current behavior as a bug, that's okay for a new release. Otherwise, we would have to go to version 2.0.0 if we change this behavior.
2. If at least one UUID is specified, the devices having these UUIDs should be used for logging.

We can use names such as power0, power1 to enumerate the devices (in the order used by nvml in case 1 or in the given order in case 2). The UUID (and possibly other data) could be added to the description.

[ranocha]

In #39, @philipheinisch implemented a sensible default value for nvml. Maybe we want to enable additional logging of specificlly chosen devices for a more general power logging library?