Use tensor cores to ensure max power consumption

[sbastrakov]

Nice tool, thanks for making it public!

Just ran it on my 8x H100 system from your container ghcr.io/huggingface/gpu-fryer:latest, getting similar results of around 51500 GFlop/s. From both the performance number and nvidia-smi dmon --gpm-metrics 5 I see that tensor cores were not used at all in that run. Since your README has ~51 TFLOP/s I think this (no tensor cores) is also the case for your runs.

This also means GPU is not used in max power mode, my H100s were all in 580 - 650W range throughout the run, out of 700W max power. Suggest making sure tensor cores are used not for the sake of a better performance number, but to ensure the GPUs are tested in the max power mode and potentially discover more issues

[Hugoch]

Hey @sbastrakov ! gpu-fryer burns using FP32 as a default for broad compatibility. We achieve near max power utilization on our H100 system, but makes sense to use Tensor Core if available.

PR #5 adds support for BF16 which is automatically detected or can be forced.

Tensor Core are correctly used as shown below. Can you check that it works as expected on your system?

[sbastrakov]

Wow that was quick.

When using the branch from PR #5 on my H100s I see high tensor core utilisation (81% mmaact) and around 690W. Performance is also roughly matching your numbers in that PR.

And the clock behaviour is more "realistic" (of course depends on perspective and use case) with a more pronounced drop to 1620 while the workload is running, as opposed to a no or minimal drop for the fp32 version. This was my main motivation to open this issue and what I meant by "GPU not used in max power mode" there.

[Hugoch]

Nice! Thx for the feedback, merging!