FLOPs numbers are inconsistent

[dsuess]

Bug description

The FLOPs numbers in throughput.py for H100 and H200 GPUs have some inconsistencies that have caused me some confusion when switching between GPU types:

• all H200 and the H100 NVL GPU seem to be using sparse FLOPs numbers
• the H100 SXM and PCIE models seem to using dense FLOPs, see: https://resources.nvidia.com/en-us-gpu-resources/h100-datasheet-24306

I think it would be good to standardize on one and also document the behaviour as there's a factor 2 difference between the two.

What version are you seeing the problem on?

master

Reproduced in studio

No response

How to reproduce the bug

Error messages and logs

# Error messages and logs here please

Environment

Current environment

#- PyTorch Lightning Version (e.g., 2.6.0):
#- PyTorch Version (e.g., 2.5):
#- Python version (e.g., 3.12):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):

More info

No response

cc @ethanwharris

[gaurav0107]

I'd like to work on this. The fix standardizes all Hopper entries to dense (without structured sparsity) per-GPU values, matching the existing H100 SXM/PCIe convention. I'll have a PR ready shortly.

[deependujha]

Hi @dsuess, thanks for raising this.

Since most workloads today are still dense, it’s more intuitive to standardize on reporting dense FLOPs. In cases where specs only provide sparse FLOPs (which is common for Tensor Core numbers), we can approximate dense performance as:

dense_flops ≈ sparse_flops / 2

assuming 2:4 structured sparsity.

For documenting this behavior, the main ambiguity shows up in MFU (Model FLOP Utilization). To make this explicit and user-controlled, one simple approach could be to extend Throughput with:

using_sparse_model: Optional[bool] = None  # User intent
sparse_cuda_acceleration_factor: float = 2.0  # Default for 2:4 sparsity

Behavior:

• If using_sparse_model is None, we issue a rank_zero_warn("MFU assumes dense model FLOPs (no sparsity acceleration). Set 'using_sparse_model=True' for mfu to use sparse flops.")
• If using_sparse_model=True, we scale FLOPs using sparse_cuda_acceleration_factor.
• If False, we use dense FLOPs directly.

This keeps the default safe while still allowing users with sparsity-aware workloads to opt in explicitly.

Curious what you think, especially if you see edge cases where this assumption might break down.

[dsuess]

@deependujha sounds like a plan! I don't think structured sparsity is sth you'd regularly encounter during training, so defaulting to dense is the way to go IMO

[reallyticsai]

I've encountered similar issues with FLOPs inconsistencies when working with different GPU architectures in our production deployments. We've had to standardize on dense FLOPs for our models to avoid confusion and ensure consistency across our infrastructure. Here are a few points that might help:

• Standardize FLOPs Calculation: I'd recommend standardizing on dense FLOPs for all GPU types. This approach provides a consistent baseline for performance comparisons.
• Documentation: Clearly document the FLOPs calculation method in the codebase and any relevant documentation. This transparency helps users understand the performance metrics they're seeing.
• Use PyTorch Profiler: For more accurate FLOPs measurement, consider integrating PyTorch's built-in profiler. It provides detailed insights into model performance and can help verify the FLOPs calculations.
• Benchmarking: Run benchmarks on both H100 and H200 GPUs using a standardized dataset and model. This will give you empirical data to support the decision to standardize on dense FLOPs.