Integration of LibKineto CUpti Events and Metrics with PyTorch Profiler

[marceloamaral]

Hi team,

It looks like LibKineto can enable GPU hardware counters (Events) and/or GPU Metrics, but the integration with PyTorch profiler is not clear for me. I can see the potential use of the CUDA metrics in kDerivedMetrics, in EventProfilerTest and its initialization in EventProfiler::initMetrics...

So my questions are:

• Is there any documentation on how to use LibKineto CUpti events and metrics within the PyTorch profiler?
• If integration with the PyTorch profiler isn’t the intended use case, what are the main applications for these metrics?
• Can LibKineto produce Chrome traces that include GPU events and/or metrics?

I could not find examples of PyTorch profiler using GPU counter events, which suggests potential integration challenges. Clarifying these points would really help in understanding how to leverage LibKineto’s capabilities. Specially because I am trying to extend LibKineto to support metrics of a different AI hardware accelerator.

[marceloamaral]

@sraikund16 and @aaronenyeshi, any comment about this?

[sraikund16]

Hello,

Sorry for the delay. You can interact with the range profiler by setting profiler_metrics and profiler_measure_per_kernel parameters in torch.profiler._ExperimentalConfig when initializing your profiler.

For example:
    with torch.profiler.profile(
        activities=[ProfilerActivity.CUDA],
        record_shapes=True,
        on_trace_ready=trace_handler,
        experimental_config=torch.profiler._ExperimentalConfig(
            profiler_metrics=[
                "kineto__tensor_core_insts",
                "dram__bytes_read.sum",
                "dram__bytes_write.sum"],
            profiler_measure_per_kernel=False),
    ) as prof:
...

The metrics will be a subset of the kDerivedMetrics you pointed out in your description.

@briancoutinho can you answer question (3)?