[Feature] Enabling dynamic profiler plugins

[briancoutinho]

Enabling dynamic profiler plugins (Feature Proposal)

Authors: @yisitu, @zli669, @briancoutinho (NVIDIA)

TLDR

• We would like to contribute a feature to Kineto to enable the plugging in of a new profiler for NVIDIA GPUs for streaming performance metrics and low-overhead tracing over a long duration.
• We propose a Dynamic plugin capability that enables Kineto to load the new profiler module. The Dynamic interface is an extension of the statically compiled ActivityProfilerPlugin API. Dynamic loading makes Kineto more extensible and decouples plugin development from Kineto/PyTorch mainline.

Motivation

Why a dynamic plugin?

• The plugin feature allows the development and extension of new profiler capabilities before they are ready to be published. Otherwise, developing in PyTorch and Kineto will require a full compile time source integration and slow iterative development cycles.
• Extending the ActivityProfiler plugin interface to enable dynamic plugins has the added benefit of making kineto more extensible. Closed source plugins from other vendors or components can be easily integrated into kineto without complex compiler source hacks. It also permits independent development and prototyping of plugins.

Overview of Changes

We will be posting a PR shortly; the changes primarily include:

1. Dynamic plugin C interface and implementation.
   1. Implement a fully C style interface for Activity Profiler plugins and add the shim necessary.
   2. Basic versioning and forward/backward compatibility support.
2. Adding a runtime flag for CUPTI to be disabled so that it does not conflict with the new profiler plugin.

Prototype Results

We have a working prototype with kineto. You can download the trace attached below

• CUDA kernel events and correlation arrows work as usual.
• Also see the “GPU PM Counter” rows. Potentially any valid set of PM counters can be sampled.

Here is a sample trace json that can be viewed in Perfetto or Chrome Trace.
resnet_training.json.gz

Please let us know any questions or concerns.

cc @sraikund16 , @bertmaher , @valentinandrei , @nadavrot

[australopitek]

Hi @briancoutinho ,
what is the status of your work on this feature? We're planning to add a new plugin to Kineto and we're wondering if it makes sense to wait a bit and do it the new way you're working on.

[yisitu]

Waiting makes sense, we'll put up the PR very soon.

[australopitek]

Waiting makes sense, we'll put up the PR very soon.

Thanks for the reply. Is the draft of change available somewhere publicly, so we can look into it already?

[briancoutinho]

Not yet but we can put a draft PR next week. Folks are on vacation this week, will get back.

…

On Fri, Sep 12, 2025 at 8:02 AM Amadeusz Skrzypczak < ***@***.***> wrote: *australopitek* left a comment (pytorch/kineto#1121) <#1121 (comment)> Waiting makes sense, we'll put up the PR very soon. Thanks for the reply. Is the draft of change available somewhere publicly, so we can look into it already? — Reply to this email directly, view it on GitHub <#1121 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABUZ7ZEP67FIQBLV7D5Y4WL3SJVXHAVCNFSM6AAAAACDMFN4Q6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEOBUGAYTIMZTGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[briancoutinho]

@australopitek set up a draft PR here :)
#1148