LLO still not visible in libtpt 0.0.37 with Xprof 2.22

[elbertwang]

Hi,
I followed this doc try to view the LLO
https://github.com/openxla/xprof/blob/master/docs/custom_call_profiling.md

I tried both
libtpu 0.0.36 with ml-diagon managed xprof 2.21 and self host xprof 2.22
libtpu 0.0.37 with ml-diagon managed xprof 2.21 and self host xprof 2.22
without luck.

[elbertwang]

fixed with libtpu 0.0.38dev nightly

[Matt-Hurd]

Hi @elbertwang, we're aware of an issue causing this and the fix is in the next libtpu release. I believe this should also be fixed in the xprof nightly (although probably not in diagon yet, but will be in the next release).

[pathfinder-pf]

@Matt-Hurd Thanks to this doc: https://github.com/openxla/xprof/blob/master/docs/custom_call_profiling.md. I have a question: what do the spikes, gaps, and continuity in the image below represent?

and in tensor core section, different color means ?

[pathfinder-pf]

@Matt-Hurd hello, Could you explain the working principle of this LLO still? thanks

[Matt-Hurd]

Hi @pathfinder-pf.

What do the spikes, gaps, and continuity in the image below represent?

In the Trace Viewer, the rectangular slices represent the active execution time of operations mapped to specific hardware units (like the MXU or Vector ALU).

• Continuity (Solid blocks): The specific hardware resource is being continuously utilized without interruption. For example, solid blocks on the MXU track indicate that the kernel is achieving dense math utilization.
• Gaps (White space): The resource is temporarily idle. This often indicates an unhidden data dependency or pipeline stall. For instance, computation units sitting idle while waiting for memory loads to complete before they can proceed.
• Spikes: Very brief bursts of resource utilization. These often represent very short or singular operations executing in isolation between larger blocks of work.

What do the different colors mean?

The distinct colors in the top-level Tensor Core track are determined by hashing the name of the operation or lexical scope. In custom kernels, these colors typically correspond to structural boundaries in your code, such as explicit jax.named_scope decorators, loops, or conditionals. This is a purely visual feature to help you distinguish between diferent operations or iterations.

What is the working principle of this LLO?

This is tough for me to answer as it's outside of my area of expertise. I found this blog post to be pretty insightful: https://patricktoulme.substack.com/p/when-xla-isnt-enough-from-pallas

[pathfinder-pf]

thanks a lot. @Matt-Hurd

[pathfinder-pf]

Could you show us which LL0 instructions are actually running in these areas where utilization is observed in the picture? @Matt-Hurd