tt-smi should report TBP draw as well as ASIC power draw

[bgoelTT]

A lot of customers and developers need a way to measure the TBP draw. This is required for many reasons including but not limited to:

• performance comparison with competitor hardware
• in total system power budget calculations

Additionally, users expect feature parity with nvidia-smi https://developer.nvidia.com/system-management-interface which does report power draw at the board level.

This feature can be considered complete when we report TBP draw in tt-smi in the telemetry tab. Specifically we need the ability to query instantaneous, and average power draw via luwen and surfaced through pyluwen. The associated issue in that project is here:

┆Issue is synchronized with this Jira Task by Unito

[sbansalTT]

@kikkiakki Do we have these values reported in telemetry table? If so it should be trivial to expose

[kikkiakki]

@sbansalTT Yup, everything there already

[bgoelTT]

@kikkiakki @sbansalTT is this scheduled to be worked on?

[sbansalTT]

Best guess, in one of the next 2 release - Priority is stabilizing reset for WH 6U galaxy in the upcoming release

[fleclairTT]

Thanks @sbansalTT! This has also come up in the context of product request for our larger deployments. The ability of TT-SMI to pull from the boards is great, but making sure that we can expose Galaxy data is also of value; for telemetry, ideally we could also pull from the BMC?

At the same time, being able to write out the data from TT-SMI into the BMC for cluster scale telemetry would be very useful; that's typically what's done at the hyperscaler level

[bgoelTT]

@kikkiakki what is the correct field to query in the telemetry table? Also, is this surfaced via pyluwen? If all the information already exists there then we could start working on a small utility to help some of our customer testing

cc: @gtobarTT

[sbansalTT]

Yeah for the galaxy boxes that number will not be coming from the asic lvl telemetry - it would be from the BMC.
Roping in @cyberDrake to give you details about that.
Asic lvl telemetry will still be from tt-smi

[sbansalTT]

btw @bgoelTT You can get the telemtry struct dump if you do a tt-smi -s If the board power is there, it should be available via the json file anyway

[kikkiakki]

the TBP tag still needs to be added to pyluwen. It won't show up in tt-smi -s until the tag is both added to luwen and exposed to tt-smi.

[cyberDrake]

I have more questions than I have answers:
are we talking about a 6u where we have a BMC or a pcie card, where we don't? Do we really want to extend tt-smi to display values that aren't going to be available for pcie users? Is chasing feature parity with NVIDIA really what we want to do here?

[bgoelTT]

@kikkiakki is this the correct field returned by tt-smi -s for board power? "MVDDQ_POWER": "0x190000". If so, how do we decode this value?

Just saw your message above sorry