Is there a recommended way to display OEM statsd atoms in the UI?

[cphlipot1]

AOSP reserves statsd atoms > 100k for OEMS to use: https://cs.android.com/android/platform/superproject/main/+/main:frameworks/proto_logging/stats/atoms.proto;l=1614?q=f:stats%2Fatoms.proto$%20message%5C%20Atom

Since these are different per-device it makes sense that the perfetto UI itself does not contain proto descriptors to understand them. Is the a recommended way to display these properly in the UI?

For reference here is what some of the OEM atoms from Pixel 7 Pro look like when collected:

[LalitMaganti]

Very funny you should ask this. The answer to this question can actually be found in #3227 which I'm literally implementing right now in
#4192.

The basic premise here is that we're going to expose a bunch of endpoints where people can "extend" the UI to give it knowledge of things that we don't know and cannot bake into the Perfetto tree itself (i.e. vendor specific things Google doesn't know about, confidental/proprietary information etc). The idea will be you can (fully optional, disabled by default) "point" the UI at some Github repo, GCS/S3 bucket or any HTTPS URL (where you can have a fully hosted server) to allow it to fetch extra information.

Specifically to start with we are planning on exposing:

1. SQL modules: you can add things which can be included with INCLUDE PERFETTO MODULE statements
2. UI macros: from https://perfetto.dev/docs/visualization/ui-automation but instead of putting in settings, you can have it centrally shared and managed
3. Proto descriptors: to allow us to parse the trace using schemas we don't have access to by default.

For statsd atoms, 3) is what is relevant: inside Google we have exactly the same problem where we have a bunch of atoms which have schemas specific to us and we cannot bake into the open source code. We have a "backchannel" mechanism which only works for Googlers atm but it's extremely fragile and not at all scalable. Server extensions (which I think should be landing very shortly) are the way we can let anyone make use of the same mechainsm.

In the future we'll also extend this to:

1. Symbolizing native heap profiles and stack sampls
2. Deobfuscating Java heap dumps, native heap profiles, stack samples and atrace slices in system traces

But that requires some more design work as unlike the above, we'll need dynamic lookup, not just static GET requests.

[LalitMaganti]

Oh of course all of this assumes that you actually have access to the statsd proto schemas in the first place. If you don't I'm afraid there is no solution to that by design of how statsd works - i.e. the ability for OEMs to log things confidential to them.

[cphlipot1]

Thanks for clarifying. I had seen the RFC previously but wasn't clear what practical usecases it was targeting.

I think the server extensions look interesting and could help with things like this but i'm curious if the implementation would be flexible enough to dynamically load different descriptors depending on the device, since i'm not sure all products would have identical or non-conflicting atom_ids in this range.

Initially I had thought we might have wanted to do something like have the statsd datasource automatically bundle descriptors in the trace the first time it sees a particular atom id that is >= 100k show up in a packet, but sounds like this is deliberately omitted by design for confidentiality reasons.

My original idea of bundling in trace would mitigate this somewhat since each trace is guaranteed to have matching descriptors since it is bundled in the trace. I think this would also address the trace processor CLI usecase as well, and not just UI. Lastly i think if descriptors were loaded into the trace processor on a per-device basis in the context of a multi-device trace and each context had its own set of descriptors you could possibly resolve oem atoms from two different device in the UI at the same time even if they have conflicting descriptors.

All that is hypothetical and doesn't really impact me at the moment, but wanted to call out in case it is something you want to handle as part of server extensions.

[cphlipot1]

I had also seen https://github.com/google/perfetto/blob/main/docs/design-docs/extensions.md previously and was not clear if this was meant to be another alternative.

it looked like you could potentially use that add more statsd descriptors to an existing trace synthetically, but i have not tried, and it is not clear how well that is supposed to be supported since it was added 6 years ago and is still labeled "work in progress"

[LalitMaganti]

Initially I had thought we might have wanted to do something like have the statsd datasource automatically bundle descriptors in the trace the first time it sees a particular atom id that is >= 100k show up in a packet, but sounds like this is deliberately omitted by design for confidentiality reasons.

The problem with this is that we don't know this schema on device. No one does actually, not the OEM, not statsd. The whole point was that this sort of thing is only available on the server side when the atoms are received (as statsd was really designed to be a field telemetry system).

My original idea of bundling in trace would mitigate this somewhat since each trace is guaranteed to have matching descriptors since it is bundled in the trace. I think this would also address the trace processor CLI usecase as well, and not just UI. Lastly i think if descriptors were loaded into the trace processor on a per-device basis in the context of a multi-device trace and each context had its own set of descriptors you could possibly resolve oem atoms from two different device in the UI at the same time even if they have conflicting descriptors.

I'm happy to support this as well but as an additional option. strike - see below, this actually works today!

The approach you propose unfortunately does not work in general because a) we don't necessarily have the descriptor as in the field case on device b) it bloats the size of the trace (these descriptors can get into the MB and at Google scale, that's a lot of unnecessary redundant data) c) it's a bad UX for people (users: why am I collecting this trace and it works differently when I collect locally vs when you give me the trace from some central dashboard where it's pre-concatenated with the descriptor).

We know the above because that's exactly how symbolization and deobfuscation works today (you need to concatenate some magic packets) and people constantly come to us like c) and it takes a bunch of resources to point people in the right direction about symbolization/deobfuscation. It's one of the places where we "lose" the most users.

All that is hypothetical and doesn't really impact me at the moment, but wanted to call out in case it is something you want to handle as part of server extensions.

Yeah it's a good point and not one I was thinking to because for our usecases, there is only one OEM we're really bothered about at any one time: moreover, it's the same for other OEMs who I was also thinking to (e.g. Samsung) where they'd only want their own descriptors.

But also the situation of "I know multiple OEMs schemas is a bit weird" actually because who exactly is in that position? It's only really people who have access to confidental information from multiple companies which, I would really imagine would be a very small number of people in the world. Not sure I want to really cater to them...

I had also seen https://github.com/google/perfetto/blob/main/docs/design-docs/extensions.md previously and was not clear if this was meant to be another alternative. it looked like you could potentially use that add more statsd descriptors to an existing trace synthetically, but i have not tried, and it is not clear how well that is supposed to be supported since it was added 6 years ago and is still labeled "work in progress"

Actually I had totally forgotten about this. This would totally work. It's actually used by Chrome to add their own "extensions" to trackevent without us knowing about it. The WIP there is a lie: it's well supported and used. We should really add a proper doc to do this instead of being a hidden away, WIP design doc.

[cphlipot1]

Thanks for the clarification. All the context was very helpful!