Emit per stream metrics - bit rate, resolution

[Strykar]

Is your feature request related to a problem? Please describe.
O's wish to measure concurrent stream count + what resolution and bit rate each was transcoded at.
This is critical to plan infrastructure growth.
We charge per pixel, we should be able to measure how many total pixels (according to LPMS / Mist) we have transcoded.

Describe the solution you'd like
Emit Prometheus metrics that let us track concurrent streams by resolution and bit rate.

[papabear99]

Can Broadcaster IP addresses be added to the stream metrics so Orchestrators can see where their work is originating?

Having this information available in Prometheus/Grafana would help Orchestrators plan their infrastructure expansion.

[hthillman]

does this address the concerns here? #2238

[papabear99]

Looks like it takes care of the pixel count and broadcast IP requests.

Doesn't look like bit rate is addressed.

@Strykar are you looking for full resolution i.e. 720x480p per stream vs. the current horizontal only resolution i.e. 480p displayed in the session profile?

[darkdarkdragon]

@papabear99 @Strykar
Pixels/second is the only metrics that shows real load.

For resolution - one stream could have 2 target renditions, other stream 4. How would you group this in metrics?

[Strykar]

@darkdarkdragon I've asked @Thulinma to chime in here

[hthillman]

I'm inclined to agree with @darkdarkdragon's assessment here

[Thulinma]

Yeah it looks like the mil_pixels_processed stat is the best one to track actual load on the system over time.

I can see use for having buckets of input/output resolutions handled (per video track in/out), too - for tracking the kind of capabilities hardware needs to have to handle average requests. Unfortunately I'm not deep enough into go-livepeer's code yet to know exactly how the prometheus metrics work; how do those eventually get created / in what format? Or is that always done by some stand-alone tool next to go-livepeer, perhaps? It would also make sense to not log every client IP there, but instead group by geographical location through for example a 3-character geohash.