Agent-side timer

[BFergerson]

I've been looking into how to implement an agent-side timer and SkywalkingTimer in the apm-toolkit-micrometer-registry appears to fulfill this purpose. Still, I'm not seeing how the histogram meter in this class works.

It calls MeterBuilder.buildHistogram with a DistributionStatisticConfig that appears always to be DistributionStatisticConfig.DEFAULT, which sets percentileHistogram to false and leaves serviceLevelObjectives as null. This configuration means that DistributionStatisticConfig.isPublishingHistogram() will always return false, which means MeterBuilder.buildHistogram will always return an empty Optional.

In what scenarios would the SkywalkingTimer.histogram not be empty?

Furthermore, how efficient would it be to use the buckets generated by PercentileHistogramBuckets? That class creates 276 buckets. That appears to be what SkywalkingTimer is trying to do. I'd like to make an agent-side timer and I'm curious how efficient that amount of buckets would be. I'm trying to create an efficient client-side timer to keep track of response times and based on https://prometheus.io/docs/practices/histograms/, I should avoid doing quantile calculations on the agent.

[wu-sheng]

@mrproliu Would you take a look at the code details?

[wu-sheng]

MeterService is the one hosting the metrics/meter registration. There is a timer inside it to fetch metrics' latest values to assemble the report. All metrics are registered to hold the latest value and provide BaseMeter#transform to build meter with tags and values.

I am not sure whether this is your asking.

[BFergerson]

That's not the timer I'm asking about. I'm asking about a timer like https://metrics.dropwizard.io/4.2.0/getting-started.html#timers. DropWizard Metrics has five metric categories:

• Meter
• Gauge
• Counter
• Histogram
• Timer

I'm asking why SkyWalking decided to only implement four of them and leave out Timer. Is it because Timer is actually a collection of Meters, Gauges, Counters, and a Histogram? I'm also asking if a histogram of 256 buckets would be efficient.

My goal is to get the max, min, avg, standard deviation, and some percentiles of many artifacts in as efficient a way as possible. I know SkyWalking can calculate percentiles on the backend so I want to gather as much information as I can without doing those calculations on the agent.

[wu-sheng]

I'm asking why SkyWalking decided to only implement four of them and leave out Timer. Is it because Timer is actually a collection of Meters, Gauges, Counters, and a Histogram?

I don't know what is the difference between a single metric and a collection. Because in SkyWalking, the metrics of a process are always reported as a collection.
I think they only have a difference if the SDK/agent would report metric one by one, and a collection mean they should be reported in one stream. But we don't have this case, actually.

I'm also asking if a histogram of 256 buckets would be efficient.

I think 256 is not a big deal. If those metrics matter, then report them.

[BFergerson]

I don't think I'm explaining myself well, but I think I've got enough information to push forward. Essentially what I was saying is that I'd like a native Timer that extends BaseMeter which collects data that is later calculated on the server. From what I understand, this can be done with tracing because each span carries the start and end time which allows for percentile calculations on the server side. I'm trying to achieve something similar without having to create a span for each operation as these operations will be called many times a second and a rough estimate would suffice. PercentileHistogramBuckets looks like the best way to achieve this so I think I'll go with that.

I have one more question related to this topic. Would it be more efficient to have a hundred metrics with no tags or one metric with 100 tags? I'm trying to keep track of the duration of a local operation (thus the timer) and I want to correlate that time to the parameter it operates on. I expect there to be a correlation so I want to add a tag (or a new metric) to keep track of that correlation.

[wu-sheng]

I'm trying to achieve something similar without having to create a span for each operation as these operations will be called many times a second and a rough estimate would suffice.

If you already had the tracing-centric system, then yes. That is our Logic Endpoint. Any metrics system would be very dangerous if you are going to calculate them on the client side. As a period reporting system(meter system), you would never know what is the correct period.

I have one more question related to this topic. Would it be more efficient to have a hundred metrics with no tags or one metric with 100 tags?

Both of these seem overburning. If that is an operation, try logic endpoint, a new entity is better than tags or traces.

1. Tag is hard to search and visualize.
2. Trace is just raw data.

I may be wrong, this eventually would rely on which kind of operations you would track, and how to set up dimensions.

[mrproliu]

In what scenarios would the SkywalkingTimer.histogram not be empty?

Have you tried on this unit test, I think you haven't setting the serviceLevelObjectives in the configuration.

[BFergerson]

That looks like it would work. I see that SkywalkingMeterRegistry.newTimer doesn't use the histogram meter, but building a timer manually would.