Show metrics in Grafana with Elasticsearch

[nisiyong]

Many people asked whether SkyWalking support showed data in the Grafana, and I try to adapt that recently. It is not really tough work, it needs to change the ES mappings and the OAP needs to add fields in some indices.

Show the screenshots first, I create four dashboards(Global Dashboard, Service Dashboard, Instance Dashboard, Endpoint Dashboard) in Grafana as the rocketbot-ui does. And the Instance Dashboard is as follows:

There are some brief changes to do this:

• Change time_bucket,last_update_time_bucket,last_ping fields as date type, it is compatible.
• Add the downsampling type for metrics indices, only add this field when the metrics data is Hour or Day
• Add name to the traffic indices for the Grafana select options.
• Need to separate the percentile metric value for each field NOT in one field.
• ...

It is possible to do this, it needs to make some changes in the storage layer, I am very willing to do this without compromising performance. And I wish to listen to the community's voice, I want to do this with the community rather than maintaining the internal branch.

[wu-sheng]

Let's focus on the time relative things first.
I am not sure your compatibility is correct. I think you need to learn how we use in backend, and we use yyyyMMddHHmm, yyyyMMddHH, yyyyMMdd, rather than just timestamp.

[nisiyong]

That is okay, and I had tested it. We could use date format provided by Elasticsearch. The time bucket will store milliseconds-since-the-epoch in UTC, when the OAP querying or writing, the time bucket will be formatted. The following mapping could do this:

"time_bucket" : {
  "format" : "yyyyMMddHHmmss||yyyyMMddHHmm||yyyyMMddHH||yyyyMMdd",
  "type" : "date"
}

[wu-sheng]

Then we face two place processing TZ, which os dangerous. Like I mentioned, SkyWalking has capacity to add names(not encoded ones) into database(at least for OAL), we could make ElasticSearch storage implementation add an extra timestamp field for tools like Grafana.

[nisiyong]

we could make ElasticSearch storage implementation add an extra timestamp field for tools like Grafana.

That's better. I think I could do this step first, it will solve the first and most important problem.

[wu-sheng]

Also, this should be optional only.
Please consider how to change and share the idea here.

[nisiyong]

Recently I had tried to adapt this, these are some small changes could to do this:

1. Add two fields long timestamp and String time_unit in org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.

2. Set timestamp value when set time_bucket, Metrics#setTimeBucket(long timeBucket) as follow:

   public void setTimeBucket(long timeBucket) {
      this.timeBucket = timeBucket;
      // set timestamp
      this.timestamp = TimeBucket.getTimestamp(timeBucket);
   }

3. Set timestamp Elasticsearch type as date in org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.StorageEsInstaller

   else if (Metrics.TIMESTAMP.equals(columnDefine.getColumnName().getName())) {
         Map<String, Object> column = new HashMap<>();
         column.put("type", "date");
         column.put("format", "epoch_millis");
         properties.put(columnDefine.getColumnName().getName(), column);
     }

4. Set time_unit when downsampling metrics, like DAY and MONTH. It is simple to do at OAL code templates, but cumbersome to set hard-code metrics manually one by one.

   metrics.setTimeUnit("HOUR");
   metrics.setTimeUnit("DAY");

Why do I do by this way?

• At the first, I want to add timestamp filed at receiver layer org.apache.skywalking.oap.server.core.source.Source, because we could set timestamp without converting time_bucket. But it should change a lot of codes and the storage layer must change together. So only at the filed in theorg.apache.skywalking.oap.server.core.analysis.metrics.Metrics will be better but the timestamp should covert from time_bucket.
• Why do we need time_unit filed? Because currently the downsampling metrics were put with the origin metrics, we should recognize them when using the aggregation function, or the aggregation results will be wrong. The time_bucket works well because it uses the number range to recognize whether there are downsampling metrics, but the timestamp should work with the time_unit field.

If there are any problems, please tell me here. and I am willing to create a PR to be reviewed.

[wu-sheng]

I have some suggestions about how to set these 2 fields.

1. Change nothing in the metrics level. This new feature is irrelevant to core.
2. Take a deep look at new converter concept in builder. It could be helpful in ElasticSearch server only.
3. Unit is fine to add, but time bucket indices it already, you don't need metric object to hold it. Do it in converter only.

Please keep this advanced feature inside ElasticSearch storage implementation only.

[nisiyong]

Okay, thanks for your guide. I haven't noticed the new converter concept because I use the 8.9.1 version.

The org.apache.skywalking.oap.server.core.storage.type.Convert2Entity looks good. Do you mean I could do this in ElasticsearchDataConverter like IoTDBDataConverter does?

And if we want to use timestamp replace time_bucket in future, I think there no problem add this in Metrics. Currently do this for Elasticsearch only, it is worth to try. I will take a deeop look at the converter.

[wu-sheng]

Do you mean I could do this in ElasticsearchDataConverter like IoTDBDataConverter does?

You could build a TimeTypeEnhancedConverter to extend HashMapConverter when you need it(when option ON).

And if we want to use timestamp replace time_bucket in future, I think there no problem add this in Metrics. Currently do this for Elasticsearch only, it is worth to try.

I think in the kernel, it would not happen. Because timestamp has no Timtzone concept, then downsampling is debatable, what are actually hour/day level metrics, such as they have different meanings in Beijing or India time zone