LogQL support

[weixiang1862]

Based on @wankai123 works on promql, I try to implement loki query protocol(LogQL) on skywalking and validated the feasibility. I want to confirm whether LogQL can be accepted by skywalking community. Should I keep working on this?

[kezhenxu94]

@weixiang1862 thanks for bringing this up, I'd love to see this feature!

[weixiang1862]

Thanks for you reply. I found loki's license is AGPL-V3, is it a limitation?

[kezhenxu94]

Thanks for you reply. I found loki's license is AGPL-V3, is it a limitation?

If we only implement their query protocol, that's totally OK, but if we need to add their code dependency, that's not OK

[weixiang1862]

Ok, I will put some design docs here when it into POC state. It is just a demo version now.

[wu-sheng]

We need to see what LogQL page says.
It is a vendor project, rather than foundation, and implementation usually needs to ref(logically copy), it is not easy to set boundaries.

Or it is better to ask Loki community, or you could make sure the codes arw totally different.

[weixiang1862]

Loki re-license it's license from apache-2 to agpl-v3 since this commit, a part of code was migrate to agpl-v3.

Now, log query protocol is under apache-2 but the logql lexer and parser is under agpl-v3：

In this commit, I found two articles about this change from grafana：

• Grafana, Loki, and Tempo will be relicensed to AGPLv3
• Q&A with Grafana Labs CEO Raj Dutt about our licensing changes

After I read it, I feel pessimistic about LogQL implemention in SW. I just want do some contributes to SW community but not troubles.
I will keep doing this in my local, if things change in the feature, I will bring this up again.

[wu-sheng]

Discussing licenses is a common process when we want to add a big component like you proposed. But this doesn't mean negative.
We just need to get things clear before moving forward.

[weixiang1862]

wihch part are you going to use?

1. query protocol, including request、repose、urlPath
2. A part of LogQL syntax(not all syntax are supported by skywalking, like regular match)

Do you need to copy codes from Loki?

No need.

do you need syntax parser?

Yes, and I will write a new antler. Because LogQL is a litter different from PromQL, for example:

{container="query-frontend",namespace="loki-dev"} |= "metrics.go"

[wu-sheng]

query protocol, including request、repose、urlPath
A part of LogQL syntax(not all syntax are supported by skywalking, like regular match)

These are fine from license perspective. But, have you validated whether the feature matched? Such as, SkyWalking has multiple storage options to provide consistent APIs for query in the kernel. Are the existing APIs enough to cover all features?

{container="query-frontend",namespace="loki-dev"} |= "metrics.go"

Is metric.go making sense to SkyWalking part?

[weixiang1862]

All Skywalking log query can be translate to LogQL. But not all features of LogQL can be covered by current storage query api.

It matchs well with skywalking in equivalent query. For example:

{service="consumer-application",service_instance_id="xxx"} |= "${keyword_contains}" | != "${keyword_not_contains}"

But it is hard to support regex matches query. For example:

{service="consumer-application",service_instance_id="xxx"} |=~ "${regex_matches}" | !~ "${regex_not_matches}"

{service !="consumer-.+",service_instance_id="xxx"} |=~ "${regex_matches}" | !~ "${regex_not_matches}"

=: exactly equal
!=: not equal
=~: regex matches
!~: regex does not match

In addition, LogQL extend log queries to calculate values based on query results which called Metric queries, for example:

# Count all the log lines within the last five minutes for the MySQL job.
count_over_time({job="mysql"}[5m])

So, LogQL features is partially supported. And need do some additional calculate in query layer if we need to support Metric queries.

[wu-sheng]

For metrics part, no, we wouldn't. Read LAL, it's there for pre aggregation already.

https://skywalking.apache.org/docs/main/next/en/concepts-and-designs/lal/#extractor

(metric extractor)

---

It is better you could write a complete proposal about your ideas.
I am feeling we need to agree on design, then codes.

[weixiang1862]

For metrics part, no, we wouldn't. Read LAL, it's there for pre aggregation already.

https://skywalking.apache.org/docs/main/next/en/concepts-and-designs/lal/#extractor

(metric extractor)

It is better you could write a complete proposal about your ideas. I am feeling we need to agree on design, then codes.

Ok, I will complete proposal doc first.

[weixiang1862]

As what we implement is not a full features LogQL, I build a dashboard using loki ui component rather than using native loki query page, I think this is what we need, now logs and metrics all can show by grafana. Do you think it is ok?

[wu-sheng]

This is why we need the design doc. We need boundaries about which are included and which are not.
We need you to write it down from LogQL perspective, not just a screenshot.

[wu-sheng]

Otherwise, this would look more like a custom component, which could be released separately.

[weixiang1862]

Ok, docs is first priority in the next.

[wu-sheng]

Thanks, the screenshot is good.
Let's be patient, and move forward step by step.

[weixiang1862]

Background

Skywalking already supported PromQL in metric query, user can easily build a metric dashboard in third-party systems which support PromQL (such as Grafana). However, skywalking is a full stack apm system, there are other features provided by skywalking like logs. If user already build a dashboard on Grafana, it's inconvenient for him to query logs between offical native ui and grafana when he find some abnormal metrics in dashboard and want to figure out what happened in that moment. If we implement LogQL, we can provide user an alternative choice in log query.

Query Data Flow

grafana --> loki-datasource(skywalking)  --> logql-query-plugin --> log-storage-api

LogQL Support

As skywalking log mechanism is different from loki, The LogQL implemented by skywalking won't be a full features LogQL.

The picture above is LogQL syntax in log querys, it is composed of the following parts(feature with [✅] is what we need):

• stream selector：select logs with specified label
• line filter: filter log content contains keyword or not contains keyword
• label filter: filter log by label
• parser: parse specified format log(json、logfmt), and convert key、value to label、label_value, which can be refilter by label filter
• formater: format log content or log label with template syntax

The operator support by LogQL is composed of the following(operator with [✅] is what we need):

• |=: exactly equal
• !=: not equal
• |~: regex matches
• !~: regex does not match

Some typical expression used in skywalking log query:

{service="$service", service_instance="$service_instance"} |= "$keyword_contains"

{service="$service", service_instance="$service_instance"} != "$keyword_not_contains"

{service="$service", service_instance="$service_instance", endpoint="$endpoint"}

{service="$service", service_instance="$service_instance", trace_id="$trace_id"}

Http API Support

1. /loki/api/v1/labels

query which labels are support in stream selector, in skywalking, they are fixed value: service、service_instance、endpoint、trace_id.

2. /loki/api/v1/label/{label_name}/values

query label values of label, in skywalking, they are service_traffic、instance_traffic、endpoint_traffic.

3. /loki/api/v1/series

Finding series by label matchers, it is used in query condition select.

4. /loki/api/v1/query_range

query logs in time range with LogQL expression.

[weixiang1862]

Ok, then we just need implement an /api/v1/query_range api.

[wu-sheng]

@kezhenxu94 @wankai123 Any feedback.

[wankai123]

Ok, then we just need implement an /api/v1/query_range API.

The following APIs are used in the variable setting for metadata
They could leverage the existing PromQL API, right?

1. /loki/api/v1/labels
2. /loki/api/v1/label/{label_name}/values
3. /loki/api/v1/series

[weixiang1862]

@wankai123 Yes.

[wankai123]

LGTM

[weixiang1862]

Sync progress in last week：

1. /labels and /label/{label_name}/values is needed
   I am sorry I was too arbitrary to get the conclusion before. In fact we need implement them in loki.
   /labels is used as health check endpoint by grafana when we add a loki datasource, there will be an obvious error tips without it.
   /label/{label_name}/values is needed too. It works will /labels to provide tag auto complete query.
   This two endpoints are used for tag key and tag value query now. Others metadata(service、instance、endpoint) query still provided by promQL service.
   

2. log query size limit
   loki doesn‘t support pagenation, it query a certain number of logs in a duration which can be defined by limit param in dashboard. The default number is 1000, do you think it is reasonable, should we adjust the default value?
   

3. KeywordOfContent and ExcludingKeywordsOfContent doesen't support multi value
   Grafana doesen't support multi value text box variables, it means user can only query logs in single keyword.
   Although we can use "," to separate keywords in a text box, It will broken the LogQL semantics, and sometimes it is ambiguity when user really want the log contains ",". Do you have suggestions here?
   

[wu-sheng]

/labels and /label/{label_name}/values is needed

So, you mean this is for extra conditions, rather than overlapping with service, instance, and endpoint, right? If so, it is fine. Let's document them clearly.

The following APIs are used in the variable setting for metadata
They could leverage the existing PromQL API, right?

/loki/api/v1/labels
/loki/api/v1/label/{label_name}/values
/loki/api/v1/series

Our biggest concern was as @wankai123 mentioned. If it is for other conditions, we are good.

loki doesn‘t support pagenation, it query a certain number of logs in a duration which can be defined by limit param in dashboard. The default number is 1000, do you think it is reasonable, should we adjust the default value?

For us, it is fine. Paging is costing a lot.

Grafana doesen't support multi value text box variables, it means user can only query logs in single keyword.
Although we can use "," to separate keywords in a text box, It will broken the LogQL semantics, and sometimes it is ambiguity when user really want the log contains ",". Do you have suggestions here?

Considering this is a Grafana limitation, I would say, I don't have a better solution than you had proposed. We either introduce a separator(don't have to be comma), or we don't support multiple values.

[weixiang1862]

So, you mean this is for extra conditions, rather than overlapping with service, instance, and endpoint, right? If so, it is fine. Let's document them clearly.

Yes, It is just for log tag condition. I will document them.

For us, it is fine. Paging is costing a lot.

Ok, thanks.

Considering this is a Grafana limitation, I would say, I don't have a better solution than you had proposed. We either introduce a separator(don't have to be comma), or we don't support multiple values.

Ok, I think single value is better in this version. And I will consistently pay attention to grafana release, if it improves in the fuature, I will adjuest the dashboard too.

[wu-sheng]

Ok, I think single value is better in this version. And I will consistently pay attention to grafana release, if it improves in the fuature, I will adjuest the dashboard too.

I am good with this. Once we have the first version, we will get more feedback from the community. Let's see.