Feature: Optionally add redis role label to all redis related metrics

[nantiferov]

Describe the problem
This is feature request.

I think it would be useful to implement optional feature to add redis role label to all redis related metrics. Atm I think there's a problem how to distinguish master from replica stats on dashboard. We have selector in our dashboards to choose proper instance/pod based on role, but then when failover happens, it becomes hard to combine metrics from different instances/pods on one graph. It's possible to achieve this by using join with redis_instance_info, but promQL query is pretty huge after that.

I can implement PR adding flag to redis_exporter to propagate redis role to all redis metrics.

P.S. I considered adding relabeling on Prometheus side to achieve this, but apparently kubernetes_sd is having bugs detecting update in labels, so it's pretty unreliable prometheus/prometheus#13322 (comment)
For ec2 it requires custom hacks to propagate role to ec2 tags.

And for both of these solutions, propagation of changed role will take time, while exporter can do it immediately.

What version of redis_exporter are you running?
Please run redis_exporter --version if you're not sure what version you're running.
[ ] 0.3x.x
[x] 1.x.x

Running the exporter
NA

Screenshots
NA

Additional context
NA

[oliver006]

Sorry about the delay here. And thanks for writing this up, that's helpful!

It's possible to achieve this by using join with redis_instance_info, but promQL query is pretty huge after that.

tbh, I don;'t think that's a good reason, this can be solved in multiple ways (templating, recording derived metrics).

In a cluster scenario, I think it'd be helpful to add the run_id to a bunch of metrics (see PR #973) so we can distinguish which process answered the metrics request. And that should also tell you what process was master / slave at a given time and from there you can go (via the query you describe) to filter or query by role.

I just don't want to pollute all the metrics with a label that can be derived (even if it's not trivial to construct the query).

Let me know if I misunderstand or miss something.

[nantiferov]

Hi. Sorry for delay, was on vacation.

this can be solved in multiple ways (templating, recording derived metrics).

I'm not sure if I'm following you here. From my perspective, only options are joins with redis_instance_info and relabeling based on labels/tags (which is not reliable at all unfortunately). I don't know how to achieve this with templating, recording derived metrics.

Moreover, I checked our dashboards for Redis and we have 30+ different panels in dashboards. If I will try to add joins to each query, they became too overcomplicated. And then there's a case with duplicated time series returned during failovers, so query should include topk, which makes it even worse, example: rate(redis_commands_processed_total{alias=~"$alias"}[$__rate_interval]) * on(instance) group_left(role) topk(1,redis_instance_info{alias=~"$alias", role="master"}) by (instance)

In a cluster scenario, I think it'd be helpful to add the run_id to a bunch of metrics (see PR #973) so we can distinguish which process answered the metrics request. And that should also tell you what process was master / slave at a given time and from there you can go (via the query you describe) to filter or query by role.

In our setup we have redis_exporter running per each node in cluster, so this issue doesn't affect us I believe.

I just don't want to pollute all the metrics with a label that can be derived (even if it's not trivial to construct the query).

But this will be under the flag in options, disabled by default, so metrics won't be polluted. My point is that redis_exporter has most up to date information about redis role and it could be relatively easy added to redis metrics, which will simplify both monitoring and alerting.

[oliver006]

(templating, recording derived metrics).

What I meant is you can reduce the manual repetition of the joins by either

• use templating for your queries (generate them via a template)
• or record metrics using the more complicated join and then grafana can use that recorded metric (that's not suitable if you want to do this with every single metric the exporter is exporting

Here's a question though: I'm not sure I understand how you plan to use that "role" label in your grafana dashboard.
Can you give me an example?
Are you trying to look at only the metrics of a node that has the role==master?
If you create a filter / variable in grafana, that wont work, right? Cause it'll apply the same alias/addr to the entire length of your graph and not switch it if the role switched somewhere along the way?
Let me think about this a bit more, maybe there's another way to do this without adding labels to most metrics.

[nantiferov]

Thank you for details, I'll try to check recording rules to achieve this.

[nantiferov]

Here's a question though: I'm not sure I understand how you plan to use that "role" label in your grafana dashboard.

Basically I believe metrics from replicas are non-actionable for usual Redis users and with this feature I'll be able to use selector in all panels like {redis_role="master"}

Variable/filter by node isn't working perfectly because at one time node could be master, then - replica. And because of this, panel will show partially metrics from master and partially from replica.

Right now it's kinda "fixed" by repeating row in Grafana by node, but it creates confusion when failover happens, end user have to collapse old the old nodes (that aren't master anymore) and find the latest one. It also makes harder to compare metrics over time for same alias(redis name).

[nantiferov]

I'll try to implement this feature in PR and test on real workloads to ensure that it actually provides what I'm looking for.
Right now it's mostly theory.

Thank you for help, I'll let you know when I have some results 🙂

[oliver006]

Ok, sounds good.
Reading your last response I think I see what the problem is with grafana and switching roles.

Maybe you can proof out what you're trying to do with a small, hacky PR and we'll take it from there.

[nantiferov]

Maybe you can proof out what you're trying to do with a small, hacky PR and we'll take it from there.

Sure, sounds good. Will check this after my vacation in couple of weeks.