[Bug]: Missing Labels on "KubeletDown" Alert

[Footur]

What happened?

When the KubeletDown alert fires, the generated alert does not contain the expected labels (e.g., cluster ID). This makes it hard to identify the affected cluster.

Probable cause

The current PromQL query (absent(up{job="kubelet"} == 1)) does not guarantee that labels are present if the metric is absent.

Proposed fix
Replace the current query with:

absent_over_time(up{job="kubelet"}[5m])

This should only fire when the kubelet metric is missing for a full 5‑minute window, and it preserves the original labels.

What parts of the codebase are affected?

Alerts

I agree to the following terms:

• I agree to follow this project's Code of Conduct.
• I have filled out all the required information above to the best of my ability.
• I have searched the issues of this repository and believe that this is not a duplicate.
• I have confirmed this bug exists in the default branch of the repository, as of the latest commit at the time of submission.

[skl]

Thanks @Footur! I tried a PR with a fix #1120, thought it seems removing the == 1 part is what actually keeps the job label, even with absent.

There is also a for duration of 15m on that alert, so it should only fire if the metric is missing for 15 minutes, so not actually sure we need the _over_time at all.

Now that I think a bit more about it, your suggestion and my PR don't cover the case where the scrape job is created successfully, but the scrape fails. In which case, there will be series like up{job="kubelet"} 0.

EDIT - I did some testing to validate this scenario. It seems up is 1 whilst the scrape is successful, then it goes to 0, then it is dropped completely. So actually I think your suggestion and my PR are ok! I attached a screenshot of what happens to the up series when a component goes down:

Updated my PR based on the above: 34ee366.

[Footur]

Hello @skl, unfortunately your approach hasn't fixed the issue. Can you reopen this report?

[skl]

Sure, could you be more specific about which labels are missing for you? I would expect that all labels present should be passed through to the alert now.

[Footur]

Sure, we are missing the value of the cluster label.

[skl]

@Footur, I see, that is tricky - as absent(up{job="kubelet"}) doesn't have a cluster label when the up series is indeed absent - the ONLY label it will have is job as that is the only label the absent check is looking for.

You would either have to create one absent check per cluster manually:

absent(up{job="kubelet", cluster="foo"})

Or we could try a different approach without absent and instead correlate against kube-state-metrics with a more elaborate query (returns one series per node, with a value of 0 if kubelet was found on that node or a value of 1 to trigger the alert if kubelet was not found):

1 - group by (cluster, node) (kube_node_info)
* on (cluster, node) group_left (job)
label_join(up{job="kubelet"}, "node", "", "instance")
or
group by (cluster, node, job) (label_replace(kube_node_info, "job", "kubelet", "job", ".*"))

The latter approach has the advantage of giving you both cluster and node labels as well, but depends on kube-state-metrics and so is not an independent check.

wdyt?

[github-actions]

This issue has not had any activity in the past 30 days, so the
stale label has been added to it.

• The stale label will be removed if there is new activity
• The issue will be closed in 7 days if there is no new activity
• Add the keepalive label to exempt this issue from the stale check action

Thank you for your contributions!