generated from onedr0p/cluster-template
-
Notifications
You must be signed in to change notification settings - Fork 0
feat(container)!: Update kube-prometheus-stack ( 70.10.0 → 79.2.1 ) #850
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
flowbie-bot
wants to merge
1
commit into
main
Choose a base branch
from
renovate/kube-prometheus-stack-79.x
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
--- kubernetes/apps/observability/kube-prometheus-stack/app Kustomization: flux-system/kube-prometheus-stack HelmRelease: observability/kube-prometheus-stack
+++ kubernetes/apps/observability/kube-prometheus-stack/app Kustomization: flux-system/kube-prometheus-stack HelmRelease: observability/kube-prometheus-stack
@@ -13,13 +13,13 @@
spec:
chart: kube-prometheus-stack
sourceRef:
kind: HelmRepository
name: prometheus-community
namespace: flux-system
- version: 70.10.0
+ version: 79.2.1
dependsOn:
- name: rook-ceph-cluster
namespace: rook-ceph
install:
remediation:
retries: 3 |
--- HelmRelease: observability/kube-prometheus-stack ConfigMap: observability/kube-prometheus-stack-grafana-overview
+++ HelmRelease: observability/kube-prometheus-stack ConfigMap: observability/kube-prometheus-stack-grafana-overview
@@ -27,13 +27,13 @@
job=~\"$job\"}[$__rate_interval])) by (le)) * 1","interval":"1m","legendFormat":"99th
Percentile","refId":"A"},{"exemplar":true,"expr":"histogram_quantile(0.50, sum(irate(grafana_http_request_duration_seconds_bucket{instance=~\"$instance\",
job=~\"$job\"}[$__rate_interval])) by (le)) * 1","interval":"1m","legendFormat":"50th
Percentile","refId":"B"},{"exemplar":true,"expr":"sum(irate(grafana_http_request_duration_seconds_sum{instance=~\"$instance\",
job=~\"$job\"}[$__rate_interval])) * 1 / sum(irate(grafana_http_request_duration_seconds_count{instance=~\"$instance\",
job=~\"$job\"}[$__rate_interval]))","interval":"1m","legendFormat":"Average","refId":"C"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Request
- Latency","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:210","format":"ms","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:211","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}}],"schemaVersion":30,"style":"dark","tags":[],"templating":{"list":[{"current":{"selected":true,"text":"dev-cortex","value":"dev-cortex"},"description":null,"error":null,"hide":0,"includeAll":false,"label":null,"multi":false,"name":"datasource","options":[],"query":"prometheus","queryValue":"","refresh":1,"regex":"","skipUrlSync":false,"type":"datasource"},{"allValue":".*","current":{"selected":false,"text":["default/grafana"],"value":["default/grafana"]},"datasource":"$datasource","definition":"label_values(grafana_build_info,
+ Latency","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:210","format":"ms","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:211","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}}],"schemaVersion":30,"tags":[],"templating":{"list":[{"current":{"selected":true,"text":"dev-cortex","value":"dev-cortex"},"description":null,"error":null,"hide":0,"includeAll":false,"label":null,"multi":false,"name":"datasource","options":[],"query":"prometheus","queryValue":"","refresh":1,"regex":"","skipUrlSync":false,"type":"datasource"},{"allValue":".*","current":{"selected":false,"text":["default/grafana"],"value":["default/grafana"]},"datasource":"$datasource","definition":"label_values(grafana_build_info,
job)","description":null,"error":null,"hide":0,"includeAll":true,"label":null,"multi":true,"name":"job","options":[],"query":{"query":"label_values(grafana_build_info,
job)","refId":"Billing Admin-job-Variable-Query"},"refresh":1,"regex":"","skipUrlSync":false,"sort":0,"tagValuesQuery":"","tagsQuery":"","type":"query","useTags":false},{"allValue":".*","current":{"selected":false,"text":"All","value":"$__all"},"datasource":"$datasource","definition":"label_values(grafana_build_info,
instance)","description":null,"error":null,"hide":0,"includeAll":true,"label":null,"multi":true,"name":"instance","options":[],"query":{"query":"label_values(grafana_build_info,
instance)","refId":"Billing Admin-instance-Variable-Query"},"refresh":1,"regex":"","skipUrlSync":false,"sort":0,"tagValuesQuery":"","tagsQuery":"","type":"query","useTags":false}]},"time":{"from":"now-6h","to":"now"},"timepicker":{"refresh_intervals":["10s","30s","1m","5m","15m","30m","1h","2h","1d"]},"timezone":
"utc","title":"Grafana Overview","uid":"6be0s85Mk","version":2}'
--- HelmRelease: observability/kube-prometheus-stack ConfigMap: observability/kube-prometheus-stack-k8s-coredns
+++ HelmRelease: observability/kube-prometheus-stack ConfigMap: observability/kube-prometheus-stack-k8s-coredns
@@ -28,54 +28,71 @@
by (zone)","interval":"1m","intervalFactor":2,"legendFormat":"{{ zone }}","refId":"A","step":60}],"title":"Requests
(by zone)","type":"timeseries"},{"datasource":{"uid":"$datasource"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":10,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"linear","lineWidth":2,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":true,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"links":[],"mappings":[],"min":0,"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"pps","unitScale":true},"overrides":[]},"gridPos":{"h":7,"w":12,"x":0,"y":7},"id":8,"links":[],"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true},"tooltip":{"mode":"multi","sort":"desc"}},"pluginVersion":"10.3.3","targets":[{"datasource":{"uid":"$datasource"},"expr":"sum(rate(coredns_dns_request_do_count_total{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\"}[5m]))
or\nsum(rate(coredns_dns_do_requests_total{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\"}[5m]))","interval":"1m","intervalFactor":2,"legendFormat":"DO","refId":"A","step":40},{"datasource":{"uid":"$datasource"},"expr":"sum(rate(coredns_dns_request_count_total{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\"}[5m]))
or\nsum(rate(coredns_dns_requests_total{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\"}[5m]))","interval":"1m","intervalFactor":2,"legendFormat":"total","refId":"B","step":40}],"title":"Requests
(DO bit)","type":"timeseries"},{"datasource":{"uid":"$datasource"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":10,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"linear","lineWidth":2,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":true,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"links":[],"mappings":[],"min":0,"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"bytes","unitScale":true},"overrides":[{"matcher":{"id":"byName","options":"tcp:90"},"properties":[{"id":"unit","value":"short"}]},{"matcher":{"id":"byName","options":"tcp:99
"},"properties":[{"id":"unit","value":"short"}]},{"matcher":{"id":"byName","options":"tcp:50"},"properties":[{"id":"unit","value":"short"}]}]},"gridPos":{"h":7,"w":6,"x":12,"y":7},"id":10,"links":[],"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"pluginVersion":"10.3.3","targets":[{"datasource":{"uid":"$datasource"},"expr":"histogram_quantile(0.99,
- sum(rate(coredns_dns_request_size_bytes_bucket{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"udp\"}[5m]))
- by (le,proto))","interval":"1m","intervalFactor":2,"legendFormat":"{{ proto }}:99
+ (sum(rate(coredns_dns_request_size_bytes{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"udp\"}[5m]))
+ by (proto)) or (sum(rate(coredns_dns_request_size_bytes_bucket{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"udp\"}[5m]))
+ by (le,proto)))","interval":"1m","intervalFactor":2,"legendFormat":"{{ proto }}:99
","refId":"A","step":60},{"datasource":{"uid":"$datasource"},"expr":"histogram_quantile(0.90,
- sum(rate(coredns_dns_request_size_bytes_bucket{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"udp\"}[5m]))
- by (le,proto))","intervalFactor":2,"legendFormat":"{{ proto }}:90","refId":"B","step":60},{"datasource":{"uid":"$datasource"},"expr":"histogram_quantile(0.50,
- sum(rate(coredns_dns_request_size_bytes_bucket{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"udp\"}[5m]))
- by (le,proto))","intervalFactor":2,"legendFormat":"{{ proto }}:50","refId":"C","step":60}],"title":"Requests
+ (sum(rate(coredns_dns_request_size_bytes{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"udp\"}[5m]))
+ by (proto)) or (sum(rate(coredns_dns_request_size_bytes_bucket{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"udp\"}[5m]))
+ by (le,proto)))","intervalFactor":2,"legendFormat":"{{ proto }}:90","refId":"B","step":60},{"datasource":{"uid":"$datasource"},"expr":"histogram_quantile(0.50,
+ (sum(rate(coredns_dns_request_size_bytes{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"udp\"}[5m]))
+ by (proto)) or (sum(rate(coredns_dns_request_size_bytes_bucket{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"udp\"}[5m]))
+ by (le,proto)))","intervalFactor":2,"legendFormat":"{{ proto }}:50","refId":"C","step":60}],"title":"Requests
(size, udp)","type":"timeseries"},{"datasource":{"uid":"$datasource"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":10,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"linear","lineWidth":2,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":true,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"links":[],"mappings":[],"min":0,"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"bytes","unitScale":true},"overrides":[]},"gridPos":{"h":7,"w":6,"x":18,"y":7},"id":12,"links":[],"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"pluginVersion":"10.3.3","targets":[{"datasource":{"uid":"$datasource"},"expr":"histogram_quantile(0.99,
- sum(rate(coredns_dns_request_size_bytes_bucket{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"tcp\"}[5m]))
- by (le,proto))","format":"time_series","interval":"1m","intervalFactor":2,"legendFormat":"{{
+ (sum(rate(coredns_dns_request_size_bytes{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"tcp\"}[5m]))
+ by (proto)) or (sum(rate(coredns_dns_request_size_bytes_bucket{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"tcp\"}[5m]))
+ by (le,proto)))","format":"time_series","interval":"1m","intervalFactor":2,"legendFormat":"{{
proto }}:99 ","refId":"A","step":60},{"datasource":{"uid":"$datasource"},"expr":"histogram_quantile(0.90,
- sum(rate(coredns_dns_request_size_bytes_bucket{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"tcp\"}[5m]))
- by (le,proto))","format":"time_series","interval":"1m","intervalFactor":2,"legendFormat":"{{
+ (sum(rate(coredns_dns_request_size_bytes{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"tcp\"}[5m]))
+ by (proto)) or (sum(rate(coredns_dns_request_size_bytes_bucket{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"tcp\"}[5m]))
+ by (le,proto)))","format":"time_series","interval":"1m","intervalFactor":2,"legendFormat":"{{
proto }}:90","refId":"B","step":60},{"datasource":{"uid":"$datasource"},"expr":"histogram_quantile(0.50,
- sum(rate(coredns_dns_request_size_bytes_bucket{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"tcp\"}[5m]))
- by (le,proto))","format":"time_series","interval":"1m","intervalFactor":2,"legendFormat":"{{
+ (sum(rate(coredns_dns_request_size_bytes{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"tcp\"}[5m]))
+ by (proto)) or (sum(rate(coredns_dns_request_size_bytes_bucket{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\",proto=\"tcp\"}[5m]))
+ by (le,proto)))","format":"time_series","interval":"1m","intervalFactor":2,"legendFormat":"{{
proto }}:50","refId":"C","step":60}],"title":"Requests (size,tcp)","type":"timeseries"},{"datasource":{"uid":"$datasource"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisBorderShow":false,"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":10,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"insertNulls":false,"lineInterpolation":"linear","lineWidth":2,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":true,"stacking":{"group":"A","mode":"normal"},"thresholdsStyle":{"mode":"off"}},"links":[],"mappings":[],"min":0,"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"pps","unitScale":true},"overrides":[]},"gridPos":{"h":7,"w":12,"x":0,"y":14},"id":14,"links":[],"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true},"tooltip":{"mode":"multi","sort":"desc"}},"pluginVersion":"10.3.3","targets":[{"datasource":{"uid":"$datasource"},"expr":"sum(rate(coredns_dns_response_rcode_count_total{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\"}[5m]))
by (rcode) or\nsum(rate(coredns_dns_responses_total{job=~\"$job\",cluster=~\"$cluster\",instance=~\"$instance\"}[5m]))
[Diff truncated by flux-local]
--- HelmRelease: observability/kube-prometheus-stack ConfigMap: observability/kube-prometheus-stack-prometheus
+++ HelmRelease: observability/kube-prometheus-stack ConfigMap: observability/kube-prometheus-stack-prometheus
@@ -25,13 +25,13 @@
by (cluster, job, scrape_job, instance) * 1e3","format":"time_series","legendFormat":"{{cluster}}:{{job}}:{{instance}}:{{scrape_job}}"}],"title":"Target
Sync","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"$datasource"},"fieldConfig":{"defaults":{"custom":{"fillOpacity":100,"lineWidth":0,"showPoints":"never","stacking":{"mode":"normal"}},"min":0,"unit":"short"}},"gridPos":{"h":7,"w":12,"x":12,"y":9},"id":5,"options":{"tooltip":{"mode":"multi","sort":"desc"}},"pluginVersion":"v11.4.0","targets":[{"datasource":{"type":"prometheus","uid":"$datasource"},"expr":"sum
by (cluster, job, instance) (prometheus_sd_discovered_targets{cluster=~\"$cluster\",
job=~\"$job\",instance=~\"$instance\"})","format":"time_series","legendFormat":"{{cluster}}:{{job}}:{{instance}}"}],"title":"Targets","type":"timeseries"},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":16},"id":6,"panels":[],"title":"Retrieval","type":"row"},{"datasource":{"type":"prometheus","uid":"$datasource"},"fieldConfig":{"defaults":{"custom":{"fillOpacity":10,"showPoints":"never"},"min":0,"unit":"ms"}},"gridPos":{"h":7,"w":8,"x":0,"y":17},"id":7,"options":{"tooltip":{"mode":"multi","sort":"desc"}},"pluginVersion":"v11.4.0","targets":[{"datasource":{"type":"prometheus","uid":"$datasource"},"expr":"rate(prometheus_target_interval_length_seconds_sum{cluster=~\"$cluster\",
job=~\"$job\",instance=~\"$instance\"}[5m]) / rate(prometheus_target_interval_length_seconds_count{cluster=~\"$cluster\",
job=~\"$job\",instance=~\"$instance\"}[5m]) * 1e3","format":"time_series","legendFormat":"{{cluster}}:{{job}}:{{instance}}
- {{interval}} configured"}],"title":"Average Scrape Interval Duration","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"$datasource"},"fieldConfig":{"defaults":{"custom":{"fillOpacity":100,"lineWidth":0,"showPoints":"never","stacking":{"mode":"normal"}},"min":0,"unit":"ms"}},"gridPos":{"h":7,"w":8,"x":8,"y":17},"id":8,"options":{"tooltip":{"mode":"multi","sort":"desc"}},"pluginVersion":"v11.4.0","targets":[{"datasource":{"type":"prometheus","uid":"$datasource"},"expr":"sum
+ {{interval}} configured"}],"title":"Average Scrape Interval Duration","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"$datasource"},"fieldConfig":{"defaults":{"custom":{"fillOpacity":100,"lineWidth":0,"showPoints":"never","stacking":{"mode":"normal"}},"min":0,"unit":"short"}},"gridPos":{"h":7,"w":8,"x":8,"y":17},"id":8,"options":{"tooltip":{"mode":"multi","sort":"desc"}},"pluginVersion":"v11.4.0","targets":[{"datasource":{"type":"prometheus","uid":"$datasource"},"expr":"sum
by (cluster, job, instance) (rate(prometheus_target_scrapes_exceeded_body_size_limit_total{cluster=~\"$cluster\",job=~\"$job\",instance=~\"$instance\"}[1m]))","format":"time_series","legendFormat":"exceeded
body size limit: {{cluster}} {{job}} {{instance}}"},{"datasource":{"type":"prometheus","uid":"$datasource"},"expr":"sum
by (cluster, job, instance) (rate(prometheus_target_scrapes_exceeded_sample_limit_total{cluster=~\"$cluster\",job=~\"$job\",instance=~\"$instance\"}[1m]))","format":"time_series","legendFormat":"exceeded
sample limit: {{cluster}} {{job}} {{instance}}"},{"datasource":{"type":"prometheus","uid":"$datasource"},"expr":"sum
by (cluster, job, instance) (rate(prometheus_target_scrapes_sample_duplicate_timestamp_total{cluster=~\"$cluster\",job=~\"$job\",instance=~\"$instance\"}[1m]))","format":"time_series","legendFormat":"duplicate
timestamp: {{cluster}} {{job}} {{instance}}"},{"datasource":{"type":"prometheus","uid":"$datasource"},"expr":"sum
@@ -48,8 +48,8 @@
job=~\"$job\",instance=~\"$instance\"}) * 1e3","format":"time_series","legendFormat":"{{slice}}"}],"title":"Stage
Duration","type":"timeseries"}],"schemaVersion":39,"tags":["prometheus-mixin"],"templating":{"list":[{"current":{"selected":false,"text":"default","value":"default"},"hide":0,"label":"Data
source","name":"datasource","query":"prometheus","type":"datasource"},{"allValue":".*","current":{"selected":false,"text":["$__all"],"value":["$__all"]},"datasource":{"type":"prometheus","uid":"${datasource}"},"hide":2,"includeAll":true,"label":"cluster","multi":true,"name":"cluster","query":"label_values(prometheus_build_info{},
cluster)","refresh":2,"sort":2,"type":"query"},{"allValue":".+","datasource":{"type":"prometheus","uid":"${datasource}"},"includeAll":true,"label":"job","multi":true,"name":"job","query":"label_values(prometheus_build_info{cluster=~\"$cluster\"},
job)","refresh":2,"sort":2,"type":"query"},{"allValue":".+","datasource":{"type":"prometheus","uid":"${datasource}"},"includeAll":true,"label":"instance","multi":true,"name":"instance","query":"label_values(prometheus_build_info{cluster=~\"$cluster\",
job=~\"$job\"}, instance)","refresh":2,"sort":2,"type":"query"}]},"time":{"from":"now-1h","to":"now"},"timepicker":{"refresh_intervals":["60s"]},"timezone":
- "utc","title":"Prometheus / Overview"}'
+ "utc","title":"Prometheus / Overview","uid":"9fa0d141-d019-4ad7-8bc5-42196ee308bd"}'
--- HelmRelease: observability/kube-prometheus-stack ClusterRole: observability/kube-state-metrics
+++ HelmRelease: observability/kube-prometheus-stack ClusterRole: observability/kube-state-metrics
@@ -30,21 +30,19 @@
resources:
- cronjobs
verbs:
- list
- watch
- apiGroups:
- - extensions
- apps
resources:
- daemonsets
verbs:
- list
- watch
- apiGroups:
- - extensions
- apps
resources:
- deployments
verbs:
- list
- watch
@@ -60,13 +58,12 @@
resources:
- horizontalpodautoscalers
verbs:
- list
- watch
- apiGroups:
- - extensions
- networking.k8s.io
resources:
- ingresses
verbs:
- list
- watch
@@ -145,13 +142,12 @@
resources:
- pods
verbs:
- list
- watch
- apiGroups:
- - extensions
- apps
resources:
- replicasets
verbs:
- list
- watch
--- HelmRelease: observability/kube-prometheus-stack ClusterRole: observability/kube-prometheus-stack-operator
+++ HelmRelease: observability/kube-prometheus-stack ClusterRole: observability/kube-prometheus-stack-operator
@@ -27,15 +27,19 @@
- prometheusagents/finalizers
- prometheusagents/status
- thanosrulers
- thanosrulers/finalizers
- thanosrulers/status
- scrapeconfigs
+ - scrapeconfigs/status
- servicemonitors
+ - servicemonitors/status
- podmonitors
+ - podmonitors/status
- probes
+ - probes/status
- prometheusrules
verbs:
- '*'
- apiGroups:
- apps
resources:
@@ -81,12 +85,13 @@
verbs:
- get
- list
- watch
- apiGroups:
- ''
+ - events.k8s.io
resources:
- events
verbs:
- patch
- create
- apiGroups:
@@ -100,7 +105,18 @@
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
verbs:
- get
+- apiGroups:
+ - discovery.k8s.io
+ resources:
+ - endpointslices
+ verbs:
+ - get
+ - create
+ - list
+ - watch
+ - update
+ - delete
--- HelmRelease: observability/kube-prometheus-stack Service: observability/kube-state-metrics
+++ HelmRelease: observability/kube-prometheus-stack Service: observability/kube-state-metrics
@@ -14,11 +14,11 @@
spec:
type: ClusterIP
ports:
- name: http
protocol: TCP
port: 8080
- targetPort: 8080
+ targetPort: http
selector:
app.kubernetes.io/name: kube-state-metrics
app.kubernetes.io/instance: kube-prometheus-stack
--- HelmRelease: observability/kube-prometheus-stack DaemonSet: observability/node-exporter
+++ HelmRelease: observability/kube-prometheus-stack DaemonSet: observability/node-exporter
@@ -40,22 +40,22 @@
runAsGroup: 65534
runAsNonRoot: true
runAsUser: 65534
serviceAccountName: node-exporter
containers:
- name: node-exporter
- image: quay.io/prometheus/node-exporter:v1.9.1
+ image: quay.io/prometheus/node-exporter:v1.10.2
imagePullPolicy: IfNotPresent
args:
- --path.procfs=/host/proc
- --path.sysfs=/host/sys
- --path.rootfs=/host/root
- --path.udev.data=/host/root/run/udev/data
- --web.listen-address=[$(HOST_IP)]:9100
- --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
- - --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
+ - --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs|erofs)$
securityContext:
readOnlyRootFilesystem: true
env:
- name: HOST_IP
value: 0.0.0.0
ports:
@@ -64,24 +64,24 @@
protocol: TCP
livenessProbe:
failureThreshold: 3
httpGet:
httpHeaders: null
path: /
- port: 9100
+ port: http-metrics
scheme: HTTP
initialDelaySeconds: 0
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
httpHeaders: null
path: /
- port: 9100
+ port: http-metrics
scheme: HTTP
initialDelaySeconds: 0
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
volumeMounts:
--- HelmRelease: observability/kube-prometheus-stack Deployment: observability/kube-state-metrics
+++ HelmRelease: observability/kube-prometheus-stack Deployment: observability/kube-state-metrics
@@ -45,13 +45,13 @@
- name: kube-state-metrics
args:
- --port=8080
- --resources=certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments
- --metric-labels-allowlist=pods=[*],deployments=[*],persistentvolumeclaims=[*]
imagePullPolicy: IfNotPresent
- image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.15.0
+ image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.17.0
ports:
- containerPort: 8080
name: http
livenessProbe:
failureThreshold: 3
httpGet:
--- HelmRelease: observability/kube-prometheus-stack Deployment: observability/kube-prometheus-stack-operator
+++ HelmRelease: observability/kube-prometheus-stack Deployment: observability/kube-prometheus-stack-operator
@@ -31,25 +31,25 @@
app: kube-prometheus-stack-operator
app.kubernetes.io/name: kube-prometheus-stack-prometheus-operator
app.kubernetes.io/component: prometheus-operator
spec:
containers:
- name: kube-prometheus-stack
- image: quay.io/prometheus-operator/prometheus-operator:v0.81.0
+ image: quay.io/prometheus-operator/prometheus-operator:v0.86.1
imagePullPolicy: IfNotPresent
args:
- --kubelet-service=kube-system/kube-prometheus-stack-kubelet
- --kubelet-endpoints=true
- --kubelet-endpointslice=false
- --localhost=127.0.0.1
- - --prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.81.0
+ - --prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.86.1
- --config-reloader-cpu-request=0
- --config-reloader-cpu-limit=0
- --config-reloader-memory-request=0
- --config-reloader-memory-limit=0
- - --thanos-default-base-image=quay.io/thanos/thanos:v0.38.0
+ - --thanos-default-base-image=quay.io/thanos/thanos:v0.40.1
- --secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1
- --web.enable-tls=true
- --web.cert-file=/cert/cert
- --web.key-file=/cert/key
- --web.listen-address=:10250
- --web.tls-min-version=VersionTLS13
--- HelmRelease: observability/kube-prometheus-stack Alertmanager: observability/kube-prometheus-stack
+++ HelmRelease: observability/kube-prometheus-stack Alertmanager: observability/kube-prometheus-stack
@@ -9,14 +9,15 @@
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/instance: kube-prometheus-stack
app.kubernetes.io/part-of: kube-prometheus-stack
release: kube-prometheus-stack
heritage: Helm
spec:
- image: quay.io/prometheus/alertmanager:v0.28.1
- version: v0.28.1
+ image: quay.io/prometheus/alertmanager:v0.29.0
+ imagePullPolicy: IfNotPresent
+ version: v0.29.0
replicas: 1
listenLocal: false
serviceAccountName: kube-prometheus-stack-alertmanager
automountServiceAccountToken: true
externalUrl: https://alertmanager...PLACEHOLDER_SECRET_DOMAIN..
paused: false
--- HelmRelease: observability/kube-prometheus-stack List: observability/kube-prometheus-stack-additional-prometheus-rules
+++ HelmRelease: observability/kube-prometheus-stack List: observability/kube-prometheus-stack-additional-prometheus-rules
@@ -1,79 +0,0 @@
----
-apiVersion: v1
-kind: List
-metadata:
- name: kube-prometheus-stack-additional-prometheus-rules
- namespace: observability
-items:
-- apiVersion: monitoring.coreos.com/v1
- kind: PrometheusRule
- metadata:
- name: kube-prometheus-stack-dockerhub-rules
- namespace: observability
- labels:
- app: kube-prometheus-stack
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/instance: kube-prometheus-stack
- app.kubernetes.io/part-of: kube-prometheus-stack
- release: kube-prometheus-stack
- heritage: Helm
- spec:
- groups:
- - name: dockerhub
- rules:
- - alert: DockerhubRateLimitRisk
- annotations:
- summary: Kubernetes cluster Dockerhub rate limit risk
- expr: count(time() - container_last_seen{image=~"(docker.io).*",container!=""}
- < 30) > 100
- labels:
- severity: critical
-- apiVersion: monitoring.coreos.com/v1
- kind: PrometheusRule
- metadata:
- name: kube-prometheus-stack-oom-rules
- namespace: observability
- labels:
- app: kube-prometheus-stack
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/instance: kube-prometheus-stack
- app.kubernetes.io/part-of: kube-prometheus-stack
- release: kube-prometheus-stack
- heritage: Helm
- spec:
- groups:
- - name: oom
- rules:
- - alert: OomKilled
- annotations:
- summary: Container {{ $labels.container }} in pod {{ $labels.namespace }}/{{
- $labels.pod }} has been OOMKilled {{ $value }} times in the last 10 minutes.
- expr: (kube_pod_container_status_restarts_total - kube_pod_container_status_restarts_total
- offset 10m >= 1) and ignoring (reason) min_over_time(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[10m])
- == 1
- labels:
- severity: critical
-- apiVersion: monitoring.coreos.com/v1
- kind: PrometheusRule
- metadata:
- name: kube-prometheus-stack-zfs-rules
- namespace: observability
- labels:
- app: kube-prometheus-stack
- app.kubernetes.io/managed-by: Helm
- app.kubernetes.io/instance: kube-prometheus-stack
- app.kubernetes.io/part-of: kube-prometheus-stack
- release: kube-prometheus-stack
- heritage: Helm
- spec:
- groups:
- - name: zfs
- rules:
- - alert: ZfsUnexpectedPoolState
- annotations:
- summary: ZFS pool {{$labels.zpool}} on {{$labels.instance}} is in a unexpected
- state {{$labels.state}}
- expr: node_zfs_zpool_state{state!="online"} > 0
- labels:
- severity: critical
-
--- HelmRelease: observability/kube-prometheus-stack Prometheus: observability/kube-prometheus-stack
+++ HelmRelease: observability/kube-prometheus-stack Prometheus: observability/kube-prometheus-stack
@@ -17,21 +17,23 @@
alertmanagers:
- namespace: observability
name: kube-prometheus-stack-alertmanager
port: http-web
pathPrefix: /
apiVersion: v2
- image: quay.io/prometheus/prometheus:v3.3.0
- version: v3.3.0
+ image: quay.io/prometheus/prometheus:v3.7.3
+ imagePullPolicy: IfNotPresent
+ version: v3.7.3
externalUrl: http://prometheus...PLACEHOLDER_SECRET_DOMAIN../
paused: false
replicas: 1
shards: 1
logLevel: info
logFormat: logfmt
listenLocal: false
+ enableOTLPReceiver: false
enableAdminAPI: true
enableFeatures:
- memory-snapshot-on-shutdown
resources:
limits:
memory: 2000Mi
@@ -77,13 +79,13 @@
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- prometheus
- - key: prometheus
+ - key: app.kubernetes.io/instance
operator: In
values:
- kube-prometheus-stack
portName: http-web
hostNetwork: false
--- HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-alertmanager.rules
+++ HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-alertmanager.rules
@@ -21,13 +21,13 @@
$labels.pod}}.
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/alertmanager/alertmanagerfailedreload
summary: Reloading an Alertmanager configuration has failed.
expr: |-
# Without max_over_time, failed scrapes could create false negatives, see
# https://www.robustperception.io/alerting-on-gauges-in-prometheus-2-0 for details.
- max_over_time(alertmanager_config_last_reload_successful{job="kube-prometheus-stack-alertmanager",namespace="observability"}[5m]) == 0
+ max_over_time(alertmanager_config_last_reload_successful{job="kube-prometheus-stack-alertmanager",container="alertmanager",namespace="observability"}[5m]) == 0
for: 10m
labels:
severity: critical
- alert: AlertmanagerMembersInconsistent
annotations:
description: Alertmanager {{ $labels.namespace }}/{{ $labels.pod}} has only
@@ -35,30 +35,30 @@
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/alertmanager/alertmanagermembersinconsistent
summary: A member of an Alertmanager cluster has not found all other cluster
members.
expr: |-
# Without max_over_time, failed scrapes could create false negatives, see
# https://www.robustperception.io/alerting-on-gauges-in-prometheus-2-0 for details.
- max_over_time(alertmanager_cluster_members{job="kube-prometheus-stack-alertmanager",namespace="observability"}[5m])
+ max_over_time(alertmanager_cluster_members{job="kube-prometheus-stack-alertmanager",container="alertmanager",namespace="observability"}[5m])
< on (namespace,service,cluster) group_left
- count by (namespace,service,cluster) (max_over_time(alertmanager_cluster_members{job="kube-prometheus-stack-alertmanager",namespace="observability"}[5m]))
+ count by (namespace,service,cluster) (max_over_time(alertmanager_cluster_members{job="kube-prometheus-stack-alertmanager",container="alertmanager",namespace="observability"}[5m]))
for: 15m
labels:
severity: critical
- alert: AlertmanagerFailedToSendAlerts
annotations:
description: Alertmanager {{ $labels.namespace }}/{{ $labels.pod}} failed
to send {{ $value | humanizePercentage }} of notifications to {{ $labels.integration
}}.
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/alertmanager/alertmanagerfailedtosendalerts
summary: An Alertmanager instance failed to send notifications.
expr: |-
(
- rate(alertmanager_notifications_failed_total{job="kube-prometheus-stack-alertmanager",namespace="observability"}[15m])
+ rate(alertmanager_notifications_failed_total{job="kube-prometheus-stack-alertmanager",container="alertmanager",namespace="observability"}[15m])
/
- ignoring (reason) group_left rate(alertmanager_notifications_total{job="kube-prometheus-stack-alertmanager",namespace="observability"}[15m])
+ ignoring (reason) group_left rate(alertmanager_notifications_total{job="kube-prometheus-stack-alertmanager",container="alertmanager",namespace="observability"}[15m])
)
> 0.01
for: 5m
labels:
severity: warning
- alert: AlertmanagerClusterFailedToSendAlerts
@@ -68,15 +68,15 @@
humanizePercentage }}.
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/alertmanager/alertmanagerclusterfailedtosendalerts
summary: All Alertmanager instances in a cluster failed to send notifications
to a critical integration.
expr: |-
min by (namespace,service, integration) (
- rate(alertmanager_notifications_failed_total{job="kube-prometheus-stack-alertmanager",namespace="observability", integration=~`.*`}[15m])
+ rate(alertmanager_notifications_failed_total{job="kube-prometheus-stack-alertmanager",container="alertmanager",namespace="observability", integration=~`.*`}[15m])
/
- ignoring (reason) group_left rate(alertmanager_notifications_total{job="kube-prometheus-stack-alertmanager",namespace="observability", integration=~`.*`}[15m])
+ ignoring (reason) group_left rate(alertmanager_notifications_total{job="kube-prometheus-stack-alertmanager",container="alertmanager",namespace="observability", integration=~`.*`}[15m])
)
> 0.01
for: 5m
labels:
severity: critical
- alert: AlertmanagerClusterFailedToSendAlerts
@@ -86,15 +86,15 @@
humanizePercentage }}.
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/alertmanager/alertmanagerclusterfailedtosendalerts
summary: All Alertmanager instances in a cluster failed to send notifications
to a non-critical integration.
expr: |-
min by (namespace,service, integration) (
- rate(alertmanager_notifications_failed_total{job="kube-prometheus-stack-alertmanager",namespace="observability", integration!~`.*`}[15m])
+ rate(alertmanager_notifications_failed_total{job="kube-prometheus-stack-alertmanager",container="alertmanager",namespace="observability", integration!~`.*`}[15m])
/
- ignoring (reason) group_left rate(alertmanager_notifications_total{job="kube-prometheus-stack-alertmanager",namespace="observability", integration!~`.*`}[15m])
+ ignoring (reason) group_left rate(alertmanager_notifications_total{job="kube-prometheus-stack-alertmanager",container="alertmanager",namespace="observability", integration!~`.*`}[15m])
)
> 0.01
for: 5m
labels:
severity: warning
- alert: AlertmanagerConfigInconsistent
@@ -102,13 +102,13 @@
description: Alertmanager instances within the {{$labels.job}} cluster have
different configurations.
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/alertmanager/alertmanagerconfiginconsistent
summary: Alertmanager instances within the same cluster have different configurations.
expr: |-
count by (namespace,service,cluster) (
- count_values by (namespace,service,cluster) ("config_hash", alertmanager_config_hash{job="kube-prometheus-stack-alertmanager",namespace="observability"})
+ count_values by (namespace,service,cluster) ("config_hash", alertmanager_config_hash{job="kube-prometheus-stack-alertmanager",container="alertmanager",namespace="observability"})
)
!= 1
for: 20m
labels:
severity: critical
- alert: AlertmanagerClusterDown
@@ -119,17 +119,17 @@
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/alertmanager/alertmanagerclusterdown
summary: Half or more of the Alertmanager instances within the same cluster
are down.
expr: |-
(
count by (namespace,service,cluster) (
- avg_over_time(up{job="kube-prometheus-stack-alertmanager",namespace="observability"}[5m]) < 0.5
+ avg_over_time(up{job="kube-prometheus-stack-alertmanager",container="alertmanager",namespace="observability"}[5m]) < 0.5
)
/
count by (namespace,service,cluster) (
- up{job="kube-prometheus-stack-alertmanager",namespace="observability"}
+ up{job="kube-prometheus-stack-alertmanager",container="alertmanager",namespace="observability"}
)
)
>= 0.5
for: 5m
labels:
severity: critical
@@ -141,17 +141,17 @@
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/alertmanager/alertmanagerclustercrashlooping
summary: Half or more of the Alertmanager instances within the same cluster
are crashlooping.
expr: |-
(
count by (namespace,service,cluster) (
- changes(process_start_time_seconds{job="kube-prometheus-stack-alertmanager",namespace="observability"}[10m]) > 4
+ changes(process_start_time_seconds{job="kube-prometheus-stack-alertmanager",container="alertmanager",namespace="observability"}[10m]) > 4
)
/
count by (namespace,service,cluster) (
- up{job="kube-prometheus-stack-alertmanager",namespace="observability"}
+ up{job="kube-prometheus-stack-alertmanager",container="alertmanager",namespace="observability"}
)
)
>= 0.5
for: 5m
labels:
severity: critical
--- HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-k8s.rules.pod-owner
+++ HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-k8s.rules.pod-owner
@@ -18,15 +18,32 @@
- expr: |-
max by (cluster, namespace, workload, pod) (
label_replace(
label_replace(
kube_pod_owner{job="kube-state-metrics", owner_kind="ReplicaSet"},
"replicaset", "$1", "owner_name", "(.*)"
- ) * on (replicaset, namespace) group_left(owner_name) topk by (replicaset, namespace) (
- 1, max by (replicaset, namespace, owner_name) (
- kube_replicaset_owner{job="kube-state-metrics"}
+ ) * on (cluster, replicaset, namespace) group_left(owner_name) topk by (cluster, replicaset, namespace) (
+ 1, max by (cluster, replicaset, namespace, owner_name) (
+ kube_replicaset_owner{job="kube-state-metrics", owner_kind=""}
+ )
+ ),
+ "workload", "$1", "replicaset", "(.*)"
+ )
+ )
+ labels:
+ workload_type: replicaset
+ record: namespace_workload_pod:kube_pod_owner:relabel
+ - expr: |-
+ max by (cluster, namespace, workload, pod) (
+ label_replace(
+ label_replace(
+ kube_pod_owner{job="kube-state-metrics", owner_kind="ReplicaSet"},
+ "replicaset", "$1", "owner_name", "(.*)"
+ ) * on (replicaset, namespace, cluster) group_left(owner_name) topk by (cluster, replicaset, namespace) (
+ 1, max by (cluster, replicaset, namespace, owner_name) (
+ kube_replicaset_owner{job="kube-state-metrics", owner_kind="Deployment"}
)
),
"workload", "$1", "owner_name", "(.*)"
)
)
labels:
@@ -43,23 +60,86 @@
workload_type: daemonset
record: namespace_workload_pod:kube_pod_owner:relabel
- expr: |-
max by (cluster, namespace, workload, pod) (
label_replace(
kube_pod_owner{job="kube-state-metrics", owner_kind="StatefulSet"},
- "workload", "$1", "owner_name", "(.*)"
- )
+ "workload", "$1", "owner_name", "(.*)")
)
labels:
workload_type: statefulset
record: namespace_workload_pod:kube_pod_owner:relabel
- expr: |-
- max by (cluster, namespace, workload, pod) (
- label_replace(
- kube_pod_owner{job="kube-state-metrics", owner_kind="Job"},
- "workload", "$1", "owner_name", "(.*)"
- )
+ group by (cluster, namespace, workload, pod) (
+ label_join(
+ group by (cluster, namespace, job_name, pod, owner_name) (
+ label_join(
+ kube_pod_owner{job="kube-state-metrics", owner_kind="Job"}
+ , "job_name", "", "owner_name")
+ )
+ * on (cluster, namespace, job_name) group_left()
+ group by (cluster, namespace, job_name) (
+ kube_job_owner{job="kube-state-metrics", owner_kind=~"Pod|"}
+ )
+ , "workload", "", "owner_name")
)
labels:
workload_type: job
record: namespace_workload_pod:kube_pod_owner:relabel
+ - expr: |-
+ max by (cluster, namespace, workload, pod) (
+ label_replace(
+ kube_pod_owner{job="kube-state-metrics", owner_kind="", owner_name=""},
+ "workload", "$1", "pod", "(.+)")
+ )
+ labels:
+ workload_type: barepod
+ record: namespace_workload_pod:kube_pod_owner:relabel
+ - expr: |-
+ max by (cluster, namespace, workload, pod) (
+ label_replace(
+ kube_pod_owner{job="kube-state-metrics", owner_kind="Node"},
+ "workload", "$1", "pod", "(.+)")
+ )
+ labels:
+ workload_type: staticpod
+ record: namespace_workload_pod:kube_pod_owner:relabel
+ - expr: |-
+ group by (cluster, namespace, workload, workload_type, pod) (
+ label_join(
+ label_join(
+ group by (cluster, namespace, job_name, pod) (
+ label_join(
+ kube_pod_owner{job="kube-state-metrics", owner_kind="Job"}
+ , "job_name", "", "owner_name")
+ )
+ * on (cluster, namespace, job_name) group_left(owner_kind, owner_name)
+ group by (cluster, namespace, job_name, owner_kind, owner_name) (
+ kube_job_owner{job="kube-state-metrics", owner_kind!="Pod", owner_kind!=""}
+ )
+ , "workload", "", "owner_name")
+ , "workload_type", "", "owner_kind")
+ OR
+
+ label_replace(
+ label_replace(
+ label_replace(
+ kube_pod_owner{job="kube-state-metrics", owner_kind="ReplicaSet"}
+ , "replicaset", "$1", "owner_name", "(.+)"
+ )
+ * on (cluster, namespace, replicaset) group_left(owner_kind, owner_name)
+ group by (cluster, namespace, replicaset, owner_kind, owner_name) (
+ kube_replicaset_owner{job="kube-state-metrics", owner_kind!="Deployment", owner_kind!=""}
+ )
+ , "workload", "$1", "owner_name", "(.+)")
+ OR
+ label_replace(
+ group by (cluster, namespace, pod, owner_name, owner_kind) (
+ kube_pod_owner{job="kube-state-metrics", owner_kind!="ReplicaSet", owner_kind!="DaemonSet", owner_kind!="StatefulSet", owner_kind!="Job", owner_kind!="Node", owner_kind!=""}
+ )
+ , "workload", "$1", "owner_name", "(.+)"
+ )
+ , "workload_type", "$1", "owner_kind", "(.+)")
+ )
+ record: namespace_workload_pod:kube_pod_owner:relabel
+
--- HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-kubelet.rules
+++ HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-kubelet.rules
@@ -12,25 +12,37 @@
release: kube-prometheus-stack
heritage: Helm
spec:
groups:
- name: kubelet.rules
rules:
- - expr: histogram_quantile(0.99, sum(rate(kubelet_pleg_relist_duration_seconds_bucket{job="kubelet",
- metrics_path="/metrics"}[5m])) by (cluster, instance, le) * on (cluster, instance)
- group_left(node) kubelet_node_name{job="kubelet", metrics_path="/metrics"})
+ - expr: |-
+ histogram_quantile(
+ 0.99,
+ sum(rate(kubelet_pleg_relist_duration_seconds_bucket{job="kubelet", metrics_path="/metrics"}[5m])) by (cluster, instance, le)
+ * on (cluster, instance) group_left (node)
+ max by (cluster, instance, node) (kubelet_node_name{job="kubelet", metrics_path="/metrics"})
+ )
labels:
quantile: '0.99'
record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile
- - expr: histogram_quantile(0.9, sum(rate(kubelet_pleg_relist_duration_seconds_bucket{job="kubelet",
- metrics_path="/metrics"}[5m])) by (cluster, instance, le) * on (cluster, instance)
- group_left(node) kubelet_node_name{job="kubelet", metrics_path="/metrics"})
+ - expr: |-
+ histogram_quantile(
+ 0.9,
+ sum(rate(kubelet_pleg_relist_duration_seconds_bucket{job="kubelet", metrics_path="/metrics"}[5m])) by (cluster, instance, le)
+ * on (cluster, instance) group_left (node)
+ max by (cluster, instance, node) (kubelet_node_name{job="kubelet", metrics_path="/metrics"})
+ )
labels:
quantile: '0.9'
record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile
- - expr: histogram_quantile(0.5, sum(rate(kubelet_pleg_relist_duration_seconds_bucket{job="kubelet",
- metrics_path="/metrics"}[5m])) by (cluster, instance, le) * on (cluster, instance)
- group_left(node) kubelet_node_name{job="kubelet", metrics_path="/metrics"})
+ - expr: |-
+ histogram_quantile(
+ 0.5,
+ sum(rate(kubelet_pleg_relist_duration_seconds_bucket{job="kubelet", metrics_path="/metrics"}[5m])) by (cluster, instance, le)
+ * on (cluster, instance) group_left (node)
+ max by (cluster, instance, node) (kubelet_node_name{job="kubelet", metrics_path="/metrics"})
+ )
labels:
quantile: '0.5'
record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile
--- HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-kubernetes-apps
+++ HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-kubernetes-apps
@@ -135,19 +135,19 @@
(
max by (namespace, statefulset, job, cluster) (
kube_statefulset_status_current_revision{job="kube-state-metrics", namespace=~".*"}
unless
kube_statefulset_status_update_revision{job="kube-state-metrics", namespace=~".*"}
)
- *
+ * on (namespace, statefulset, job, cluster)
(
kube_statefulset_replicas{job="kube-state-metrics", namespace=~".*"}
!=
kube_statefulset_status_replicas_updated{job="kube-state-metrics", namespace=~".*"}
)
- ) and (
+ ) and on (namespace, statefulset, job, cluster) (
changes(kube_statefulset_status_replicas_updated{job="kube-state-metrics", namespace=~".*"}[5m])
==
0
)
for: 15m
labels:
@@ -160,25 +160,25 @@
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubedaemonsetrolloutstuck
summary: DaemonSet rollout is stuck.
expr: |-
(
(
kube_daemonset_status_current_number_scheduled{job="kube-state-metrics", namespace=~".*"}
- !=
+ !=
kube_daemonset_status_desired_number_scheduled{job="kube-state-metrics", namespace=~".*"}
) or (
kube_daemonset_status_number_misscheduled{job="kube-state-metrics", namespace=~".*"}
- !=
+ !=
0
) or (
kube_daemonset_status_updated_number_scheduled{job="kube-state-metrics", namespace=~".*"}
- !=
+ !=
kube_daemonset_status_desired_number_scheduled{job="kube-state-metrics", namespace=~".*"}
) or (
kube_daemonset_status_number_available{job="kube-state-metrics", namespace=~".*"}
- !=
+ !=
kube_daemonset_status_desired_number_scheduled{job="kube-state-metrics", namespace=~".*"}
)
) and (
changes(kube_daemonset_status_updated_number_scheduled{job="kube-state-metrics", namespace=~".*"}[5m])
==
0
@@ -282,7 +282,24 @@
kube_horizontalpodautoscaler_status_current_replicas{job="kube-state-metrics", namespace=~".*"}
==
kube_horizontalpodautoscaler_spec_max_replicas{job="kube-state-metrics", namespace=~".*"}
for: 15m
labels:
severity: warning
+ - alert: KubePdbNotEnoughHealthyPods
+ annotations:
+ description: PDB {{ $labels.cluster }}/{{ $labels.namespace }}/{{ $labels.poddisruptionbudget
+ }} expects {{ $value }} more healthy pods. The desired number of healthy
+ pods has not been met for at least 15m.
+ runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubepdbnotenoughhealthypods
+ summary: PDB does not have enough healthy pods.
+ expr: |-
+ (
+ kube_poddisruptionbudget_status_desired_healthy{job="kube-state-metrics", namespace=~".*"}
+ -
+ kube_poddisruptionbudget_status_current_healthy{job="kube-state-metrics", namespace=~".*"}
+ )
+ > 0
+ for: 15m
+ labels:
+ severity: warning
--- HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-kubernetes-resources
+++ HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-kubernetes-resources
@@ -20,57 +20,105 @@
description: Cluster {{ $labels.cluster }} has overcommitted CPU resource
requests for Pods by {{ printf "%.2f" $value }} CPU shares and cannot tolerate
node failure.
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubecpuovercommit
summary: Cluster has overcommitted CPU resource requests.
expr: |-
- sum(namespace_cpu:kube_pod_container_resource_requests:sum{}) by (cluster) - (sum(kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"}) by (cluster) - max(kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"}) by (cluster)) > 0
- and
- (sum(kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"}) by (cluster) - max(kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"}) by (cluster)) > 0
+ # Non-HA clusters.
+ (
+ (
+ sum by (cluster) (namespace_cpu:kube_pod_container_resource_requests:sum{})
+ -
+ sum by (cluster) (kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"}) > 0
+ )
+ and
+ count by (cluster) (max by (cluster, node) (kube_node_role{job="kube-state-metrics", role="control-plane"})) < 3
+ )
+ or
+ # HA clusters.
+ (
+ sum by (cluster) (namespace_cpu:kube_pod_container_resource_requests:sum{})
+ -
+ (
+ # Skip clusters with only one allocatable node.
+ (
+ sum by (cluster) (kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"})
+ -
+ max by (cluster) (kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"})
+ ) > 0
+ ) > 0
+ )
for: 10m
labels:
severity: warning
- alert: KubeMemoryOvercommit
annotations:
description: Cluster {{ $labels.cluster }} has overcommitted memory resource
requests for Pods by {{ $value | humanize }} bytes and cannot tolerate node
failure.
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubememoryovercommit
summary: Cluster has overcommitted memory resource requests.
expr: |-
- sum(namespace_memory:kube_pod_container_resource_requests:sum{}) by (cluster) - (sum(kube_node_status_allocatable{resource="memory", job="kube-state-metrics"}) by (cluster) - max(kube_node_status_allocatable{resource="memory", job="kube-state-metrics"}) by (cluster)) > 0
- and
- (sum(kube_node_status_allocatable{resource="memory", job="kube-state-metrics"}) by (cluster) - max(kube_node_status_allocatable{resource="memory", job="kube-state-metrics"}) by (cluster)) > 0
+ # Non-HA clusters.
+ (
+ (
+ sum by (cluster) (namespace_memory:kube_pod_container_resource_requests:sum{})
+ -
+ sum by (cluster) (kube_node_status_allocatable{job="kube-state-metrics",resource="memory"}) > 0
+ )
+ and
+ count by (cluster) (max by (cluster, node) (kube_node_role{job="kube-state-metrics", role="control-plane"})) < 3
+ )
+ or
+ # HA clusters.
+ (
+ sum by (cluster) (namespace_memory:kube_pod_container_resource_requests:sum{})
+ -
+ (
+ # Skip clusters with only one allocatable node.
+ (
+ sum by (cluster) (kube_node_status_allocatable{job="kube-state-metrics",resource="memory"})
+ -
+ max by (cluster) (kube_node_status_allocatable{job="kube-state-metrics",resource="memory"})
+ ) > 0
+ ) > 0
+ )
for: 10m
labels:
severity: warning
- alert: KubeCPUQuotaOvercommit
annotations:
- description: Cluster {{ $labels.cluster }} has overcommitted CPU resource
+ description: Cluster {{ $labels.cluster }} has overcommitted CPU resource
requests for Namespaces.
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubecpuquotaovercommit
summary: Cluster has overcommitted CPU resource requests.
expr: |-
- sum(min without(resource) (kube_resourcequota{job="kube-state-metrics", type="hard", resource=~"(cpu|requests.cpu)"})) by (cluster)
- /
- sum(kube_node_status_allocatable{resource="cpu", job="kube-state-metrics"}) by (cluster)
- > 1.5
+ sum by (cluster) (
+ min without(resource) (kube_resourcequota{job="kube-state-metrics", type="hard", resource=~"(cpu|requests.cpu)"})
+ )
+ /
+ sum by (cluster) (
+ kube_node_status_allocatable{resource="cpu", job="kube-state-metrics"}
+ ) > 1.5
for: 5m
labels:
severity: warning
- alert: KubeMemoryQuotaOvercommit
annotations:
description: Cluster {{ $labels.cluster }} has overcommitted memory resource
requests for Namespaces.
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubememoryquotaovercommit
summary: Cluster has overcommitted memory resource requests.
expr: |-
- sum(min without(resource) (kube_resourcequota{job="kube-state-metrics", type="hard", resource=~"(memory|requests.memory)"})) by (cluster)
- /
- sum(kube_node_status_allocatable{resource="memory", job="kube-state-metrics"}) by (cluster)
- > 1.5
+ sum by (cluster) (
+ min without(resource) (kube_resourcequota{job="kube-state-metrics", type="hard", resource=~"(memory|requests.memory)"})
+ )
+ /
+ sum by (cluster) (
+ kube_node_status_allocatable{resource="memory", job="kube-state-metrics"}
+ ) > 1.5
for: 5m
labels:
severity: warning
- alert: KubeQuotaAlmostFull
annotations:
description: Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage
--- HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-kubernetes-system-kubelet
+++ HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-kubernetes-system-kubelet
@@ -123,15 +123,24 @@
- alert: KubeletPodStartUpLatencyHigh
annotations:
description: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds
on node {{ $labels.node }} on cluster {{ $labels.cluster }}.
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubeletpodstartuplatencyhigh
summary: Kubelet Pod startup latency is too high.
- expr: histogram_quantile(0.99, sum(rate(kubelet_pod_worker_duration_seconds_bucket{job="kubelet",
- metrics_path="/metrics"}[5m])) by (cluster, instance, le)) * on (cluster,
- instance) group_left(node) kubelet_node_name{job="kubelet", metrics_path="/metrics"}
+ expr: |-
+ histogram_quantile(0.99,
+ sum by (cluster, instance, le) (
+ topk by (cluster, instance, le, operation_type) (1,
+ rate(kubelet_pod_worker_duration_seconds_bucket{job="kubelet", metrics_path="/metrics"}[5m])
+ )
+ )
+ )
+ * on (cluster, instance) group_left(node)
+ topk by (cluster, instance, node) (1,
+ kubelet_node_name{job="kubelet", metrics_path="/metrics"}
+ )
> 60
for: 15m
labels:
severity: warning
- alert: KubeletClientCertificateExpiration
annotations:
--- HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-node-exporter
+++ HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-node-exporter
@@ -357,12 +357,13 @@
severity: warning
- alert: NodeBondingDegraded
annotations:
description: Bonding interface {{ $labels.master }} on {{ $labels.instance
}} is in degraded state due to one or more slave failures.
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodebondingdegraded
- summary: Bonding interface is degraded
- expr: (node_bonding_slaves - node_bonding_active) != 0
+ summary: Bonding interface is degraded.
+ expr: (node_bonding_slaves{job="node-exporter"} - node_bonding_active{job="node-exporter"})
+ != 0
for: 5m
labels:
severity: warning
--- HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-prometheus
+++ HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-prometheus
@@ -194,14 +194,14 @@
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/prometheus/prometheusremotewritebehind
summary: Prometheus remote write is behind.
expr: |-
# Without max_over_time, failed scrapes could create false negatives, see
# https://www.robustperception.io/alerting-on-gauges-in-prometheus-2-0 for details.
(
- max_over_time(prometheus_remote_storage_highest_timestamp_in_seconds{job="kube-prometheus-stack-prometheus",namespace="observability"}[5m])
- - ignoring(remote_name, url) group_right
+ max_over_time(prometheus_remote_storage_queue_highest_timestamp_seconds{job="kube-prometheus-stack-prometheus",namespace="observability"}[5m])
+ -
max_over_time(prometheus_remote_storage_queue_highest_sent_timestamp_seconds{job="kube-prometheus-stack-prometheus",namespace="observability"}[5m])
)
> 120
for: 15m
labels:
severity: critical
--- HelmRelease: observability/kube-prometheus-stack ValidatingWebhookConfiguration: observability/kube-prometheus-stack-admission
+++ HelmRelease: observability/kube-prometheus-stack ValidatingWebhookConfiguration: observability/kube-prometheus-stack-admission
@@ -10,13 +10,13 @@
app.kubernetes.io/part-of: kube-prometheus-stack
release: kube-prometheus-stack
heritage: Helm
app.kubernetes.io/name: kube-prometheus-stack-prometheus-operator
app.kubernetes.io/component: prometheus-operator-webhook
webhooks:
-- name: prometheusrulemutate.monitoring.coreos.com
+- name: prometheusrulevalidate.monitoring.coreos.com
failurePolicy: Ignore
rules:
- apiGroups:
- monitoring.coreos.com
apiVersions:
- '*'
@@ -32,7 +32,29 @@
path: /admission-prometheusrules/validate
timeoutSeconds: 10
admissionReviewVersions:
- v1
- v1beta1
sideEffects: None
+- name: alertmanagerconfigsvalidate.monitoring.coreos.com
+ failurePolicy: Ignore
+ rules:
+ - apiGroups:
+ - monitoring.coreos.com
+ apiVersions:
+ - v1alpha1
+ resources:
+ - alertmanagerconfigs
+ operations:
+ - CREATE
+ - UPDATE
+ clientConfig:
+ service:
+ namespace: observability
+ name: kube-prometheus-stack-operator
+ path: /admission-alertmanagerconfigs/validate
+ timeoutSeconds: 10
+ admissionReviewVersions:
+ - v1
+ - v1beta1
+ sideEffects: None
--- HelmRelease: observability/kube-prometheus-stack ConfigMap: observability/kube-prometheus-stack-crds-upgrade
+++ HelmRelease: observability/kube-prometheus-stack ConfigMap: observability/kube-prometheus-stack-crds-upgrade
@@ -15,8 +15,8 @@
release: kube-prometheus-stack
heritage: Helm
app: crds-operator
app.kubernetes.io/name: crds-prometheus-operator
app.kubernetes.io/component: crds-upgrade
binaryData:
[Diff truncated by flux-local]
--- HelmRelease: observability/kube-prometheus-stack Job: observability/kube-prometheus-stack-admission-create
+++ HelmRelease: observability/kube-prometheus-stack Job: observability/kube-prometheus-stack-admission-create
@@ -30,13 +30,13 @@
heritage: Helm
app.kubernetes.io/name: kube-prometheus-stack-prometheus-operator
app.kubernetes.io/component: prometheus-operator-webhook
spec:
containers:
- name: create
- image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.2
+ image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.4
imagePullPolicy: IfNotPresent
args:
- create
- --host=kube-prometheus-stack-operator,kube-prometheus-stack-operator.observability.svc
- --namespace=observability
- --secret-name=kube-prometheus-stack-admission
--- HelmRelease: observability/kube-prometheus-stack Job: observability/kube-prometheus-stack-admission-patch
+++ HelmRelease: observability/kube-prometheus-stack Job: observability/kube-prometheus-stack-admission-patch
@@ -30,13 +30,13 @@
heritage: Helm
app.kubernetes.io/name: kube-prometheus-stack-prometheus-operator
app.kubernetes.io/component: prometheus-operator-webhook
spec:
containers:
- name: patch
- image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.2
+ image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.4
imagePullPolicy: IfNotPresent
args:
- patch
- --webhook-name=kube-prometheus-stack-admission
- --namespace=observability
- --secret-name=kube-prometheus-stack-admission
--- HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-dockerhub-rules
+++ HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-dockerhub-rules
@@ -0,0 +1,25 @@
+---
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+ name: kube-prometheus-stack-dockerhub-rules
+ namespace: observability
+ labels:
+ app: kube-prometheus-stack
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/instance: kube-prometheus-stack
+ app.kubernetes.io/part-of: kube-prometheus-stack
+ release: kube-prometheus-stack
+ heritage: Helm
+spec:
+ groups:
+ - name: dockerhub
+ rules:
+ - alert: DockerhubRateLimitRisk
+ annotations:
+ summary: Kubernetes cluster Dockerhub rate limit risk
+ expr: count(time() - container_last_seen{image=~"(docker.io).*",container!=""}
+ < 30) > 100
+ labels:
+ severity: critical
+
--- HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-oom-rules
+++ HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-oom-rules
@@ -0,0 +1,27 @@
+---
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+ name: kube-prometheus-stack-oom-rules
+ namespace: observability
+ labels:
+ app: kube-prometheus-stack
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/instance: kube-prometheus-stack
+ app.kubernetes.io/part-of: kube-prometheus-stack
+ release: kube-prometheus-stack
+ heritage: Helm
+spec:
+ groups:
+ - name: oom
+ rules:
+ - alert: OomKilled
+ annotations:
+ summary: Container {{ $labels.container }} in pod {{ $labels.namespace }}/{{
+ $labels.pod }} has been OOMKilled {{ $value }} times in the last 10 minutes.
+ expr: (kube_pod_container_status_restarts_total - kube_pod_container_status_restarts_total
+ offset 10m >= 1) and ignoring (reason) min_over_time(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[10m])
+ == 1
+ labels:
+ severity: critical
+
--- HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-zfs-rules
+++ HelmRelease: observability/kube-prometheus-stack PrometheusRule: observability/kube-prometheus-stack-zfs-rules
@@ -0,0 +1,25 @@
+---
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+ name: kube-prometheus-stack-zfs-rules
+ namespace: observability
+ labels:
+ app: kube-prometheus-stack
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/instance: kube-prometheus-stack
+ app.kubernetes.io/part-of: kube-prometheus-stack
+ release: kube-prometheus-stack
+ heritage: Helm
+spec:
+ groups:
+ - name: zfs
+ rules:
+ - alert: ZfsUnexpectedPoolState
+ annotations:
+ summary: ZFS pool {{$labels.zpool}} on {{$labels.instance}} is in a unexpected
+ state {{$labels.state}}
+ expr: node_zfs_zpool_state{state!="online"} > 0
+ labels:
+ severity: critical
+ |
3269ea5 to
f193aaa
Compare
f193aaa to
120d194
Compare
120d194 to
4e89fb6
Compare
4e89fb6 to
510538b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains the following updates:
70.10.0->79.2.1Release Notes
prometheus-community/helm-charts (kube-prometheus-stack)
v79.2.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@kube-prometheus-stack-79.2.0...kube-prometheus-stack-79.2.1
v79.2.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-27.44.0...kube-prometheus-stack-79.2.0
v79.1.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
New Contributors
Full Changelog: prometheus-community/helm-charts@prometheus-27.42.2...kube-prometheus-stack-79.1.1
v79.1.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-pushgateway-3.4.2...kube-prometheus-stack-79.1.0
v79.0.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-nginx-exporter-1.15.1...kube-prometheus-stack-79.0.1
v79.0.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
New Contributors
Full Changelog: prometheus-community/helm-charts@prometheus-conntrack-stats-exporter-0.5.28...kube-prometheus-stack-79.0.0
v78.5.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-nginx-exporter-1.15.0...kube-prometheus-stack-78.5.0
v78.4.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-stackdriver-exporter-4.12.2...kube-prometheus-stack-78.4.0
v78.3.2Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@kube-prometheus-stack-78.3.1...kube-prometheus-stack-78.3.2
v78.3.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
New Contributors
Full Changelog: prometheus-community/helm-charts@prometheus-27.41.1...kube-prometheus-stack-78.3.1
v78.3.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-27.41.0...kube-prometheus-stack-78.3.0
v78.2.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
New Contributors
Full Changelog: prometheus-community/helm-charts@kube-prometheus-stack-78.2.0...kube-prometheus-stack-78.2.1
v78.2.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-druid-exporter-1.2.0...kube-prometheus-stack-78.2.0
v78.1.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-pgbouncer-exporter-0.9.0...kube-prometheus-stack-78.1.0
v78.0.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-27.40.0...kube-prometheus-stack-78.0.0
v77.14.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@alertmanager-1.27.0...kube-prometheus-stack-77.14.0
v77.13.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-smartctl-exporter-0.16.0...kube-prometheus-stack-77.13.0
v77.12.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
New Contributors
Full Changelog: prometheus-community/helm-charts@prometheus-postgres-exporter-7.3.0...kube-prometheus-stack-77.12.1
v77.12.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
d3f136aby @renovate[bot] in #6166Full Changelog: prometheus-community/helm-charts@kube-prometheus-stack-77.11.1...kube-prometheus-stack-77.12.0
v77.11.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-postgres-exporter-7.2.0...kube-prometheus-stack-77.11.1
v77.11.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-stackdriver-exporter-4.11.0...kube-prometheus-stack-77.11.0
v77.10.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-27.38.0...kube-prometheus-stack-77.10.0
v77.9.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
labelandlabelValuethrutplby @mahendrapaipuri in #6131New Contributors
Full Changelog: prometheus-community/helm-charts@kube-prometheus-stack-77.9.0...kube-prometheus-stack-77.9.1
v77.9.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@kube-prometheus-stack-77.8.0...kube-prometheus-stack-77.9.0
v77.8.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
ad9d690by @renovate[bot] in #6138Full Changelog: prometheus-community/helm-charts@kube-prometheus-stack-77.7.0...kube-prometheus-stack-77.8.0
v77.7.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-kafka-exporter-2.17.0...kube-prometheus-stack-77.7.0
v77.6.2Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
New Contributors
Full Changelog: prometheus-community/helm-charts@prom-label-proxy-0.15.1...kube-prometheus-stack-77.6.2
v77.6.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@kube-prometheus-stack-77.6.0...kube-prometheus-stack-77.6.1
v77.6.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-conntrack-stats-exporter-0.5.27...kube-prometheus-stack-77.6.0
v77.5.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
New Contributors
Full Changelog: prometheus-community/helm-charts@prometheus-27.34.0...kube-prometheus-stack-77.5.0
v77.4.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@kube-state-metrics-6.3.0...kube-prometheus-stack-77.4.0
v77.3.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
0de7907by @renovate[bot] in #6107Full Changelog: prometheus-community/helm-charts@kube-prometheus-stack-77.2.1...kube-prometheus-stack-77.3.0
v77.2.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
New Contributors
Full Changelog: prometheus-community/helm-charts@prometheus-27.33.0...kube-prometheus-stack-77.2.1
v77.2.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@kube-state-metrics-6.2.0...kube-prometheus-stack-77.2.0
v77.1.3Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@kube-prometheus-stack-77.1.2...kube-prometheus-stack-77.1.3
v77.1.2Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@kube-prometheus-stack-77.1.1...kube-prometheus-stack-77.1.2
v77.1.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@kube-prometheus-stack-77.1.0...kube-prometheus-stack-77.1.1
v77.1.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-mongodb-exporter-3.13.0...kube-prometheus-stack-77.1.0
v77.0.2Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@kube-state-metrics-6.1.5...kube-prometheus-stack-77.0.2
v77.0.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-blackbox-exporter-11.3.1...kube-prometheus-stack-77.0.1
v77.0.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-redis-exporter-6.16.0...kube-prometheus-stack-77.0.0
v76.5.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-ipmi-exporter-0.6.3...kube-prometheus-stack-76.5.1
v76.5.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
e5654afby @renovate[bot] in #6079Full Changelog: prometheus-community/helm-charts@prometheus-27.32.0...kube-prometheus-stack-76.5.0
v76.4.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@alertmanager-1.25.0...kube-prometheus-stack-76.4.1
v76.4.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@kube-prometheus-stack-76.3.1...kube-prometheus-stack-76.4.0
v76.3.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
New Contributors
Full Changelog: prometheus-community/helm-charts@alertmanager-snmp-notifier-2.1.0...kube-prometheus-stack-76.3.1
v76.3.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
75f1a1dby @renovate[bot] in #6047Full Changelog: prometheus-community/helm-charts@kube-prometheus-stack-76.2.2...kube-prometheus-stack-76.3.0
v76.2.2Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-adapter-5.1.0...kube-prometheus-stack-76.2.2
v76.2.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
New Contributors
Full Changelog: prometheus-community/helm-charts@prometheus-27.30.0...kube-prometheus-stack-76.2.1
v76.2.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
3f733e2by @renovate[bot] in #6034Full Changelog: prometheus-community/helm-charts@prometheus-adapter-5.0.0...kube-prometheus-stack-76.2.0
v76.1.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-operator-admission-webhook-0.29.3...kube-prometheus-stack-76.1.0
v76.0.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@kube-state-metrics-6.1.4...kube-prometheus-stack-76.0.0
v75.18.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
New Contributors
Full Changelog: prometheus-community/helm-charts@kube-prometheus-stack-75.18.0...kube-prometheus-stack-75.18.1
v75.18.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@kube-state-metrics-6.1.2...kube-prometheus-stack-75.18.0
v75.17.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
Full Changelog: prometheus-community/helm-charts@prometheus-kafka-exporter-2.16.0...kube-prometheus-stack-75.17.1
v75.17.0Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
New Contributors
Full Changelog: prometheus-community/helm-charts@prometheus-27.29.1...kube-prometheus-stack-75.17.0
v75.16.1Compare Source
kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
What's Changed
New Contributors
Full Changelog: prometheus-community/helm-charts@kube-prometheus-stack-75.16.0...kube-prometheus-stack-75.16.1
[
v75.16.0](https://redirect.github.com/prometheus-community/helm-chConfiguration
📅 Schedule: Branch creation - "every weekend" (UTC), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR has been generated by Renovate Bot.