fix(helm): Avoid scraping metrics from gateway pod #13350

Skaronator · 2024-06-28T09:39:07Z

What this PR does / why we need it:

Rename the nginx port name to avoid beeing scraped by the ServiceMonitor:

loki/production/helm/loki/templates/monitoring/servicemonitor.yaml

Lines 32 to 33 in 91a3486

    
           - port: http-metrics 
        
             path: /metrics

Which issue(s) this PR fixes:
Fixes #13201

Special notes for your reviewer:

Checklist

Reviewed the CONTRIBUTING.md guide (required)
Documentation added
Tests updated
Title matches the required conventional commits format, see here
- Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

CLAassistant · 2024-06-28T09:39:12Z

All committers have signed the CLA.

Skaronator · 2024-06-28T09:46:40Z

I am not sure if we need to do a major version bump. Technically, it's a breaking change, but I don't think it will affect anyone.

adinhodovic · 2024-07-02T18:24:53Z

Ah opened a similar PR a while back #12746, was alerting. Seems to resolve the same thing.

…crape

Skaronator · 2024-07-05T06:14:01Z

I've rebased and fixed the merge conflicts that have come up since creating this PR.

production/helm/loki/CHANGELOG.md

Co-authored-by: Hervé Nicol <[email protected]>

…crape

Skaronator · 2024-07-22T07:22:31Z

I've rebased this PR again and fixed the merge conflicts that have come up.

… into fix-nginx-metrics-scrape

lblazewski · 2024-08-01T09:24:36Z

Looking forward for this PR!

…crape

Skaronator · 2024-08-12T13:32:37Z

Resolved merge conflicts again. Hoping for a timely review.

Skaronator · 2024-08-12T13:33:43Z

cc @vlad-diachenko

vlad-diachenko · 2024-08-13T11:06:27Z

production/helm/loki/templates/gateway/deployment-gateway-enterprise.yaml

@@ -101,7 +101,7 @@ spec:
            {{ toYaml .Values.enterpriseGateway.extraVolumeMounts | nindent 12 }}
            {{- end }}
          ports:
-            - name: http-metrics
+            - name: http


when the loki/gel image is run in mode gateway(-target=gateway), it exposes the metrics under /metrics url ....

vlad-diachenko · 2024-08-13T11:14:22Z

cc @vlad-diachenko

I believe that we have the issue only when nginx is deployed as a gateway.... so, I would configure nginx to expose the metrics under /metrics path. wdyt?

francesco-kekko-damico · 2024-10-30T08:39:19Z

Hi, is there an update on this? We need this fix as we have the service monitor target for nginx pod down for quite long time now

nlamirault · 2024-12-26T15:07:53Z

any news on this fix ?

Starefossen · 2025-01-27T09:56:32Z

@vlad-diachenko @hervenicol anything I can do to help unblock a fix for this issue?

vlad-diachenko · 2025-01-31T12:44:40Z

@vlad-diachenko @hervenicol anything I can do to help unblock a fix for this issue?

Hey @Starefossen .
We can not merge it because of the thing I mentioned in the comment .
This port should be scraped if GEL image is used for the gateway.
So, if the issue happens only when NGING is used as a gateway, I would add label to the gateway pod prometheus.io/scrape: "false" , so it won't be scraped. (it needs to be tested because I googled this solution)
wdyt?

Starefossen · 2025-02-03T08:20:18Z

I can confirm that adding the prometheus.io/service-monitor: "false" label to the gateway pods disables the erroneous scraping:

  gateway:
    service:
      labels:
        prometheus.io/service-monitor: "false"

vlad-diachenko · 2025-02-03T10:04:24Z

I can confirm that adding the prometheus.io/service-monitor: "false" label to the gateway pods disables the erroneous scraping:
  gateway:
    service:
      labels:
        prometheus.io/service-monitor: "false"

I believe that it's the way we need to fix the issue.
Huge thanks for the confirmation

adinhodovic · 2025-02-03T10:07:34Z

Can we merge this? #12746 - opened this fix a while back and works locally, just want it upstream.

vlad-diachenko · 2025-02-05T12:17:21Z

this PR can be closed because the issue is fixed in #12746

Avoid scraping metrics from gateway pod

80c75a9

Skaronator requested a review from a team as a code owner June 28, 2024 09:39

pull-request-size bot added the size/S label Jun 28, 2024

github-actions bot added the area/helm label Jun 28, 2024

Bump Version and add changelog

dc02e25

Skaronator changed the title ~~Avoid scraping metrics from gateway pod~~ fix: Avoid scraping metrics from gateway pod Jun 28, 2024

Skaronator changed the title ~~fix: Avoid scraping metrics from gateway pod~~ fix(helm): Avoid scraping metrics from gateway pod Jun 28, 2024

Skaronator force-pushed the fix-nginx-metrics-scrape branch from 2fed5f0 to dc02e25 Compare July 5, 2024 06:09

Skaronator added 2 commits July 5, 2024 08:11

Merge remote-tracking branch 'upstream/main' into fix-nginx-metrics-s…

7465a61

…crape

Adjust changelog

61e8503

hervenicol reviewed Jul 14, 2024

View reviewed changes

production/helm/loki/CHANGELOG.md Outdated Show resolved Hide resolved

Skaronator and others added 2 commits July 22, 2024 09:19

Update production/helm/loki/CHANGELOG.md

6e8c709

Co-authored-by: Hervé Nicol <[email protected]>

Merge remote-tracking branch 'upstream/main' into fix-nginx-metrics-s…

86c547e

…crape

Skaronator added 3 commits July 22, 2024 09:23

Fix changelog

3def812

Merge branch 'main' into fix-nginx-metrics-scrape

6793811

Merge branch 'fix-nginx-metrics-scrape' of github.com:Skaronator/loki…

9a0de3a

… into fix-nginx-metrics-scrape

Merge remote-tracking branch 'upstream/main' into fix-nginx-metrics-s…

36a6570

…crape

vlad-diachenko reviewed Aug 13, 2024

View reviewed changes

vlad-diachenko self-assigned this Aug 13, 2024

sathieu mentioned this pull request Jan 8, 2025

2025 Issues kubitus-project/upstream-issues#1

Open

vlad-diachenko closed this Feb 5, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(helm): Avoid scraping metrics from gateway pod #13350

fix(helm): Avoid scraping metrics from gateway pod #13350

Skaronator commented Jun 28, 2024 •

edited

Loading

CLAassistant commented Jun 28, 2024 •

edited

Loading

Skaronator commented Jun 28, 2024

adinhodovic commented Jul 2, 2024

Skaronator commented Jul 5, 2024

Skaronator commented Jul 22, 2024

lblazewski commented Aug 1, 2024

Skaronator commented Aug 12, 2024

Skaronator commented Aug 12, 2024

vlad-diachenko Aug 13, 2024

vlad-diachenko commented Aug 13, 2024

francesco-kekko-damico commented Oct 30, 2024

nlamirault commented Dec 26, 2024

Starefossen commented Jan 27, 2025

vlad-diachenko commented Jan 31, 2025

Starefossen commented Feb 3, 2025

vlad-diachenko commented Feb 3, 2025

adinhodovic commented Feb 3, 2025

vlad-diachenko commented Feb 5, 2025

fix(helm): Avoid scraping metrics from gateway pod #13350

fix(helm): Avoid scraping metrics from gateway pod #13350

Conversation

Skaronator commented Jun 28, 2024 • edited Loading

CLAassistant commented Jun 28, 2024 • edited Loading

Skaronator commented Jun 28, 2024

adinhodovic commented Jul 2, 2024

Skaronator commented Jul 5, 2024

Skaronator commented Jul 22, 2024

lblazewski commented Aug 1, 2024

Skaronator commented Aug 12, 2024

Skaronator commented Aug 12, 2024

vlad-diachenko Aug 13, 2024

Choose a reason for hiding this comment

vlad-diachenko commented Aug 13, 2024

francesco-kekko-damico commented Oct 30, 2024

nlamirault commented Dec 26, 2024

Starefossen commented Jan 27, 2025

vlad-diachenko commented Jan 31, 2025

Starefossen commented Feb 3, 2025

vlad-diachenko commented Feb 3, 2025

adinhodovic commented Feb 3, 2025

vlad-diachenko commented Feb 5, 2025

Skaronator commented Jun 28, 2024 •

edited

Loading

CLAassistant commented Jun 28, 2024 •

edited

Loading