The problem of reconciliation in case of incorrect config

[tropnikovvl]

Hello!

If you make an error in the configuration of the Alertmanager or, for example, use the wrong Slack token, then the operator does not synchronize the Prometheus service monitor objects to objects for Victoria Metrics with this parameters

  operator:
    disable_prometheus_converter: false
    enable_converter_ownership: true

[f41gh7]

Hello, could you please provide any erorr examples and operator version?

There was a similar issue 3cf837f that was fixed at v0.58.1 version

[tropnikovvl]

@f41gh7 ,
Unfortunately I don't have the error at hand, but it was related to it not being able to apply the configuration to the Alertmanager.

I used version v0.60.2

[f41gh7]

I'd suggest to check status of VMAlertmanager, it should have a reason of fail, if operator cannot apply configuration changes.

[tropnikovvl]

Hi @f41gh7 ,

All possible logs that can be potentially interesting

{"level":"info","ts":"2025-07-24T11:17:25Z","logger":"controller.VMAlertmanager","msg":"selected VMAlertmanagerConfigs count=1, invalid rules count=1, namespaced names vmetrics/vmetrics-victoria-metrics-k8s-stack","vmalertmanager":"vmetrics-victoria-metrics-k8s-stack","namespace":"vmetrics"}
{"level":"error","ts":"2025-07-24T11:17:25Z","logger":"controller.VMAlertmanager","msg":"skip config generation for resources: parent=vmetrics-victoria-metrics-k8s-stack.vmetrics.vmalertmanager config=namespace/name=vmetrics/vmetrics-victoria-metrics-k8s-stack error text: invalid URL 123 in key SLACK_TOKEN from secret alertmanager-slack-webhook-url: unsupported scheme "" for URL","vmalertmanager":"vmetrics-victoria-metrics-k8s-stack","namespace":"vmetrics","error":"vmetrics-victoria-metrics-k8s-stack.vmetrics.vmalertmanager have errors"}

{"level":"info","ts":"2025-07-24T11:16:06Z","logger":"controller.VMAgent","msg":"selected VMServiceScrape count=10, invalid rules count=0,...

kubectl get vmservicescrapes -A
NAMESPACE   NAME                                                           AGE     STATUS        SYNC ERROR
vlogs       vlogs-victoria-logs-single-server                              10m     operational
vmetrics    vmagent-vmetrics-victoria-metrics-k8s-stack                    7m44s   operational
vmetrics    vmalert-vmetrics-victoria-metrics-k8s-stack                    9m16s   operational
vmetrics    vmalertmanager-vmetrics-victoria-metrics-k8s-stack             9m34s   operational
vmetrics    vmetrics-victoria-metrics-k8s-stack-core-dns                   10m     operational
vmetrics    vmetrics-victoria-metrics-k8s-stack-kube-api-server            10m     operational
vmetrics    vmetrics-victoria-metrics-k8s-stack-kube-state-metrics         10m     operational
vmetrics    vmetrics-victoria-metrics-k8s-stack-prometheus-node-exporter   10m     operational
vmetrics    vmetrics-victoria-metrics-operator                             10m     operational
vmetrics    vmsingle-vmetrics-victoria-metrics-k8s-stack                   8m42s   operational

kubectl get servicemonitors -A
NAMESPACE      NAME                               AGE
external-dns   external-dns                       13m
grafana        grafana                            10m

[tropnikovvl]

Hi @f41gh7 ,
Could you please take a look?

[f41gh7]

Hi @f41gh7 , Could you please take a look?

I tried to reproduce that and for me it works as expected.

Could you please check the following metrics:

operator_prometheus_converter_active_watchers
operator_prometheus_converter_watch_events_total

[makasim]

@tropnikovvl, just a kind reminder, could you please share the metrics requested above when you get a chance?

[makasim]

Closing as the problem could not be reproduced, and no more info was provided. Please reopen if more details become available.

[tropnikovvl]

@f41gh7, @makasim

To understand what I expect, I'm assuming that despite the problems with the alertmanager configuration, the operator will continue to reconcile the remaining instances, in this case, servicemonitors.
Which doesn't happen.

version: 0.63.0

operator_prometheus_converter_active_watchers{namespace="ALL_NAMESPACES"} 6
operator_prometheus_converter_watch_events_total{event_type="ADDED",namespace="ALL_NAMESPACES",object_type_name="pod_monitors"} 1
operator_prometheus_converter_watch_events_total{event_type="ADDED",namespace="ALL_NAMESPACES",object_type_name="service_monitors"} 21

kubectl get vmservicescrapes -A
NAMESPACE           NAME                                                           AGE     STATUS        SYNC ERROR
blackbox-exporter   blackbox-exporter-prometheus-blackbox-exporter                 26m     operational
blackbox-exporter   blackbox-exporter-prometheus-blackbox-exporter-1         26m     operational
blackbox-exporter   blackbox-exporter-prometheus-blackbox-exporter-2   26m     operational
blackbox-exporter   blackbox-exporter-prometheus-blackbox-exporter-3     26m     operational
blackbox-exporter   blackbox-exporter-prometheus-blackbox-exporter-4   26m     operational
blackbox-exporter   blackbox-exporter-prometheus-blackbox-exporter-5           8m41s   operational
blackbox-exporter   blackbox-exporter-prometheus-blackbox-exporter-6        8m41s   operational
vlogs               vlogs-victoria-logs-single-server                              30m     operational
vmetrics            vmagent-vmetrics-victoria-metrics-k8s-stack                    27m     operational
vmetrics            vmalert-vmetrics-victoria-metrics-k8s-stack                    28m     operational
vmetrics            vmalertmanager-vmetrics-victoria-metrics-k8s-stack             28m     operational
vmetrics            vmetrics-victoria-metrics-k8s-stack-core-dns                   29m     operational
vmetrics            vmetrics-victoria-metrics-k8s-stack-kube-api-server            29m     operational
vmetrics            vmetrics-victoria-metrics-k8s-stack-kube-state-metrics         29m     operational
vmetrics            vmetrics-victoria-metrics-k8s-stack-prometheus-node-exporter   29m     operational
vmetrics            vmetrics-victoria-metrics-operator                             29m     operational
vmetrics            vmsingle-vmetrics-victoria-metrics-k8s-stack                   28m     operational

kubectl get servicemonitors -A
NAMESPACE           NAME                                                           AGE
blackbox-exporter   blackbox-exporter-prometheus-blackbox-exporter                 26m
blackbox-exporter   blackbox-exporter-prometheus-blackbox-exporter-1         26m
blackbox-exporter   blackbox-exporter-prometheus-blackbox-exporter-2   26m
blackbox-exporter   blackbox-exporter-prometheus-blackbox-exporter-3     26m
blackbox-exporter   blackbox-exporter-prometheus-blackbox-exporter-4   26m
blackbox-exporter   blackbox-exporter-prometheus-blackbox-exporter-5           8m47s
blackbox-exporter   blackbox-exporter-prometheus-blackbox-exporter-6        8m47s
external-dns        external-dns                                                   43m
grafana             grafana                                                        29m
kube-system         alb-aws-load-balancer-controller                               44m
test                 test-app-1                                        41m
test                 test-app-2                                 41m
test                 test-app-3                                               41m
test                 test-app-4                                                       41m
test                 test-app-5                                                   41m
test                 test-app-6                                               41m
test                 test-app-7                                      41m
test                 test-app-8                                  41m
test                 test-app-9                                                     41m
test                 test-app-10                                              41m
test                 test-app-11                                                         43m

selected VMServiceScrape count=31, invalid rules count=0 ...

selected VMAlertmanagerConfigs count=1, invalid rules count=1, namespaced names vmetrics/vmetrics-victoria-metrics-k8s-stack

skip config generation for resources: parent=vmetrics-victoria-metrics-k8s-stack.vmetrics.vmalertmanager config=namespace/name=vmetrics/vmetrics-victoria-metrics-k8s-stack error text: invalid URL 123 in key SLACK_TOKEN from secret alertmanager-slack-webhook-url: unsupported scheme "" for URL