Prometheus reloading

[hholst80]

I have deployed VictoraMetrics (promethus compatible) as a Deployment with replicas=1 and I would like to reload Prometheus using Stakater/Reloader.
The problem I am running into is that VictoraMetrics will not allow the database to be re-allocated to a new pod while the old pod is still running. I cannot see a way to redeploy VM using this process, but rather I would in this case employ some kind of process signaling to reload the configuration. Is that something that Reloader can support directly or indirectly?

...
prometheus-5cf78d6949-qnpt5          1/1     Running            0          21h
prometheus-697ddcbfd9-lrll2          0/1     CrashLoopBackOff   3          84s

https://www.robustperception.io/reloading-prometheus-configuration

[faizanahmad055]

Currently, the reloader does not have such functionality to signal the change towards the container directly. Reloader interacts with the parent resource of the pod (Deployment, Statefulset, etc.) to reload the pod. But we welcome PRs for all the enhancements :)

[hholst80]

For anyone facing the same issue this looks promising:
https://www.weave.works/blog/prometheus-configmaps-continuous-deployment/

[Feder1co5oave]

Prometheus has embedded config reload. I use a sidecar container with https://hub.docker.com/r/jimmidyson/configmap-reload
that calls /-/reload on prometheus whenever the configmap gets updated

      - name: config-autoreload
        image: docker.io/jimmidyson/configmap-reload
        args:
        - --volume-dir=/etc/prometheus/config
        - --webhook-url=http://localhost:9090/-/reload
        volumeMounts:
        - name: conf
          mountPath: /etc/prometheus/config
          readOnly: true

(you must add --web.enable-lifecycle to prometheus startup args).

I run Victoria with no configuration file, the scraping is done by a separate vmagent deployment, that automatically reloads the prometheus scrape configuration if -promscrape.configCheckInterval=5m is enabled.