feature: `victoria-metrics-k8s-stack` allow multiple vmsingle replicas (servers)

[YuryHrytsuk]

Chart name and version, where you feel a lack of requested feature
chart: victoria-metrics-k8s-stack
version: v0.62.1

Is your feature request related to a problem? Please describe.
I want to deploy highly available victoria-metrics cluster for Kubernetes (including pre-configured scrapers). I don't want to use vm-distributed chart as it looks an overkill for my setup (I want 1 vm-agent, 1 vm-auth and 2+ VM Single)

Describe the solution you'd like
Setting replicaCount: 2 on vmsingle object would be ideal. Configuring 2 vmsingle (since vmsingle CR does not support >1 replicas) is also fine

Describe alternatives you've considered

1. I tried to configure all services on my own without operator CRs (using charts directly) but then I am missing pre-configured scrapers that I want
2. victoria-metrics-distributed is an overkill

Additional context
I am looking for an easy to configure simple highly available victoria metrics stack for Kubernetes. I don't want to configure scraper on my own (e.g. for kube-state-metrics, kubelet, ...). Moreover, documentation states that people should rather re-use k8s-stack or operator for scraper configuration

[Haleygo]

Hi!
The HA you want is the reason victoria-metrics-distributed exists. Both victoria-metrics-k8s-stack and victoria-metrics-distributed charts provide HA ability, but in different ways:

1. victoria-metrics-k8s-stack achieves it by setting -replicationFactor > 1.
2. victoria-metrics-distributed achieves it by creating multiple independent vmsingle/vmcluster.

So if you want a single vmcluster, use victoria-metrics-k8s-stack; if you want multiple vmsingle/vmcluster, use victoria-metrics-distributed.

And in different charts, the configs are different.
For example, with multiple independent vmsingle/vmcluster instances, there must be a vmagent to write data to all instances, a read proxy(like vmauth) to query data from all instances, and clients like vmalert or grafana must use the read proxy as their datasource.
So for victoria-metrics-k8s-stack, it's not only about adding a new vmsingle, but also requires changing the configurations mentioned above, and all those things are already handled in victoria-metrics-distributed.

I want 1 vm-agent, 1 vm-auth and 2+ VM Single

You can have them as a minimum set up in victoria-metrics-distributed, extra vmauth like vmauth-read-proxy are optional and can be disabled specifically. But in most cases, it's recommended to keep them, as their proxy policies enhance availability during downtime, and they don’t consume many resources as proxies when running.

[YuryHrytsuk]

Hi! The HA you want is the reason victoria-metrics-distributed exists. Both victoria-metrics-k8s-stack and victoria-metrics-distributed charts provide HA ability, but in different ways:

1. `victoria-metrics-k8s-stack` achieves it by setting `-replicationFactor > 1`.

2. `victoria-metrics-distributed` achieves it by creating multiple independent vmsingle/vmcluster.

So if you want a single vmcluster, use victoria-metrics-k8s-stack; if you want multiple vmsingle/vmcluster, use victoria-metrics-distributed.

And in different charts, the configs are different. For example, with multiple independent vmsingle/vmcluster instances, there must be a vmagent to write data to all instances, a read proxy(like vmauth) to query data from all instances, and clients like vmalert or grafana must use the read proxy as their datasource. So for victoria-metrics-k8s-stack, it's not only about adding a new vmsingle, but also requires changing the configurations mentioned above, and all those things are already handled in victoria-metrics-distributed.

I want 1 vm-agent, 1 vm-auth and 2+ VM Single

You can have them as a minimum set up in victoria-metrics-distributed, extra vmauth like vmauth-read-proxy are optional and can be disabled specifically. But in most cases, it's recommended to keep them, as their proxy policies enhance availability during downtime, and they don’t consume many resources as proxies when running.

Thanks @Haleygo,

One thing I don't understand with vm-distributed is why would I need 3 vm-agents (and 3 vm-auth respectively)? I can have 1 vm-agent writing to all vm-single servers (and 1 vm-auth) that is not bound to any zone (so that if 1 zone fails, it goes to a healthy zone). Why 3? What's the matter?

[YuryHrytsuk]

In fact it is even more complicated, instead of having just one vm-agent there are 4 components: extra-vm-agent, vm-auth-global-write (??), vm-agent-zone1 and vm-agent-zone2 🤔

[AndrewChubatiuk]

For you case you can disable:

• global write balancer: .Values.write.global.vmauth.enabled: false
• per az vmagent: .Values.zoneTpl.vmagent.enabled: false
• per az vmauth: .Values.zoneTpl.read.vmauth.enabled: false
• disable vmcluster: .Values.zoneTpl.vmcluster.enabled: false
• enable vmsingle: .Values.zoneTpl.vmsingle.enabled: true

With these properties set you'll have a single global read vmauth in front of multiple vmsingle instances
this pr besides mixed vmauth for read and write also allows to point extra vmagent to vmsingle/vminsert directly, but with distributed vmalert should still be manually configured.

To have everything configured automatically is only achievable by merging distributed and k8s-stack charts, but in minimal HA setup it still leads to additional VMAuth for VMSingle/VMCluster (besides one, which is managed by vmauth.enabled) and multiple breaking changes to existing k8s-stack chart to support multiple VMSingle/VMCluster configuration options

[YuryHrytsuk]

1. victoria-metrics-k8s-stack achieves it by setting -replicationFactor > 1.

Here you actually imply using vmcluster. But using vmcluster is not recommended in majority of cases. So it does not seem to be a 100% proper / clean solution in case of victoria-metrics-k8s-stack chart

PS reference for others https://docs.victoriametrics.com/victoriametrics/faq/#which-victoriametrics-type-is-recommended-for-use-in-production---single-node-or-cluster

[Haleygo]

One thing I don't understand with vm-distributed is why would I need 3 vm-agents (and 3 vm-auth respectively)? I can have 1 vm-agent writing to all vm-single servers (and 1 vm-auth) that is not bound to any zone (so that if 1 zone fails, it goes to a healthy zone). Why 3? What's the matter?

See the chart details for the purposes of each component:

For write:

extra-vmagent(optional): scrapes external targets and all the components installed by this chart, sends data to global write entrypoint.
vmauth-global-write: global write entrypoint, proxies requests to one of the zone vmagent with least_loaded policy.
vmagent(per-zone): remote writes data to availability zones that enabled .Values.availabilityZones[*].write.allow, and buffer data on disk when zone is unavailable to ingest.
vmauth-write-balancer(per-zone): proxies requests to vminsert instances inside it’s zone with least_loaded policy.
vmcluster(per-zone): processes write requests and stores data.
For read:

vmcluster(per-zone): processes query requests and returns results.
vmauth-read-balancer(per-zone): proxies requests to vmselect instances inside it’s zone with least_loaded policy.
vmauth-read-proxy(per-zone): uses all the vmauth-read-balancer as servers if zone has .Values.availabilityZones[*].read.allow enabled, always prefer “local” vmauth-read-balancer to reduce cross-zone traffic with first_available policy.
vmauth-global-read: global query entrypoint, proxies requests to one of the zone vmauth-read-proxy with first_available policy.
grafana(optional): uses vmauth-global-read as default datasource.

For your question, extra-vmagent is optional and stands for clients that send data to VM stack, it can be the extra-vmagent, any exsiting vmagent users already have, or any other agents like otel-collector. The main responsibility of vmagent(per-zone) is to buffer data on disk when zone is unavailable to ingest.
And the full topology of victoria-metrics-distributed is designed to tolerate zone outages by deploying components in every availability zone and enabling the minimum-downtime switch during outages.

You can have a minimum set up by disabling componets you don't need, as @AndrewChubatiuk mentioned here.

Here you actually imply using vmcluster. But using vmcluster is not recommended in majority of cases. So it does not seem to be a 100% proper / clean solution in case of victoria-metrics-k8s-stack chart

No, I'm not. If vmsingle is enough for your usage, you should use it.
What I mean is, if you want HA with vmsingle(which requires multiple deployments), you should go with victoria-metrics-distributed instead of victoria-metrics-k8s-stack, because of the configuration changes I mentioned here.