diff --git a/charts/victoria-metrics-distributed/CHANGELOG.md b/charts/victoria-metrics-distributed/CHANGELOG.md index 38bc33930..2668afa8b 100644 --- a/charts/victoria-metrics-distributed/CHANGELOG.md +++ b/charts/victoria-metrics-distributed/CHANGELOG.md @@ -1,5 +1,7 @@ ## Next release +**Update note**: This release contains breaking changes. Please follow [upgrade guide](https://docs.victoriametrics.com/helm/victoria-metrics-distributed/#upgrade-to-0240) + - enable ingest only mode for VMAgents. See [#1594](https://github.com/VictoriaMetrics/operator/issues/1594). ## 0.23.0 diff --git a/charts/victoria-metrics-distributed/_index.md.gotmpl b/charts/victoria-metrics-distributed/_index.md.gotmpl index fff8db389..afa6565ba 100644 --- a/charts/victoria-metrics-distributed/_index.md.gotmpl +++ b/charts/victoria-metrics-distributed/_index.md.gotmpl @@ -33,26 +33,27 @@ For write: 1. extra-vmagent(optional): scrapes external targets and all the components installed by this chart, sends data to global write entrypoint. 2. vmauth-global-write: global write entrypoint, proxies requests to one of the zone `vmagent` with `least_loaded` policy. 3. vmagent(per-zone): remote writes data to availability zones that enabled `.Values.availabilityZones[*].write.allow`, and [buffer data on disk](https://docs.victoriametrics.com/victoriametrics/vmagent/#calculating-disk-space-for-persistence-queue) when zone is unavailable to ingest. -4. vmauth-write-balancer(per-zone): proxies requests to vminsert instances inside it's zone with `least_loaded` policy. -5. vmcluster(per-zone): processes write requests and stores data. +4. vmcluster(per-zone): processes write requests and stores data. For read: 1. vmcluster(per-zone): processes query requests and returns results. -2. vmauth-read-balancer(per-zone): proxies requests to vmselect instances inside it's zone with `least_loaded` policy. -3. vmauth-read-proxy(per-zone): uses all the `vmauth-read-balancer` as servers if zone has `.Values.availabilityZones[*].read.allow` enabled, always prefer "local" `vmauth-read-balancer` to reduce cross-zone traffic with `first_available` policy. -4. vmauth-global-read: global query entrypoint, proxies requests to one of the zone `vmauth-read-proxy` with `first_available` policy. -5. grafana(optional): uses `vmauth-global-read` as default datasource. +2. vmauth-read-proxy(per-zone): proxies query requests to zones with `.Values.availabilityZones[*].read.allow` enabled, preferring the "local" zone to reduce cross-zone traffic using the `first_available` policy. +3. vmauth-global-read: global query entrypoint, proxies requests to one of the zone `vmauth-read-proxy` with `first_available` policy. +4. grafana(optional): uses `vmauth-global-read` as default datasource. >Note: As the topology shown above, this chart doesn't include components like vmalert, alertmanager, etc by default. You can install them using dependency [victoria-metrics-k8s-stack](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-k8s-stack) or having separate release. +>Note: +The default topology tolerates zone outages by deploying components in every availability zone and enabling minimum-downtime during outages. If not required, some components(including vmauth-global-write, vmagent(per-zone), vmauth-read-proxy(per-zone)) are optional and can be disabled based on your use case, please refer to [Parameters](#parameters) section for details. + ### Why use `victoria-metrics-distributed` chart? One of the best practice of running production kubernetes cluster is running with [multiple availability zones](https://kubernetes.io/docs/setup/best-practices/multiple-zones/). And apart from kubernetes control plane components, we also want to spread our application pods on multiple zones, to continue serving even if zone outage happens. VictoriaMetrics supports [data replication](https://docs.victoriametrics.com/victoriametrics/cluster-victoriametrics/#replication-and-data-safety) natively which can guarantees data availability when part of the vmstorage instances failed. But it doesn't works well if vmstorage instances are spread on multiple availability zones, since data replication could be stored on single availability zone, which will be lost when zone outage happens. -To avoid this, vmcluster must be installed on multiple availability zones, each containing a 100% copy of data. As long as one zone is available, both global write and read entrypoints should work without interruption. +To avoid this, database(such as vmcluster or vmsingle) must be deployed across multiple zones, with each zone containing a full copy of the data. As long as one zone remains available, both global write and read entrypoints should operate without interruption. ### How to write data? @@ -71,7 +72,7 @@ You can also pick other proxies like kubernetes service which supports [Topology If availability zone `zone-eu-1` is experiencing an outage, `vmauth-global-write` and `vmauth-global-read` will work without interruption: 1. `vmauth-global-write` stops proxying write requests to `zone-eu-1` automatically; 2. `vmauth-global-read` and `vmauth-read-proxy` stops proxying read requests to `zone-eu-1` automatically; -3. `vmagent` on `zone-us-1` fails to send data to `zone-eu-1.vmauth-write-balancer`, starts to buffer data on disk(unless `-remoteWrite.disableOnDiskQueue` is specified, which is not recommended for this topology); +3. `vmagent` on `zone-us-1` fails to send data to `zone-eu-1`, starts to buffer data on disk(unless `-remoteWrite.disableOnDiskQueue` is specified, which is not recommended for this topology); To keep data completeness for all the availability zones, make sure you have enough disk space on vmagent for buffer, see [this doc](https://docs.victoriametrics.com/victoriametrics/vmagent/#calculating-disk-space-for-persistence-queue) for size recommendation. And to avoid getting incomplete responses from `zone-eu-1` which gets recovered from outage, check vmagent on `zone-us-1` to see if persistent queue has been drained. If not, remove `zone-eu-1` from serving query by setting `.Values.availabilityZones.{zone-eu-1}.read.allow=false` and change it back after confirm all data are restored. @@ -132,6 +133,64 @@ First, performing update on availability zone `zone-eu-1`: Then, perform update on availability zone `zone-us-1` with the same steps1~4. +### Upgrade to 0.24.0 + +Starting this release first item of `*.vmauth.spec.unauthorizedUserAccessSpec.url_map` is no longer merged with default backend configuration. +Custom settings for default backend should now be defined using `defaultUrlMapItem..`: + +For global read VMAuth: + +``` +read: + global: + vmauth: + spec: + unauthorizedUserAccessSpec: + url_map: + - load_balancing_policy: first_available + retry_status_codes: [500, 502, 503] +``` +is now +``` +defaultUrlMapItem: + read: + vmsingle: # if you're using VMSingle + load_balancing_policy: first_available + retry_status_codes: [500, 502, 503] + vmcluster: # if you're using VMCluster + load_balancing_policy: first_available + retry_status_codes: [500, 502, 503] +``` + +For global write VMAuth: + +``` +write: + global: + vmauth: + spec: + unauthorizedUserAccessSpec: + url_map: + - load_balancing_policy: first_available + retry_status_codes: [500, 502, 503] +``` +is now +``` +defaultUrlMapItem: + write: + vmagent: # if you have VMAgents queue enabled + load_balancing_policy: first_available + retry_status_codes: [500, 502, 503] + vmsingle: # if you're using VMSingle without VMAgents enabled + load_balancing_policy: first_available + retry_status_codes: [500, 502, 503] + vmcluster: # if you're using VMCluster without VMAgents enabled + load_balancing_policy: first_available + retry_status_codes: [500, 502, 503] +``` + +Same is for per zone VMAuth proxy `zoneTpl.read.vmauth`. + ### Upgrade to 0.13.0 Introduction of VMCluster's [`requestsLoadBalancer`](https://docs.victoriametrics.com/operator/resources/vmcluster/#requests-load-balancing) allowed to simplify distributed chart setup by removing VMAuth CRs for read and write load balancing. Some parameters are not needed anymore: diff --git a/charts/victoria-metrics-distributed/img/victoriametrics-distributed-topology.drawio b/charts/victoria-metrics-distributed/img/victoriametrics-distributed-topology.drawio index 6e1606b4b..731093adc 100644 --- a/charts/victoria-metrics-distributed/img/victoriametrics-distributed-topology.drawio +++ b/charts/victoria-metrics-distributed/img/victoriametrics-distributed-topology.drawio @@ -1,242 +1,290 @@ - + - + - - + + - + - - + + - + - + - + - - + + - - - - - + + - - - - + - + - + - - - - - + + + + - - + + + + - + - + - + - + - + - - - - - - - - - - - + + + + - + - + - - + + - + - + - - + + - - + + - - - - - + + - - - - + - + - + - - - - - + + + + - - + + + + - + - + - + - + - + - - - - - - - - - - - + + + + - + - + - - + + - - + + - + - + - + - + - + - - + + - + - + - + - + - + - + - - + + - + - + - + - + - + - - + + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/charts/victoria-metrics-distributed/img/victoriametrics-distributed-topology.webp b/charts/victoria-metrics-distributed/img/victoriametrics-distributed-topology.webp index 3119cffe3..4eab6988b 100644 Binary files a/charts/victoria-metrics-distributed/img/victoriametrics-distributed-topology.webp and b/charts/victoria-metrics-distributed/img/victoriametrics-distributed-topology.webp differ diff --git a/charts/victoria-metrics-distributed/templates/extra-vmagent.yaml b/charts/victoria-metrics-distributed/templates/extra-vmagent.yaml deleted file mode 100644 index 63b9e4f01..000000000 --- a/charts/victoria-metrics-distributed/templates/extra-vmagent.yaml +++ /dev/null @@ -1,19 +0,0 @@ -{{- if ((.Values.extra).vmagent).enabled }} -{{- $ctx := dict "helm" . "appKey" (list "extra" "vmagent" "spec") -}} -{{- $ns := include "vm.namespace" $ctx }} ---- -apiVersion: operator.victoriametrics.com/v1beta1 -kind: VMAgent -metadata: - labels: {{ include "vm.labels" $ctx | nindent 4 }} - name: {{ include "vm.cr.fullname" $ctx }} - namespace: {{ $ns }} -{{- $spec := (((.Values).extra).vmagent).spec }} -{{- $write := deepCopy (.Values.write).global }} -{{- $_ := set $write.vmauth "spec" (mergeOverwrite (deepCopy $.Values.common.vmauth.spec) $write.vmauth.spec) }} -{{- $fullname := include "vm.fullname" . }} -{{- $ctx := dict "helm" . "appKey" (list "globalWrite" "vmauth" "spec") "globalWrite" $write "style" "managed" "fullname" $fullname }} -{{- $remoteWrites := list (dict "url" (printf "%s/prometheus/api/v1/write" (include "vm.url" $ctx))) }} -{{- $_ := set $spec "remoteWrite" (concat $remoteWrites ($spec.remoteWrite | default list)) }} -spec: {{ tpl (toYaml $spec) . | nindent 2 }} -{{- end }} diff --git a/charts/victoria-metrics-distributed/templates/grafana-datasource.yaml b/charts/victoria-metrics-distributed/templates/grafana-datasource.yaml index 751086d26..f1ed104a5 100644 --- a/charts/victoria-metrics-distributed/templates/grafana-datasource.yaml +++ b/charts/victoria-metrics-distributed/templates/grafana-datasource.yaml @@ -1,16 +1,21 @@ -{{ if and (((.Values.read).global).vmauth).enabled (index .Values "victoria-metrics-k8s-stack" "grafana" "enabled") }} -{{- $read := deepCopy (.Values.read).global }} -{{- $fullname := include "vm.fullname" . }} -{{- $_ := set $read.vmauth "spec" (mergeOverwrite (deepCopy $.Values.common.vmauth.spec) $read.vmauth.spec) }} -{{- $ctx := dict "helm" . "appKey" (list "globalRead" "vmauth" "spec") "globalRead" $read "style" "managed" "fullname" $fullname }} -{{- $url := (printf "%s/select/0/prometheus/" (include "vm.url" $ctx)) }} -{{- $ns := include "vm.namespace" $ctx }} +{{- $read := default dict }} +{{- if (((.Values.read).global).vmauth).enabled }} + {{- $read = deepCopy ((.Values.read).global).vmauth }} +{{- else if (((.Values.rw).global).vmauth).enabled }} + {{- $read = deepCopy ((.Values.rw).global).vmauth }} +{{- end }} +{{- $common := .Values.common.vmauth.spec }} +{{ if and $read.enabled (index .Values "victoria-metrics-k8s-stack" "grafana" "enabled") }} + {{- $fullname := include "vm.fullname" . }} + {{- $_ := set $read "spec" (mergeOverwrite (deepCopy $common) (deepCopy $read.spec)) }} + {{- $ctx := dict "helm" . "appKey" (list "globalRead" "vmauth" "spec") "globalRead" (dict "vmauth" $read) "style" "managed" "fullname" $fullname }} + {{- $url := (printf "%s/select/0/prometheus/" (include "vm.url" $ctx)) }} --- apiVersion: v1 kind: ConfigMap metadata: - name: {{ include "vm.fullname" . }}-grafana-ds - namespace: {{ $ns }} + name: {{ $fullname }}-grafana-ds + namespace: {{ include "vm.namespace" $ctx }} labels: {{ include "vm.labels" . | nindent 4 }} {{ index .Values "victoria-metrics-k8s-stack" "grafana" "sidecar" "datasources" "label" }}: "1" data: diff --git a/charts/victoria-metrics-distributed/templates/per-az/vmauth-read-proxy.yaml b/charts/victoria-metrics-distributed/templates/per-az/vmauth.yaml similarity index 54% rename from charts/victoria-metrics-distributed/templates/per-az/vmauth-read-proxy.yaml rename to charts/victoria-metrics-distributed/templates/per-az/vmauth.yaml index f969ed750..85aae7918 100644 --- a/charts/victoria-metrics-distributed/templates/per-az/vmauth-read-proxy.yaml +++ b/charts/victoria-metrics-distributed/templates/per-az/vmauth.yaml @@ -1,8 +1,11 @@ -{{- $ns := include "vm.namespace" . }} {{- $ctx := dict "helm" . }} {{- $urlCtx := dict "helm" . }} -{{- range $i, $z := .Values.availabilityZones }} - {{- $zone := mergeOverwrite (deepCopy $.Values.zoneTpl) (deepCopy $z) }} +{{- $values := .Values }} +{{- $common := $values.common | default dict }} +{{- $zoneTpl := $values.zoneTpl }} +{{- $defaultUrlMapItem := $values.defaultUrlMapItem }} +{{- range $i, $z := $values.availabilityZones }} + {{- $zone := mergeOverwrite (deepCopy $zoneTpl) (deepCopy $z) }} {{- if $zone.read.vmauth.enabled }} --- apiVersion: operator.victoriametrics.com/v1beta1 @@ -11,63 +14,66 @@ metadata: {{- $_ := set $ctx "appKey" (list "zone" "read" "vmauth") }} {{- if $zone.vmcluster.enabled }} {{- $urlSpec := $zone.vmcluster.spec | default dict }} - {{- $_ := set $zone.vmcluster "spec" (mergeOverwrite (deepCopy $.Values.common.vmcluster.spec) $urlSpec) }} + {{- $_ := set $zone.vmcluster "spec" (mergeOverwrite (deepCopy $common.vmcluster.spec) $urlSpec) }} {{- else if $zone.vmsingle.enabled }} {{- $urlSpec := $zone.vmsingle.spec | default dict }} - {{- $_ := set $zone.vmsingle "spec" (mergeOverwrite (deepCopy $.Values.common.vmsingle.spec) $urlSpec) }} + {{- $_ := set $zone.vmsingle "spec" (mergeOverwrite (deepCopy $common.vmsingle.spec) $urlSpec) }} {{- end }} {{- $_ := set $ctx "zone" $zone }} {{- $_ := set $urlCtx "zone" $zone }} name: {{ include "vm.cr.fullname" $ctx }} - namespace: {{ $ns }} + namespace: {{ include "vm.namespace" $ctx }} labels: {{ include "vm.labels" $ctx | nindent 4 }} {{- with $zone.read.vmauth.annotations }} annotations: {{ toYaml . | nindent 4 }} {{- end }} - {{- $spec := mergeOverwrite (deepCopy $.Values.common.vmauth.spec) (deepCopy (($zone.common).spec)) (deepCopy ((($zone.read).vmauth).spec)) }} + {{- $spec := mergeOverwrite (deepCopy $common.vmauth.spec) (deepCopy (($zone.common).spec)) (deepCopy ((($zone.read).vmauth).spec)) }} {{- $_ := set $urlCtx "style" "managed" }} - {{- $urls := dict "cluster" (default list) "single" (default list) }} + {{- $urls := dict "vmcluster" (default list) "vmsingle" (default list) }} {{- if $zone.read.allow }} {{- if ($zone.vmcluster).enabled }} {{- $_ := set $urlCtx "appKey" (list "zone" "vmcluster" "spec" "vmselect") }} - {{- $_ := set $urls "cluster" (append $urls.cluster (include "vm.url" $urlCtx)) }} + {{- $_ := set $urls "vmcluster" (append $urls.vmcluster (include "vm.url" $urlCtx)) }} {{- else if ($zone.vmsingle).enabled }} {{- $_ := set $urlCtx "appKey" (list "zone" "vmsingle" "spec") }} - {{- $_ := set $urls "single" (append $urls.single (include "vm.url" $urlCtx)) }} + {{- $_ := set $urls "vmsingle" (append $urls.vmsingle (include "vm.url" $urlCtx)) }} {{- end }} {{- end }} - {{- range $j, $c := $.Values.availabilityZones }} - {{- $cross := mergeOverwrite (deepCopy $.Values.zoneTpl) (deepCopy $c) }} + {{- range $j, $c := $values.availabilityZones }} + {{- $cross := mergeOverwrite (deepCopy $zoneTpl) (deepCopy $c) }} {{- if and (ne $j $i) $cross.read.allow }} {{- if ($cross.vmcluster).enabled }} {{- $_ := set $urlCtx "appKey" (list "zone" "vmcluster" "spec" "vmselect") }} - {{- $_ := set $cross.vmcluster "spec" (mergeOverwrite (deepCopy $.Values.common.vmcluster.spec) (deepCopy ( $cross.vmcluster.spec | default dict))) }} + {{- $_ := set $cross.vmcluster "spec" (mergeOverwrite (deepCopy $common.vmcluster.spec) (deepCopy ( $cross.vmcluster.spec | default dict))) }} {{- $_ := set $urlCtx "zone" $cross }} - {{- $_ := set $urls "cluster" (append $urls.cluster (include "vm.url" $urlCtx)) }} + {{- $_ := set $urls "vmcluster" (append $urls.vmcluster (include "vm.url" $urlCtx)) }} {{- else if ($cross.vmsingle).enabled }} {{- $_ := set $urlCtx "appKey" (list "zone" "vmsingle" "spec") }} - {{- $_ := set $cross.vmsingle "spec" (mergeOverwrite (deepCopy $.Values.common.vmsingle.spec) (deepCopy ( $cross.vmsingle.spec | default dict))) }} + {{- $_ := set $cross.vmsingle "spec" (mergeOverwrite (deepCopy $common.vmsingle.spec) (deepCopy ( $cross.vmsingle.spec | default dict))) }} {{- $_ := set $urlCtx "zone" $cross }} - {{- $_ := set $urls "single" (append $urls.single (include "vm.url" $urlCtx)) }} + {{- $_ := set $urls "vmsingle" (append $urls.vmsingle (include "vm.url" $urlCtx)) }} {{- end }} {{- end }} {{- end }} {{- $_ := unset $urlCtx "style" }} - {{- if and (empty $urls.cluster) (empty $urls.single) }} - {{- fail (printf "No read backend urls defined for vmauth in AZ %s" $zone.name) -}} - {{- end }} - {{- $paths := dict "cluster" (list "/select/.+" "/admin/tenants") "single" (list "/select/.+") }} - {{- $dropPrefixParts := dict "cluster" 0 "single" 2 }} + {{- $hasBackends := false }} + {{- range $_, $backends := $urls }} + {{- if gt (len $backends) 0 -}} + {{- $hasBackends = true -}} + {{- end -}} + {{- end -}} + {{- if not $hasBackends }} + {{- fail (printf "No backend urls for vmauth in AZ %q" $zone.name) -}} + {{- end -}} + {{- $accessSpec := $spec.unauthorizedUserAccessSpec | default dict }} - {{- $urlMap := $accessSpec.url_map | default (list (default dict)) }} - {{- $firstItem := index $urlMap 0 }} - {{- $urlMap = slice $urlMap 1 }} - {{- range $kind, $backends := $urls }} - {{- if $backends }} - {{- $item := dict "src_paths" (get $paths $kind) "url_prefix" $backends "drop_src_path_prefix_parts" (get $dropPrefixParts $kind) }} - {{- $item = mergeOverwrite (deepCopy $item) (deepCopy $firstItem) }} + {{- $urlMap := $accessSpec.url_map | default list }} + {{- range $type, $backendUrls := $urls }} + {{- if $backendUrls }} + {{- $item := deepCopy (index $defaultUrlMapItem "read" $type) }} + {{- $_ := set $item "url_prefix" $backendUrls }} {{- $urlMap = prepend $urlMap $item }} {{- end }} {{- end }} diff --git a/charts/victoria-metrics-distributed/templates/vmagent.yaml b/charts/victoria-metrics-distributed/templates/vmagent.yaml new file mode 100644 index 000000000..64ffa3171 --- /dev/null +++ b/charts/victoria-metrics-distributed/templates/vmagent.yaml @@ -0,0 +1,48 @@ +{{- if ((.Values.extra).vmagent).enabled }} +{{- $urlCtx := dict "helm" . }} +{{- $_ := set $urlCtx "style" "managed" }} +{{- $ctx := dict "helm" . "appKey" (list "extra" "vmagent" "spec") -}} +{{- $values := .Values }} +{{- $spec := (($values.extra).vmagent).spec }} +{{- $zoneTpl := $values.zoneTpl }} +{{- $common := .Values.common }} +{{- $isMultitenant := $values.enableMultitenancy }} +{{- $tenant := ternary "multitenant" "0" $isMultitenant }} + +{{- $urls := default list }} +{{- if $values.write.global.vmauth.enabled -}} + {{- $write := deepCopy (($values.write).global).vmauth }} + {{- $_ := set $write "spec" (mergeOverwrite (deepCopy $common.vmauth.spec) $write.spec) }} + {{- $fullname := include "vm.fullname" . }} + {{- $urlCtx := dict "helm" . "appKey" (list "globalWrite" "vmauth" "spec") "globalWrite" (dict "vmauth" $write) "style" "managed" "fullname" $fullname }} + {{- $urls = append $urls (dict "url" (printf "%s/prometheus/api/v1/write" (include "vm.url" $urlCtx))) }} +{{- else }} + {{- range $i, $z := $values.availabilityZones }} + {{- $zone := mergeOverwrite (deepCopy $zoneTpl) $z }} + {{- if $zone.vmcluster.enabled }} + {{- $urlSpec := mergeOverwrite (deepCopy ((($common.vmcluster).spec | default dict))) (deepCopy ($zone.vmcluster).spec | default dict) }} + {{- $_ := set $zone.vmcluster "spec" $urlSpec }} + {{- $_ := set $urlCtx "zone" $zone }} + {{- $_ := set $urlCtx "appKey" (list "zone" "vmcluster" "spec" "vminsert") }} + {{- $urls = append $urls (dict "url" (printf "%s/insert/%s/prometheus/api/v1/write" (include "vm.url" $urlCtx) $tenant)) }} + {{- else if $zone.vmsingle.enabled }} + {{- $urlSpec := mergeOverwrite (deepCopy ((($common.vmsingle).spec | default dict))) (deepCopy ($zone.vmsingle).spec | default dict) }} + {{- $_ := set $zone.vmsingle "spec" $urlSpec }} + {{- $_ := set $urlCtx "zone" $zone }} + {{- $_ := set $urlCtx "appKey" (list "zone" "vmsingle" "spec") }} + {{- $urls = append $urls (dict "url" (printf "%s/prometheus/api/v1/write" (include "vm.url" $urlCtx))) }} + {{- end }} + {{- end }} +{{- end -}} + +{{- $_ := set $spec "remoteWrite" (concat ($spec.remoteWrite | default list) $urls) }} +--- +apiVersion: operator.victoriametrics.com/v1beta1 +kind: VMAgent +metadata: + labels: {{ include "vm.labels" $ctx | nindent 4 }} + name: {{ include "vm.cr.fullname" $ctx }} + namespace: {{ include "vm.namespace" $ctx }} +spec: {{ tpl (toYaml $spec) . | nindent 2 }} +{{- $_ := unset $urlCtx "style" -}} +{{- end }} diff --git a/charts/victoria-metrics-distributed/templates/vmauth-read.yaml b/charts/victoria-metrics-distributed/templates/vmauth-read.yaml deleted file mode 100644 index d70c9855b..000000000 --- a/charts/victoria-metrics-distributed/templates/vmauth-read.yaml +++ /dev/null @@ -1,70 +0,0 @@ -{{- $ctx := dict "helm" . }} -{{- $zoneTpl := .Values.zoneTpl }} -{{- $common := .Values.common | default dict }} -{{- $urlCtx := dict "helm" . }} -{{- $fullname := include "vm.fullname" . }} -{{- if ((((.Values).read).global).vmauth).enabled }} - {{- $commonAuth := dict "spec" (($common.vmauth).spec | default dict) }} - {{- $auth := .Values.read.global.vmauth | default dict }} - {{- $auth = mergeOverwrite (deepCopy $commonAuth) (deepCopy $auth) }} - {{- $_ := set $ctx "appKey" (list "vmauth" "spec") }} - {{- $_ := set $ctx "fullname" $fullname }} - {{- $_ := set $ctx "vmauth" $auth }} ---- -apiVersion: operator.victoriametrics.com/v1beta1 -kind: VMAuth -metadata: - name: {{ include "vm.cr.fullname" $ctx }} - namespace: {{ include "vm.namespace" $ctx }} - labels: {{ include "vm.labels" $ctx | nindent 4 }} - {{- with .Values.read.global.vmauth.annotations }} - annotations: {{ toYaml . | nindent 4 }} - {{- end }} - - {{- $_ := set $urlCtx "style" "managed" }} - {{- $urls := dict "cluster" (default list) "single" (default list) }} - {{- range $i, $z := .Values.availabilityZones }} - {{- $zone := mergeOverwrite (deepCopy $zoneTpl) $z }} - {{- if $zone.read.vmauth.enabled -}} - {{- $urlSpec := (($zone.read).vmauth).spec | default dict }} - {{- $_ := set $zone.read.vmauth "spec" (mergeOverwrite (deepCopy $commonAuth.spec) $urlSpec) }} - {{- $_ := set $urlCtx "zone" $zone }} - {{- $_ := set $urlCtx "appKey" (list "zone" "read" "vmauth" "spec") }} - {{- $_ = set $urls "cluster" (append $urls.cluster (include "vm.url" $urlCtx)) }} - {{- else if $zone.vmcluster.enabled -}} - {{- $urlSpec := mergeOverwrite (deepCopy ((($common.vmcluster).spec | default dict))) (deepCopy ($zone.vmcluster).spec | default dict) }} - {{- $_ := set $zone.vmcluster "spec" $urlSpec }} - {{- $_ := set $urlCtx "zone" $zone }} - {{- $_ := set $urlCtx "appKey" (list "zone" "vmcluster" "spec" "vmselect") }} - {{- $_ := set $urls "cluster" (append $urls.cluster (include "vm.url" $urlCtx)) }} - {{- else if $zone.vmsingle.enabled -}} - {{- $urlSpec := mergeOverwrite (deepCopy ((($common.vmsingle).spec | default dict))) (deepCopy ($zone.vmsingle).spec | default dict) }} - {{- $_ := set $zone.vmsingle "spec" $urlSpec }} - {{- $_ := set $urlCtx "zone" $zone }} - {{- $_ := set $urlCtx "appKey" (list "zone" "vmsingle" "spec") }} - {{- $_ := set $urls "single" (append $urls.single (include "vm.url" $urlCtx)) }} - {{- end -}} - {{- end }} - {{- $_ := unset $urlCtx "style" }} - {{- if and (empty $urls.cluster) (empty $urls.single) }} - {{- fail "No global vmauth read backend urls defined" -}} - {{- end }} - {{- $paths := dict "cluster" (list "/select/.+" "/admin/tenants") "single" (list "/select/.+") }} - {{- $dropPrefixParts := dict "cluster" 0 "single" 2 }} - {{- $spec := deepCopy $auth.spec | default dict -}} - {{- $accessSpec := $spec.unauthorizedUserAccessSpec | default dict }} - {{- $urlMap := $accessSpec.url_map | default (list (default dict)) }} - {{- $firstItem := index $urlMap 0 }} - {{- $urlMap = slice $urlMap 1 }} - {{- range $kind, $backends := $urls }} - {{- if $backends }} - {{- $item := dict "src_paths" (get $paths $kind) "url_prefix" $backends "drop_src_path_prefix_parts" (get $dropPrefixParts $kind) }} - {{- $item = mergeOverwrite (deepCopy $item) (deepCopy $firstItem) }} - {{- $urlMap = prepend $urlMap $item }} - {{- end }} - {{- end }} - {{- $_ := set $accessSpec "url_map" $urlMap }} - {{- $_ := set $spec "unauthorizedUserAccessSpec" $accessSpec }} -spec: {{ tpl (toYaml $spec) $ctx | nindent 2 }} -{{- end }} - diff --git a/charts/victoria-metrics-distributed/templates/vmauth-write.yaml b/charts/victoria-metrics-distributed/templates/vmauth-write.yaml deleted file mode 100644 index 90eba76fb..000000000 --- a/charts/victoria-metrics-distributed/templates/vmauth-write.yaml +++ /dev/null @@ -1,48 +0,0 @@ -{{- $ctx := dict "helm" . }} -{{- $urlCtx := dict "helm" . }} -{{- $fullname := include "vm.fullname" . }} -{{- if ((((.Values).write).global).vmauth).enabled }} - {{- $commonAuth := dict "spec" ((((.Values).common).vmauth).spec | default dict) }} - {{- $auth := .Values.write.global.vmauth | default dict }} - {{- $auth = mergeOverwrite (deepCopy $commonAuth) (deepCopy $auth) }} - {{- $_ := set $ctx "appKey" (list "vmauth" "spec") }} - {{- $_ := set $ctx "fullname" $fullname }} - {{- $_ := set $ctx "vmauth" $auth }} ---- -apiVersion: operator.victoriametrics.com/v1beta1 -kind: VMAuth -metadata: - name: {{ include "vm.cr.fullname" $ctx }} - namespace: {{ include "vm.namespace" $ctx }} - labels: {{ include "vm.labels" $ctx | nindent 4 }} - {{- with ((((.Values).write).global).vmauth).annotations }} - annotations: {{ toYaml . | nindent 4 }} - {{- end }} - - {{- $_ := set $urlCtx "style" "managed" }} - {{- $urls := default list }} - {{- $commonAgentSpec := (((.Values).common).vmagent).spec | default dict }} - {{- range $i, $z := $.Values.availabilityZones }} - {{- $zone := mergeOverwrite (deepCopy $.Values.zoneTpl) $z }} - {{- if $zone.vmagent.enabled -}} - {{- $agentSpec := $zone.vmagent.spec | default dict }} - {{- $_ := set $zone.vmagent "spec" (mergeOverwrite (deepCopy $commonAgentSpec) $agentSpec) }} - {{- $_ := set $urlCtx "zone" $zone }} - {{- $_ := set $urlCtx "appKey" (list "zone" "vmagent" "spec") }} - {{- $urls = append $urls (include "vm.url" $urlCtx) }} - {{- end -}} - {{- end }} - {{- $_ := unset $urlCtx "style" }} - {{- if empty $urls }} - {{- fail "No global vmauth write backend urls defined" -}} - {{- end }} - {{- $defaultPaths := list "/api/v1/write" "/prometheus/api/v1/write" "/write" "/api/v1/import" "/api/v1/import/.+" "/opentelemetry/.+" "/insert/.+"}} - {{- $defaultUrlMapItem := dict "src_paths" $defaultPaths "url_prefix" $urls }} - {{- $spec := deepCopy $auth.spec | default dict -}} - {{- $accessSpec := $spec.unauthorizedUserAccessSpec | default dict }} - {{- $urlMap := $accessSpec.url_map | default (list (default dict)) }} - {{- $firstUrlMapItem := mergeOverwrite $defaultUrlMapItem (deepCopy (index $urlMap 0)) }} - {{- $_ := set $accessSpec "url_map" (prepend (slice $urlMap 1) $firstUrlMapItem) }} - {{- $_ := set $spec "unauthorizedUserAccessSpec" $accessSpec }} -spec: {{ tpl (toYaml $spec) $ctx | nindent 2 }} -{{- end }} diff --git a/charts/victoria-metrics-distributed/templates/vmauth.yaml b/charts/victoria-metrics-distributed/templates/vmauth.yaml new file mode 100644 index 000000000..d3e3da0f0 --- /dev/null +++ b/charts/victoria-metrics-distributed/templates/vmauth.yaml @@ -0,0 +1,99 @@ +{{- $ctx := dict "helm" . }} +{{- $urlCtx := dict "helm" . }} +{{- $values := .Values }} +{{- $common := $values.common | default dict }} +{{- $zoneTpl := $values.zoneTpl }} +{{- $fullname := include "vm.fullname" . }} +{{- $kinds := list "read" "write" }} +{{- $defaultUrlMapItem := $values.defaultUrlMapItem }} +{{- range $kind := $kinds }} + {{- $auth := index $values $kind "global" "vmauth" | default dict }} + {{- if $auth.enabled }} + {{- $commonAuth := dict "spec" (deepCopy (($common.vmauth).spec | default dict)) }} + {{- $commonAgent := dict "spec" (($common.vmagent).spec | default dict) }} + {{- $auth = mergeOverwrite (deepCopy $commonAuth) (deepCopy $auth) }} + {{- $_ := set $ctx "appKey" (list "vmauth" "spec") }} + {{- $_ := set $ctx "fullname" $fullname }} + {{- $_ := set $ctx "vmauth" $auth }} +--- +apiVersion: operator.victoriametrics.com/v1beta1 +kind: VMAuth +metadata: + name: {{ include "vm.cr.fullname" $ctx }} + namespace: {{ include "vm.namespace" $ctx }} + labels: {{ include "vm.labels" $ctx | nindent 4 }} + {{- with $auth.annotations }} + annotations: {{ toYaml . | nindent 4 }} + {{- end }} + + {{- $_ := set $urlCtx "style" "managed" }} + {{- $urls := dict "vmcluster" (default list) "vmsingle" (default list) "vmagent" (default list) }} + {{- range $i, $z := $values.availabilityZones }} + {{- $zone := mergeOverwrite (deepCopy $zoneTpl) $z }} + {{- if has $kind (list "read") -}} + {{- if $zone.read.vmauth.enabled -}} + {{- $urlSpec := (($zone.read).vmauth).spec | default dict }} + {{- $_ := set $zone.read.vmauth "spec" (mergeOverwrite (deepCopy $commonAuth.spec) $urlSpec) }} + {{- $_ := set $urlCtx "zone" $zone }} + {{- $_ := set $urlCtx "appKey" (list "zone" "read" "vmauth" "spec") }} + {{- if $zone.vmcluster.enabled -}} + {{- $_ = set $urls "vmcluster" (append $urls.vmcluster (include "vm.url" $urlCtx)) -}} + {{- else if $zone.vmsingle.enabled -}} + {{- $_ := set $urls "vmsingle" (append $urls.vmsingle (include "vm.url" $urlCtx)) }} + {{- end -}} + {{- else if $zone.vmcluster.enabled -}} + {{- $urlSpec := mergeOverwrite (deepCopy ((($common.vmcluster).spec | default dict))) (deepCopy ($zone.vmcluster).spec | default dict) }} + {{- $_ := set $zone.vmcluster "spec" $urlSpec }} + {{- $_ := set $urlCtx "zone" $zone }} + {{- $_ := set $urlCtx "appKey" (list "zone" "vmcluster" "spec" "vmselect") }} + {{- $_ := set $urls "vmcluster" (append $urls.vmcluster (include "vm.url" $urlCtx)) }} + {{- else if $zone.vmsingle.enabled -}} + {{- $urlSpec := mergeOverwrite (deepCopy ((($common.vmsingle).spec | default dict))) (deepCopy ($zone.vmsingle).spec | default dict) }} + {{- $_ := set $zone.vmsingle "spec" $urlSpec }} + {{- $_ := set $urlCtx "zone" $zone }} + {{- $_ := set $urlCtx "appKey" (list "zone" "vmsingle" "spec") }} + {{- $_ := set $urls "vmsingle" (append $urls.vmsingle (include "vm.url" $urlCtx)) }} + {{- end -}} + {{- end -}} + {{- if has $kind (list "write") -}} + {{- if $zone.vmagent.enabled -}} + {{- $agentSpec := $zone.vmagent.spec | default dict }} + {{- $_ := set $zone.vmagent "spec" (mergeOverwrite (deepCopy $commonAgent.spec) $agentSpec) }} + {{- $_ := set $urlCtx "zone" $zone }} + {{- $_ := set $urlCtx "appKey" (list "zone" "vmagent" "spec") }} + {{- $_ := set $urls "vmagent" (append $urls.vmagent (include "vm.url" $urlCtx)) }} + {{- end -}} + {{- end -}} + + {{- end }} + {{- $_ := unset $urlCtx "style" -}} + + {{- $hasBackends := false }} + {{- range $_, $backends := $urls }} + {{- if gt (len $backends) 0 -}} + {{- $hasBackends = true -}} + {{- end -}} + {{- end -}} + {{- if not $hasBackends }} + {{- fail (printf "No backend urls for global %q vmauth" $kind) -}} + {{- end -}} + + {{- $spec := deepCopy $auth.spec | default dict -}} + {{- $accessSpec := $spec.unauthorizedUserAccessSpec | default dict }} + {{- $urlMap := $accessSpec.url_map | default list }} + + {{- range $type, $backends := $urls }} + {{- if $backends }} + {{- $itemKind := ternary "write" "read" (eq $type "vmagent") }} + {{- $item := deepCopy (index $defaultUrlMapItem $itemKind $type ) }} + {{- $_ := set $item "url_prefix" $backends }} + {{- $urlMap = prepend $urlMap $item }} + {{- end }} + {{- end }} + {{- $_ := set $accessSpec "url_map" $urlMap }} + {{- $_ := set $spec "unauthorizedUserAccessSpec" $accessSpec }} +spec: {{ tpl (toYaml $spec) $ctx | nindent 2 }} + + {{- end }} +{{- end }} + diff --git a/charts/victoria-metrics-distributed/values.yaml b/charts/victoria-metrics-distributed/values.yaml index 0ef882c39..53099b350 100644 --- a/charts/victoria-metrics-distributed/values.yaml +++ b/charts/victoria-metrics-distributed/values.yaml @@ -37,6 +37,39 @@ common: spec: port: "8428" +# -- Default VMAuth url_map item for each access type for each backend type +defaultUrlMapItem: + write: + vmagent: + src_paths: + - /api/v1/write + - /prometheus/api/v1/write + - /write + - /api/v1/import + - /api/v1/import/.+ + - /opentelemetry/.+ + - /insert/.+ + vmcluster: + src_paths: + - /insert/.+ + vmsingle: + drop_src_path_prefix_parts: 2 + src_paths: + - /insert/.+ + read: + vmcluster: + load_balancing_policy: first_available + retry_status_codes: [500, 502, 503] + src_paths: + - /select/.+ + - /admin/tenants + vmsingle: + load_balancing_policy: first_available + retry_status_codes: [500, 502, 503] + drop_src_path_prefix_parts: 2 + src_paths: + - /select/.+ + write: global: vmauth: @@ -55,11 +88,7 @@ read: # -- Override the name of the vmauth object name: "vmauth-global-read-{{ .fullname }}" # -- Spec for VMAuth CRD, see [here](https://docs.victoriametrics.com/operator/api/#vmauthspec) - spec: - unauthorizedUserAccessSpec: - url_map: - - load_balancing_policy: first_available - retry_status_codes: [500, 502, 503] + spec: {} # -- Default config for each availability zone components, including vmagent, vmcluster, vmsingle, vmauth etc. # Defines a template for each availability zone, which can be overridden for each availability zone at `availabilityZones[*]` @@ -97,11 +126,7 @@ zoneTpl: # -- Override the name of the vmauth object name: "vmauth-read-proxy-{{ (.zone).name }}" # -- Spec for VMAuth CRD, see [here](https://docs.victoriametrics.com/operator/api/#vmauthspec) - spec: - unauthorizedUserAccessSpec: - url_map: - - load_balancing_policy: first_available - retry_status_codes: [500, 502, 503] + spec: {} # vmagent here only meant to proxy write requests to each az, # doesn't support customized other remote write address. vmagent: @@ -193,7 +218,7 @@ availabilityZones: extra: vmagent: enabled: false - name: test-vmagent + name: extra spec: selectAllByDefault: true