Skip to content

operations: add per-zone config for enabling "ingester multi-az" #15000

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ldufr merged 1 commit into from
Apr 13, 2026
Merged

operations: add per-zone config for enabling "ingester multi-az" #15000

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -375,6 +375,7 @@
* [ENHANCEMENT] Multi-zone: Add config validation for `-querier.prefer-availability-zones` flag on querier and ruler-querier deployments. #14539
* [ENHANCEMENT] Distributor: render the experimental `-distributor.max-active-series-per-user` flag on distributor if `$._config.limits.max_active_series_per_user` is set. #14636
* [ENHANCEMENT] Ingester: Add `$._config.ingest_storage_set_client_rack` to pass `-ingest-storage.kafka.client-rack` when zone-aware replication is enabled. #14654
* [ENHANCEMENT] Ingester: Add `$._config.multi_zone_ingester_multi_az_zone_(a|b|c)_enabled` to simplify migrations not using a temporary zone-c. #15000
* [BUGFIX] Ingester: Fix `$._config.ingest_storage_ingester_autoscaling_max_owned_series_threshold` default value, to compute it based on the configured `$._config.ingester_instance_limits.max_series`. #13448

### Documentation
Expand Down
13 changes: 13 additions & 0 deletions operations/mimir-tests/test-multi-az-read-path-migration-step-1-generated.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4374,6 +4374,14 @@ spec:
rollout-group: ingester
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- us-east-2a
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
Expand Down Expand Up @@ -4468,6 +4476,11 @@ spec:
securityContext:
runAsUser: 0
terminationGracePeriodSeconds: 1200
tolerations:
- effect: NoSchedule
key: topology
operator: Equal
value: secondary-az
volumes:
- configMap:
name: overrides
Expand Down
4 changes: 4 additions & 0 deletions operations/mimir-tests/test-multi-az-read-path-migration-step-1.jsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,9 @@
multi_zone_query_frontend_routing_enabled: false,
multi_zone_ruler_routing_enabled: false,
multi_zone_memcached_routing_enabled: false,

// Enable multi-az config for the ingester zone-a to prevent a rollout
// of zone-a when `multi_zone_ingester_multi_az_enabled` is set to true.
multi_zone_ingester_zone_a_multi_az_enabled: true,
},
}
13 changes: 13 additions & 0 deletions operations/mimir-tests/test-multi-az-read-path-migration-step-2-generated.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4426,6 +4426,14 @@ spec:
rollout-group: ingester
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- us-east-2a
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
Expand Down Expand Up @@ -4520,6 +4528,11 @@ spec:
securityContext:
runAsUser: 0
terminationGracePeriodSeconds: 1200
tolerations:
- effect: NoSchedule
key: topology
operator: Equal
value: secondary-az
volumes:
- configMap:
name: overrides
Expand Down
13 changes: 13 additions & 0 deletions operations/mimir-tests/test-multi-az-read-path-migration-step-2a-generated.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4426,6 +4426,14 @@ spec:
rollout-group: ingester
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- us-east-2a
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
Expand Down Expand Up @@ -4520,6 +4528,11 @@ spec:
securityContext:
runAsUser: 0
terminationGracePeriodSeconds: 1200
tolerations:
- effect: NoSchedule
key: topology
operator: Equal
value: secondary-az
volumes:
- configMap:
name: overrides
Expand Down
13 changes: 13 additions & 0 deletions operations/mimir-tests/test-multi-az-read-path-migration-step-2b-generated.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4426,6 +4426,14 @@ spec:
rollout-group: ingester
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- us-east-2a
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
Expand Down Expand Up @@ -4520,6 +4528,11 @@ spec:
securityContext:
runAsUser: 0
terminationGracePeriodSeconds: 1200
tolerations:
- effect: NoSchedule
key: topology
operator: Equal
value: secondary-az
volumes:
- configMap:
name: overrides
Expand Down
13 changes: 13 additions & 0 deletions operations/mimir-tests/test-multi-az-read-path-migration-step-2c-generated.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4426,6 +4426,14 @@ spec:
rollout-group: ingester
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- us-east-2a
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
Expand Down Expand Up @@ -4520,6 +4528,11 @@ spec:
securityContext:
runAsUser: 0
terminationGracePeriodSeconds: 1200
tolerations:
- effect: NoSchedule
key: topology
operator: Equal
value: secondary-az
volumes:
- configMap:
name: overrides
Expand Down
13 changes: 13 additions & 0 deletions operations/mimir-tests/test-multi-az-read-path-migration-step-2d-generated.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4426,6 +4426,14 @@ spec:
rollout-group: ingester
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- us-east-2a
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
Expand Down Expand Up @@ -4520,6 +4528,11 @@ spec:
securityContext:
runAsUser: 0
terminationGracePeriodSeconds: 1200
tolerations:
- effect: NoSchedule
key: topology
operator: Equal
value: secondary-az
volumes:
- configMap:
name: overrides
Expand Down
13 changes: 13 additions & 0 deletions operations/mimir-tests/test-multi-az-read-path-migration-step-2e-generated.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4403,6 +4403,14 @@ spec:
rollout-group: ingester
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- us-east-2a
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
Expand Down Expand Up @@ -4497,6 +4505,11 @@ spec:
securityContext:
runAsUser: 0
terminationGracePeriodSeconds: 1200
tolerations:
- effect: NoSchedule
key: topology
operator: Equal
value: secondary-az
volumes:
- configMap:
name: overrides
Expand Down
13 changes: 13 additions & 0 deletions operations/mimir-tests/test-multi-az-read-path-migration-step-2f-generated.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4403,6 +4403,14 @@ spec:
rollout-group: ingester
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- us-east-2a
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
Expand Down Expand Up @@ -4497,6 +4505,11 @@ spec:
securityContext:
runAsUser: 0
terminationGracePeriodSeconds: 1200
tolerations:
- effect: NoSchedule
key: topology
operator: Equal
value: secondary-az
volumes:
- configMap:
name: overrides
Expand Down
13 changes: 8 additions & 5 deletions operations/mimir/multi-zone-ingester.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@

// Controls whether the multi (virtual) zone ingester should also be deployed multi-AZ.
multi_zone_ingester_multi_az_enabled: $._config.multi_zone_read_path_multi_az_enabled,
multi_zone_ingester_zone_a_multi_az_enabled: self.multi_zone_ingester_multi_az_enabled,
multi_zone_ingester_zone_b_multi_az_enabled: self.multi_zone_ingester_multi_az_enabled,
multi_zone_ingester_zone_c_multi_az_enabled: self.multi_zone_ingester_multi_az_enabled,
},

local container = $.core.v1.container,
Expand All @@ -27,12 +30,12 @@
local service = $.core.v1.service,
local podAntiAffinity = $.apps.v1.deployment.mixin.spec.template.spec.affinity.podAntiAffinity,

local isMultiAZEnabled = $._config.multi_zone_ingester_multi_az_enabled,
local isZoneAEnabled = isMultiAZEnabled && std.length($._config.multi_zone_availability_zones) >= 1,
local isZoneBEnabled = isMultiAZEnabled && std.length($._config.multi_zone_availability_zones) >= 2,
local isZoneCEnabled = isMultiAZEnabled && std.length($._config.multi_zone_availability_zones) >= 3,
Comment thread
cursor[bot] marked this conversation as resolved.
local isZoneAEnabled = $._config.multi_zone_ingester_zone_a_multi_az_enabled && std.length($._config.multi_zone_availability_zones) >= 1,
local isZoneBEnabled = $._config.multi_zone_ingester_zone_b_multi_az_enabled && std.length($._config.multi_zone_availability_zones) >= 2,
local isZoneCEnabled = $._config.multi_zone_ingester_zone_c_multi_az_enabled && std.length($._config.multi_zone_availability_zones) >= 3,

assert !isMultiAZEnabled || $._config.multi_zone_ingester_enabled : 'ingester multi-AZ deployment requires ingester multi-zone to be enabled',
local isMultiAZAtLeastOnceEnabled = isZoneAEnabled || isZoneBEnabled || isZoneCEnabled,
assert !isMultiAZAtLeastOnceEnabled || $._config.multi_zone_ingester_enabled : 'ingester multi-AZ deployment requires ingester multi-zone to be enabled',
Comment thread
ldufr marked this conversation as resolved.
assert !$._config.multi_zone_ingester_zpdb_enabled || $._config.rollout_operator_webhooks_enabled : 'zpdb configuration requires rollout_operator_webhooks_enabled=true',

//
Expand Down
Loading