Proposal: Availability Zone aware target allocation for consistent-hashing and least-weighted strategies

[szibis]

Summary

Add topology/availability zone (AZ) awareness to the Target Allocator's consistent-hashing and least-weighted allocation strategies, so that targets are preferentially assigned to collectors running in the same AZ. This minimizes cross-AZ network traffic and cloud data transfer costs during Prometheus scraping.

Motivation

In multi-AZ Kubernetes clusters on AWS/GCP/Azure:

• Cross-AZ data transfer is billed ($0.01-0.02/GB in both directions)
• The Target Allocator currently assigns targets without considering topology — a collector in us-east-1a may scrape targets in us-east-1c
• At scale (thousands of targets, 15-30s scrape intervals), cross-AZ traffic accumulates significantly
• The per-node strategy avoids this but requires DaemonSet mode and cannot balance load

Proposed Design

Approach: Extend existing strategies with built-in AZ awareness

Rather than creating new strategies or a decorator pattern, add zone-aware logic directly into consistent-hashing and least-weighted. Activated via a new topology config section:

allocation_strategy: "least-weighted"
topology:
  zone_aware: true
  zone_label: "topology.kubernetes.io/zone"  # default

When zone_aware: false (default), behavior is identical to today. Zero breaking changes.

Algorithm

GetCollectorForTarget(collectors, target):
  target_zone = target.Labels.Get(target_zone_label)

  IF target_zone != "" AND same_zone_collectors_exist(target_zone):
    return inner_strategy(same_zone_collectors_only, target)
  ELSE:
    return inner_strategy(all_collectors, target)  # FAILOVER

• consistent-hashing: Maintains per-zone hash rings + one global ring. Same-zone → zone ring. Failover → global ring.
• least-weighted: Maintains collectorsPerZone index. Same-zone → pick least-loaded in zone. Failover → pick least-loaded globally.

Zone Detection

Collectors: NodeZoneResolver watches K8s Node objects, maps pod.Spec.NodeName → topology.kubernetes.io/zone label. Populates new Collector.Zone field.

Targets: Read from __meta_kubernetes_endpointslice_endpoint_zone Prometheus SD label (available since Prometheus 2.48). Falls back to node-based resolution.

Failover

• If target's AZ has no collectors → distribute across ALL collectors using base strategy
• Targets without zone info → global distribution (no zone preference)
• Clusters without zone labels → zone awareness is a no-op (graceful degradation)

Observability

New metrics:

• opentelemetry_allocator_zone_cross_zone_assignments — targets assigned to different AZ
• opentelemetry_allocator_zone_uncovered_count — zones with targets but no collectors
• opentelemetry_allocator_zone_collector_count{zone} — collectors per AZ
• opentelemetry_allocator_zone_target_count{zone} — targets per AZ

New API endpoint: GET /zones — returns zone topology snapshot (collector/target distribution, uncovered zones, balance ratio)

Log warnings when zone coverage gaps detected.

Example Scenarios

4 collectors, 3 AZs (1 in az-a, 1 in az-b, 2 in az-c):

• az-a targets → collector-0
• az-b targets → collector-1
• az-c targets → collector-2 + collector-3 (split 50/50 by inner strategy)

2 collectors, 3 AZs (1 in az-a, 1 in az-b, 0 in az-c):

• az-a targets → collector-0 (same-zone)
• az-b targets → collector-1 (same-zone)
• az-c targets → FAILOVER spread across collector-0 + collector-1
• Metric: zone_uncovered_count=1, log warning emitted

Implementation Plan

1. Config + Collector.Zone field + NodeZoneResolver — adds zone field, no behavior change
2. ZoneTopology map + metrics — builds topology, records metrics
3. Zone-aware consistent-hashing — per-zone hash rings with failover
4. Zone-aware least-weighted — per-zone collector index with failover
5. API /zones endpoint + HTML views — read-only topology endpoints
6. Integration tests + documentation

Kubernetes Alignment

• Uses standard topology.kubernetes.io/zone label (stable since K8s 1.17)
• Leverages EndpointSlice zone field (K8s 1.21+)
• Mirrors Kubernetes topology-aware routing hints pattern (KEP-2433)
• Complements topologySpreadConstraints on collector StatefulSet

RBAC

Requires nodes watch permission (cluster-scoped) for the zone resolver:

- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch"]

Alternatives Considered

1. Decorator/wrapper pattern — rejected for complexity; integrating into strategies is cleaner
2. New strategy names (e.g., zone-consistent-hashing) — rejected; combinatorial explosion
3. Prometheus relabel_configs — works for standalone but doesn't solve TA-managed allocation
4. Extend loadbalancingexporter — solves a different problem (export routing, not scrape assignment)

Open Questions

1. Should zone info be provided via config/annotations as alternative to node label lookup (for restricted RBAC environments)?

/kind feature
/area target-allocator

[szibis]

Design update: maxSkew configuration

Adding a maxSkew parameter to the proposal so operators can control the tradeoff between zone affinity (minimize cross-AZ traffic) and load balance (avoid hotspots). The semantics mirror Kubernetes' topologySpreadConstraints.maxSkew so the concept is already familiar to operators.

Problem maxSkew solves

Pure zone affinity can produce severe load imbalance when targets are not evenly distributed across zones. Example:

• zone-a: 200 targets + 1 collector
• zone-b: 10 targets + 1 collector

Without a skew limit, the zone-a collector handles 200 targets and the zone-b collector handles 10 — a 20:1 imbalance. Cross-zone traffic is zero, but one collector is overloaded and may drop scrapes.

Updated configuration

allocation_strategy: ""least-weighted""
topology:
  zone_aware: true
  max_skew: 10                                                       # new field, default 0 (disabled)
  zone_label: ""topology.kubernetes.io/zone""                          # default
  target_zone_label: ""__meta_kubernetes_endpointslice_endpoint_zone"" # default for K8s SD

Semantics

Define ${\rm skew} = \max_i({\rm Collector}_i.{\rm NumTargets}) - \min_i({\rm Collector}_i.{\rm NumTargets})$ across all collectors in all zones.

When assigning a target to its desired zone:

1. Find the least-loaded same-zone collector (call it $C_{\rm zone}$).
2. If $C_{\rm zone}.{\rm NumTargets} + 1 - \min_i({\rm Collector}_i.{\rm NumTargets}) > {\rm maxSkew}$, the assignment would push the global skew over the limit. In that case, assign the target to the globally least-loaded collector instead (cross-zone ""spillover"").
3. Otherwise, assign to $C_{\rm zone}$ (zone-local, default behavior).

Recommended values

max_skew	Behavior	When to use
0 (default)	Pure zone affinity — no skew limit. Targets always stay in-zone when possible.	Minimize cross-AZ egress at all costs. Acceptable when zones are well-balanced or when a few hot collectors are tolerable.
1	Near-uniform distribution across all collectors. Zone affinity has minimal effect.	Not generally recommended — defeats the purpose of zone awareness.
5–20	Practical range. Keeps collectors within a reasonable load band while still preferring same-zone assignment.	Most production deployments.
50+	Loose skew limit. Strong zone affinity, allows spillover only for extreme imbalance.	Cost-optimization-heavy deployments with somewhat-balanced zones.

Per-strategy implementation

• least-weighted: natural fit. Already iterates collectors to pick least-loaded; just add a skew pre-check before constraining to same-zone collectors. Cheap to enforce per-target.
• consistent-hashing: per-zone hash rings are used by default for zone affinity. When the chosen zone-ring collector would exceed maxSkew, fall back to the global hash ring for that target. Spillover targets land deterministically (still hash-based) but on the global ring rather than the zone ring.

Observability

Every spillover event is recorded with both originating and destination zones:

opentelemetry_allocator_zone_spillover{from_zone=""us-east-1a"", to_zone=""us-east-1c""} 42

Operators can use this metric to verify their maxSkew setting matches their cost/reliability tradeoff. If spillover is unexpectedly high, either zones are extremely imbalanced (scale collectors differently per zone) or maxSkew is too tight.

Validation

The allocator rejects misconfigurations at startup:

• max_skew < 0 → error: ""topology.max_skew must be >= 0 (0 disables skew limiting)""
• max_skew > 0 && !zone_aware → error: ""topology.max_skew requires topology.zone_aware to be enabled""

Failover vs spillover (terminology)

Two distinct conditions trigger cross-zone assignment, both surfaced via the same zone_spillover counter (distinguishable by from_zone/to_zone labels):

1. Failover — target's desired zone has no collectors at all (zone is uncovered). Allocation falls back to the global pool. This is unconditional and not affected by maxSkew.
2. Spillover — target's desired zone has collectors, but assigning there would exceed maxSkew. Allocation goes to the globally least-loaded collector. This is gated by maxSkew > 0.

This is part of PR1 (foundation, already in progress). The actual enforcement lands in PR3 (zone-aware consistent-hashing) and PR4 (zone-aware least-weighted).

[szibis]

Why this matters at scale: cost & bandwidth savings

Same-AZ scraping is the central economic argument for this feature. Prometheus scrapes are continuous: thousands of targets at 15–30s intervals add up to terabytes per month of east-west traffic. When that traffic crosses availability zones in a public cloud, the bill grows linearly with target count and scrape rate. With zone-aware allocation, the same workload stays inside each AZ and the cross-AZ bytes go to zero (modulo failover and maxSkew spillover).

Cloud provider cross-AZ pricing

Provider	Cross-AZ (intra-region)	Same-AZ
AWS	$0.01/GB each direction (so $0.02/GB round-trip)	Free
GCP	$0.01/GB egress + $0.01/GB ingress between zones	Free within zone
Azure	$0.01/GB ingress + $0.01/GB egress between AZs	Free within AZ
OCI	$0.0085/GB egress within region	Free intra-AD
Alibaba Cloud	$0.005/GB inter-zone	Free intra-zone

All three majors charge both directions for cross-AZ, so the effective rate for a scrape (Prometheus → target → response back) is roughly $0.02/GB. The bytes show up under labels like "inter-AZ transfer", "regional data transfer", or "zone egress" in cloud bills.

Realistic Prometheus scrape response sizes (Kubernetes workloads)

The previous draft of this comment used 10 KB as the average response size — that's a fair number for plain application metrics, but it dramatically underestimates real Kubernetes observability stacks. In production K8s clusters the scrape volume is dominated by infrastructure exporters that emit hundreds of metrics per scrape:

Exporter / endpoint	Typical scrape size	Notes
node_exporter	50–150 KB	Default collectors enabled; grows with CPU/disk/network device count
cAdvisor (kubelet /metrics/cadvisor)	100–500 KB	Per-container metrics, scales with pod density per node
kubelet /metrics	50–200 KB	Kubelet itself, runtime metrics
kube-state-metrics	50–500 KB (often higher)	Scales with object count: pods, services, configmaps, etc.
kube-proxy	10–30 KB	Lightweight
prometheus federation	100 KB – several MB	When used for cross-cluster aggregation
Database exporters (postgres, mysql, redis)	50–200 KB	Per-DB metric cardinality
Application metrics (typical Go/Java service)	5–50 KB	Lower-end of the range
Istio/Envoy sidecars	100–500 KB	Connection-level metrics balloon size

A realistic mid-range for a Kubernetes observability deployment is 50–100 KB per scrape when you average across nodes (carrying node_exporter + cAdvisor + kubelet), pods (carrying app metrics + sidecars), and control-plane components.

Bandwidth math with realistic K8s scrape sizes

Assume:

• Average scrape response: 50 KB (mid-range for K8s — node_exporter + cAdvisor + app metrics blended)
• Scrape interval: 15 s (4 scrapes per minute per target)
• Without zone affinity in a 3-AZ cluster, ~67% of scrapes cross zones

Targets	Bytes/month	Without zone affinity (cross-AZ)	Monthly AWS cost @ $0.02/GB	Annual AWS cost
1,000	~8.6 TB	~5.8 TB	~$120	~$1,440
10,000	~86 TB	~58 TB	~$1,200	~$14,400
50,000	~432 TB	~289 TB	~$5,900	~$71,000
100,000	~864 TB	~579 TB	~$11,900	~$143,000
500,000	~4.3 PB	~2.9 PB	~$59,300	~$712,000

High-cardinality scenario (heavier infra)

For clusters dominated by cAdvisor, kube-state-metrics with many objects, or Istio sidecars, an average of 200 KB per scrape is realistic (often higher):

Targets	Bytes/month	Cross-AZ (67%)	Monthly AWS cost	Annual
1,000	~35 TB	~23 TB	~$475	~$5,700
10,000	~346 TB	~232 TB	~$4,750	~$57,000
50,000	~1.7 PB	~1.16 PB	~$23,750	~$285,000
100,000	~3.5 PB	~2.3 PB	~$47,500	~$570,000

Light scenario (mostly app metrics, 10 KB)

For setups dominated by lightweight app exporters (no node_exporter, minimal infra metrics):

Targets	Bytes/month	Cross-AZ (67%)	Monthly AWS cost	Annual
1,000	~1.75 TB	~1.17 TB	~$24	~$288
10,000	~17.5 TB	~11.7 TB	~$240	~$2,880
100,000	~175 TB	~117 TB	~$2,400	~$28,800

Scaling factors that push the bill higher

• Higher scrape rate: 5–10s intervals (common for hot infra) multiply volume 1.5–3×.
• More AZs: in 4-AZ deployments ~75% of scrapes cross zones; in 5-AZ deployments ~80%.
• Federation / remote_write: doubles the math if scraped metrics get forwarded cross-AZ.
• High-cardinality labels (per-container, per-request): can push individual scrape responses to 500 KB – 1 MB.

Secondary benefits beyond raw cost

• Lower scrape latency: same-AZ RTT is typically 0.3–1 ms vs 1–3 ms cross-AZ on AWS — meaningful when scraping thousands of fast-changing endpoints.
• Higher reliability under AZ partition / brownout: a target in zone-a can keep being scraped by a collector in zone-a even when zone-b has connectivity issues to zone-a.
• Lower control-plane stress: less inter-AZ traffic means less load on managed routers and lower risk of throttling on burst.
• Cleaner cost attribution: when chargeback dashboards split by AZ, zone-local scraping makes observability costs traceable to the workload's own AZ instead of polluting cross-AZ totals.

Where this matters most

The economics flip from "nice to have" to "obvious" once you hit the 10k targets / TB-per-day range:

• Tier-3 / SaaS providers: large fleets of customer-owned workloads scraped from a shared observability plane. The bill scales with customer count.
• CI / build farms: many short-lived nodes scraped at high frequency for per-job runtime metrics.
• Edge / multi-region deployments: zone-aware is a stepping stone toward region-aware allocation; the same machinery handles both.
• High-cardinality workloads: when each scrape returns hundreds of KB (Prometheus blackbox, OpenMetrics endpoints with detailed labels, Istio/Envoy sidecars), zone-locality compounds.

Backward-compatibility guarantee

This feature is strictly additive. When `topology.zone_aware` is unset or `false`:

• Allocation behavior, hash-ring construction, and target distribution are byte-for-byte identical to today (verified by a dedicated backward-compat test suite).
• No new metrics are emitted (per-zone gauges are only registered when zone-aware is on).
• The `/zones` API endpoint exists for clients that want to probe it but reports `enabled: false` with an empty payload — clients can detect zone-aware setups without needing to inspect config.
• Existing strategies (`consistent-hashing`, `least-weighted`, `per-node`) retain their current semantics. Zone awareness is a layer that activates only when enabled.

Users opt in when they want the savings; everyone else sees zero behavioral change.

[szibis]

Previously only achievable with per-node strategy

The cost-saving behavior — keeping scrape traffic in the same AZ — has historically been reachable in the Target Allocator only by switching to the per-node strategy. That works, but it's a heavy hammer with serious constraints:

Aspect	per-node (existing)	zone-aware consistent-hashing / least-weighted (this proposal)
Deployment mode required	DaemonSet (one collector per node)	Any: Deployment, StatefulSet, DaemonSet
Resource overhead	One collector pod per node — at hundreds of nodes this is a lot of pods, CPU, and memory just for collectors	Scales independently of node count; can run 3–10 collector pods and serve a 1,000-node cluster
Load balancing across collectors	None — each collector only sees targets on its own node, no balancing across nodes within a zone	Built-in: consistent-hashing or least-weighted distributes load across all collectors in the same zone
Failure recovery	If the node-local collector dies, its targets aren't scraped at all until the pod restarts on the same node	Targets in the affected zone are picked up by other collectors in the same zone (zone-local failover)
Cross-AZ failover	Not supported — the strategy is strictly node-pinned	Failover to other zones when an entire zone has no collectors
Operator simplicity	Requires running every collector as a privileged DaemonSet pod	Standard Deployment/STS with normal HA semantics
Cost on small clusters	Cheap pod-wise but loses scale efficiency	Fewer collectors → less waste
Cost on large clusters	Hundreds of collector pods even for a fleet of small scrape targets	Linear in zone count, not node count

When per-node was the only option

Operators who needed to minimize cross-AZ scrape traffic and could afford DaemonSet overhead would use per-node. The downside list above was the price: heavy footprint, no load balancing, no resilience to single-pod failure.

What zone-aware allocation changes

Zone-aware consistent-hashing and least-weighted give the same same-AZ scraping savings without forcing DaemonSet mode. You can run 3 collector pods (one suggested per AZ for HA) and still get:

• ~100% same-AZ traffic under normal operation
• Load balancing across collectors within a zone (maxSkew controls intra-zone tightness)
• Cross-zone failover when a zone has no collectors
• Optional cross-zone spillover under maxSkew to avoid hot-spotting a single collector

For the cost-conscious large-scale operator, this collapses to: the same egress savings, at 1–10% of the collector footprint, with better resilience.