RFC: Standardize Collector Error Handling

[stephan-rayner]

Background

Collector error handling is inconsistent across the codebase at both initialization and scrape time. The design intent is:

• A failing collector should not block other collectors from the same provider
• A provider can fail if none of its collectors initialize successfully
• Failures must be observable without relying solely on log monitoring

In practice, 13 collectors implement at least four distinct behaviours at each phase, and there is no agreed standard. This issue exists to reach consensus on what that standard should be across both phases.

Note: This issue is about operational metrics - metrics that describe the health of the exporter itself (e.g. cloudcost_exporter_collector_scrape_errors_total) - not the cost rate metrics that collectors emit (e.g. cloudcost_aws_ec2_instance_cpu_usd_per_core_hour).

---

Current State

Initialization

Collector	New() returns error?	What happens on failure
AWS EC2	YES	Returns error; provider logs & skips
AWS S3	YES	Partial - bad regions logged as warning, no error returned
GCP GCS	YES	Returns error; provider logs & skips
GCP GKE	YES	Returns error; provider logs & skips
GCP CLB	YES	Returns error; provider logs & skips
GCP Cloud SQL	YES	Returns error; provider logs & skips
Azure AKS	YES	Returns error; provider fails entirely
AWS RDS	NO	Never fails on init - defers all errors to scrape time
AWS NAT Gateway	NO	Never fails on init - background refresh, errors logged
AWS ELB	NO	Never fails on init - defers all errors to scrape time
AWS MSK	NO	Never fails on init - background refresh, errors logged
AWS VPC	NO	Swallows init error, logs it, always returns a collector
GCP VPC	NO	Swallows init error, logs it, always returns a collector

Four Patterns in Practice:

1. Returns error -> provider logs & skips the collector (6 collectors)
2. Returns error -> provider fails entirely (1 collector: Azure AKS)
3. Never fails on init -> defers to scrape or background refresh (4 collectors)
4. Swallows init error -> logs it but always returns a healthy-looking collector (2 collectors)

Scrape Time

Quick Clarification of "Partial metrics on failure" Column:
The column called "Partial metrics on failure" refers to whether a collector that hits an error mid-scrape still emits some cost rate metrics (e.g. cloudcost_gcp_gke_instance_cpu_usd_per_core_hour) before giving up.

For example, if the GKE collector fails to list instances in us-east1 but succeeds in us-west1 and europe-west1, a YES means you still get metrics for those two regions. A NO means you get nothing at all for that scrape.

Collector	Collect() returns error?	Partial metrics on failure?	Stale/cached values served?
AWS EC2	Only if client missing; region errors logged & skipped	YES	YES (pricing maps via background ticker)
AWS S3	YES	NO	YES (last successful billing data retained)
AWS RDS	YES (pricing validation only); region errors logged & skipped	YES	YES (in-memory pricing cache)
AWS NAT Gateway	NO - always returns nil	NO	YES (snapshot from background ticker)
AWS ELB	YES	YES	YES (conditional on scrape interval)
AWS VPC	YES (context cancellation only); pricing misses logged & skipped	YES	YES (background ticker, 24h)
AWS MSK	YES (context cancellation only); unpriceable clusters skipped	YES	YES (snapshot from background ticker)
GCP GCS	YES (service lookup); export errors logged & skipped	YES	YES (interval-throttled refresh)
GCP GKE	YES (zone listing); per-zone errors logged & skipped	YES	YES (pricing map via background ticker)
GCP CLB	YES (forwarding rule fetch); per-region errors logged & skipped	YES	YES (pricing map via background ticker, 24h)
GCP VPC	YES (context cancellation only); pricing misses logged at debug	YES	YES (background ticker, 24h)
GCP Cloud SQL	YES; project-level error blocks entire collection	YES (per-instance)	YES (SKU cache via background ticker, 24h)
Azure AKS	NO - always returns nil; unpriceable VMs/disks logged & skipped	YES	YES (3 separate background tickers)

Four Patterns in Practice:

1. Returns error on hard failures only; silently skips per-region or per-resource errors (most collectors)
2. Never returns an error; all failures are silent (AWS NAT Gateway, Azure AKS)
3. Returns error that blocks the entire collection for that provider service (GCP Cloud SQL on project list failure)
4. Returns error only on context cancellation, not on data failures (AWS VPC, AWS MSK, GCP VPC)

Unlike initialization, scrape-time failures are recurring. A collector that fails on init fails once. A collector that fails on scrape fails on every scrape interval until the underlying issue is resolved.

---

The Core Question

At both phases, the same two tensions apply:

Resilience: a failing collector should not prevent healthy collectors from running.

Observability: a failing collector must be detectable - ideally via a metric so it can be alerted on, not just a log line.

The current debate is whether to surface failures via an error return (allowing the provider to propagate or track it) or via a logged metric increment (keeping function signatures clean while still making failures queryable). This question applies equally to init and scrape, and the answer should be consistent across both.

---

Options

Option A - Return Error, Provider Logs and Skips

At init, New() returns an error and the provider logs it and skips the collector. At scrape, Collect() returns an error and the provider logs it. Other collectors are unaffected at both phases. Failures are only observable via logs.

Example: If the EC2 collector fails to initialize, the AWS provider logs the error and continues without EC2. S3, RDS, and other collectors still run. The failure is only visible in the exporter logs - there is no metric to alert on.

• Pro:
  • Resilient.
  • Clean Go error handling.
  • Consistent with the majority of existing collectors.
• Con:
  • Silent in Mimir.
  • No metric to alert on.
  • Log-based alerting is fragile.

Option B - Return Error, Log, Skip, and Increment a Metric

Same as A, but the provider also increments an error counter (e.g. cloudcost_exporter_collector_init_errors_total, cloudcost_exporter_collector_scrape_errors_total) labelled by collector name.

Example: If the EC2 collector fails to initialize, the AWS provider logs the error, skips EC2, and increments cloudcost_exporter_collector_init_errors_total{collector="ec2"}. S3, RDS, and other collectors still run. An alert can fire on that counter without any log monitoring.

• Pro:
  • Resilient and alertable.
  • Errors are queryable via Mimir.
  • Consistent observability across both phases.
• Con:
  • Requires new or extended metrics.
  • The gatherer may already track some of this; needs investigation before adding duplication.

Option C - Never Fail, Defer All Errors

At init, New() always succeeds. At scrape, Collect() re-emits stale cached values or serves background-refreshed data rather than returning an error.

Example: If the EC2 collector's pricing API is down at init, it starts anyway with an empty pricing map. At scrape time it serves whatever cached data it has, or emits no metrics for regions it cannot price. There is no error anywhere in the system - the collector appears healthy.

• Pro:
  • Simplest signatures.
  • No partial-initialization states.
  • No gaps in metric output.
• Con:
  • A broken collector silently appears healthy.
  • Stale values are misleading.
  • Alerting on cost changes becomes unreliable.
  • Already the pattern for NAT Gateway and MSK, but inconsistent with the majority.

Option D - Fail Fast, Fail the Provider or Scrape

Any collector failure fails the entire provider at init, or fails the entire scrape at scrape time.

Example: If the EC2 collector fails to initialize, the entire AWS provider fails to start - S3, RDS, and all other AWS collectors stop running too. At scrape time, a single EC2 error causes Mimir to mark the entire scrape as failed and discard all metrics from that provider.

• Pro:
  • No silent failures.
  • Obvious signal that something is wrong.
• Con:
  • One broken collector takes down all metrics for that provider.
  • Contradicts the stated design intent.

---

Open Questions

1. What is the right observability mechanism? Logs, Mimir metric, or both? If a metric, is there an existing one that fits or does a new one need to be defined?

2. Should the standard be the same for init and scrape? The consequence of a silent failure differs: an init failure is a one-time event, a scrape failure recurs indefinitely.

3. What should happen when a collector that defers init work to the background (NAT Gateway, MSK) fails its first refresh? Is "never fail on init" acceptable for these, or should they also return an error?

4. Should the provider fail if zero collectors initialize successfully? This is the one case where a provider-level failure may still be appropriate regardless of the chosen option.

5. How should partial scrapes be handled? If a collector emits 5 metrics then fails, should those 5 be kept or discarded?

6. What is the acceptable staleness window? If a collector fails repeatedly, at what point should an alert fire? This may inform whether a counter or a gauge (time since last successful scrape) is the right metric shape.

---

References

• fix(aws): skip failing RDS and VPC collectors instead of aborting provider #863
• AWS: Handle collector errors gracefully #716
• EC2: Add tickers to populate pricing maps #664 (comment)

Proposed Direction

I propose we persue Option B as it keeps the exporter resilient to collector failures, gets us the most observability. This is a little more work than Option A as no collector implements it now, but I think that is the best choice.

As for deployment, rolling this out in Dev and leaving it that way for a little while makes sense to me to get a feel for the error volume. Since this came out of a meeting which also discussed alert fatigue it would be unfortunate, if we induce alert fatigue. Also getting the tuning and thresholds right with this change will be important and could take some revision. Doing it in dev for a little while will allow us time to get that tuning right without making a bunch of noise for on-callers.

[simaarfania]

I'm not 100% confident in Option C. The AKS collector always returns nil from Collect() even when it can't price hundreds of VMs. So from the gatherer's perspective the scrape succeeded, the error counter never moves, and Mimir shows a healthy collector that's silently dropping metrics. The only signal was a warn log, which isn't something you can reliably alert on.

Landing on Option B feels right to me. I'd push back slightly on the framing that it requires a lot of new work. pkg/collectormetrics already increments cloudcost_exporter_collector_error when Collect() returns an error. The gatherer infrastructure is mostly there. The gap is just the collectors that swallow failures and return nil, meaning the gatherer never gets the signal to increment anything.

The one thing I'd add to the open questions is the startup race for collectors that defer init to background goroutines. For AKS specifically, the price store takes quite a bit to populate after pod start, but nothing stops Collect() from being called in that window with an empty map. "Never fail on init" is fine for these collectors, but the first Collect() should return an error if the background init hasn't finished yet rather than silently serving empty data.

For Q6: I think a gauge for time since last successful scrape would be more useful than a counter for actual alerting, since a counter doesn't tell you whether something is broken right now or just had a rough start three weeks ago.

[stephan-rayner]

@leonorfmartins followed up on this with me via her phone on slack. She is away from her computer for the next few days so I said I would migrate her comments here:

---

So, regarding option B, if we can use collector metrics, which are the base for our current SLOs, it would be great. There's a tiny wrapper called around every collect method which observes latencies and errors and emits those metrics. Specifically for errors, those metrics are

• cloudcost_exporter_collector_total
• cloudcost_exporter_collector_errors

(I'm on my phone, so I can't check precisely the names but they are something like that)

Total will collect all attempts (successful or not) and error will collect only errors. They are both counters

Those labels have the collector name as a label and the region. There's an issue to add the provider additionally. So we could know if a whole provider is having issues or not.

---

Now, some thoughts from the top of my head:
I think that any change that we do we should validate in dev if this turns the SLO too noisy or not. We prob don't want to get alerted if one collector fails for one VM once for some reason.

Some examples for us to explore:

• collector init fails (for example it can't reach the pricing API) —> maybe we want to error there?
• Collector init works but on price refresh or resources refresh, things fail —> maybe also error?
• Emitting one metric for one resource fails —> maybe not needed to error?

[stephan-rayner]

I thought it might be helpful to get a view of which collectors align with which option.

Based on the current state of the codebase, here is how each collector maps to the options defined above. Note that no collector implements Option B and none increment an error metric.

Collector	Init	Scrape	Notes
AWS EC2	A	A	Scrape only errors if client is missing; region errors are silently skipped
AWS S3	A*	A	Init partially swallows the error - bad regions logged as warning, no error returned
AWS RDS	C	A	Defers all init errors to scrape; partial metrics at scrape
AWS NAT Gateway	C	C	Never errors at either phase
AWS ELB	C	A	Defers init errors; returns error at scrape
AWS VPC	C*	A	Swallows init error but logs it; scrape only errors on context cancellation
AWS MSK	C	A	Scrape only errors on context cancellation, not data failures
GCP GCS	A	A
GCP GKE	A	A	Per-zone scrape errors are silently skipped
GCP CLB	A	A	Per-region scrape errors are silently skipped
GCP VPC	C*	A	Swallows init error but logs it; scrape only errors on context cancellation
GCP Cloud SQL	A	A/D	Project-level scrape error blocks the entire collection
Azure AKS	D	C	Only collector that fails the provider on init; never errors at scrape

* = hybrid of A and C: error is swallowed and never returned, but is logged

Summary:

• Option A: Most collectors (init and scrape)
• Option B: Nobody
• Option C: Collectors that defer init entirely; NAT Gateway and AKS at scrape
• Option D: Azure AKS at init; GCP Cloud SQL partially at scrape

[rooneyshuman]

+1 on Option B. D is too aggressive. One broken collector taking down a whole provider contradicts the whole point of this exporter. B gives us resilience & observability.

1. Right observability mechanism?
   Both. Metric so we can alert on it, logs so we have context when the alert fires.

2. Same standard for init and scrape?
   Same principle, different shape. Init failure is a state --> a gauge makes sense there since a counter that fires once at startup isn't useful for alerting. Scrape failure is a rate --> the existing error counter already handles that.

3. Background-init collectors (NAT Gateway, MSK)?
   Never failing New() is fine since the init is genuinely async. But the first Collect() should return an error if the store isn't ready yet, not silently serve empty data. The AKS select { case <-Done(): ... default: ... } pattern is close — the default branch just needs to return error instead of return nil.

4. Fail the provider if zero collectors initialize?
   Yes, this is the one place where a provider-level failure makes sense. Zero collectors means zero metrics for that provider with no signal. Currently AWS/GCP don't enforce this, and Azure overcorrects by failing on any single collector failure. The right middle ground: skip failed collectors, fail only if none succeed.

5. Keep or discard partial scrapes?
   Keep them. You can't recall metrics already written to the channel without buffering everything, and partial data beats no data. The error return covers the failure signal.

6. Staleness window and metric shape?
   Gauge over counter, specifically last_scrape_time as a Unix timestamp (we already have cloudcost_exporter_collector_last_scrape_time). For thresholds: warn at 3 × scrape_interval, page at 10 ×. Validate the multipliers in dev before rolling to prod.

[nikimanoledaki]

+1 on B. Great analysis and teamwork! 👏

Couple of additional notes that haven't been mentioned yet:

• Operational metrics for failing collectors are a must. The convention is to alert on metrics rather than expect logs to be there. In the past, at Grafana Labs, we used to have some alerts configured around logs but we moved away from this in favor of metrics (maybe 2 years ago?). IIRC part of this is that logs have a shorter retention period compared to metrics. If an incident happens, we need those operational metrics to be there for a long enough period for us to do investigations, look back in time etc to find patterns and identify root causes.
• The second thing that I wanted to highlight is testing. It's a lot of extra work to write unit tests for logs. Requires reading from a local buffer - yuck. B gives us an opportunity to get unit testing right across the collectors since we'll be able to check the error and the metric. I would really lean on unit tests (hybrid TDD?) for this so that we ensure collectors fail in a cohesive way and that this behaviour is also tested in a cohesive way 👍

[stephan-rayner]

I didn't include a conversation that @leonorfmartins and I had following here earlier post. I wanted to include this in the discussion for transparency and for future reference.

---

Leonor:

Adjusting the SLO is also totally fine, if we decide so btw! How we error and how we measure are really interconnected and the result should work for us, not against us

Stephan:

Agreed

Stephan:

Did you mean cloudcost_exporter_collector_last_scrape_error when you said cloudcost_exporter_collector_errors`

Leonor:

No, that one is deprecated, I think

this one:

cloudcost-exporter/pkg/collectormetrics/collectormetrics.go

Line 24 in ec0747e

var errorCounterVec = prometheus.NewCounterVec(

Stephan:

Oh, ok that is good to know, thank you!

---