Skip to content

fix(config): Apply common object storage config to the ruler storage#22366

Open
sakiphan wants to merge 1 commit into
grafana:mainfrom
sakiphan:fix/ruler-storage-thanos-tls-22319
Open

fix(config): Apply common object storage config to the ruler storage#22366
sakiphan wants to merge 1 commit into
grafana:mainfrom
sakiphan:fix/ruler-storage-thanos-tls-22319

Conversation

@sakiphan

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

When use_thanos_objstore is enabled, the ruler reads its bucket configuration from the separate ruler_storage section. Unlike every legacy storage type (s3, gcs, azure, swift, ...), the common.storage.object_store config was never propagated there, so settings such as TLS CA, credentials and endpoints configured in the common section were silently ignored by the ruler. In a simple scalable deployment this surfaces as tls: failed to verify certificate: x509: certificate signed by unknown authority errors on the backend target (which runs the ruler), while the read and write components work fine.

This PR applies the common object storage config to ruler_storage as well, mirroring how the legacy storage types have always been applied to the ruler:

  • The ruler storage backend is inferred from which single object store backend is configured in the common config; when none or several backends are configured, the ruler storage is left untouched (preserving today's behavior, including "ruler does not start when its storage is unconfigured").
  • Nothing changes when use_thanos_objstore is disabled, so legacy setups are unaffected.
  • An explicitly configured ruler_storage section still takes precedence, since the config file is re-applied on top of the dynamic config.

This also resolves two long-standing TODO: common config should be set on ruler bucket config comments in the config wrapper tests.

Which issue(s) this PR fixes:
Fixes #22319

Special notes for your reviewer:

The new test "when common object_store config is provided, the http config such as TLS settings should be applied to the ruler storage" reproduces the exact scenario from the issue (custom CA via s3.http.tls_ca_path). Since the ruler may now start in setups where common.storage.object_store is set but ruler_storage was intentionally left unconfigured, this behavior change is documented in the upgrade guide.

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@sakiphan sakiphan requested a review from a team as a code owner June 10, 2026 19:11
@vladst3f

Copy link
Copy Markdown

The problem affects more components than just the ruler. See #20741

@sakiphan

Copy link
Copy Markdown
Contributor Author

The problem affects more components than just the ruler. See #20741

Thanks for the pointer. I went through the call sites on main to double check this. The compactor (incl. the delete request store), index gateway and bloom components all build their clients from storage_config.object_store, which already inherits common.storage.object_store — and the thanos S3 client does pass http.tls_ca_path / http.insecure_skip_verify through (pkg/storage/bucket/s3/bucket_client.go). That was already the case in v3.6.4 as well.

The ruler was the odd one out, since it reads from the separate ruler_storage section which never got the common config applied. That's what this PR fixes.

So I couldn't find a code path on current main where the compactor would ignore the TLS settings if #20741 still reproduces on a recent version it's probably something else, happy to dig into that separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

backend ignores s3 http config when use_thanos_objstore is true

2 participants