Skip to content

use authorization.credentials instead of bearerToekn#45

Merged
anishasthana merged 3 commits intoopendatahub-io:mainfrom
Gregory-Pereira:ocp-use-authorization-credentials-for-metrics
Mar 27, 2026
Merged

use authorization.credentials instead of bearerToekn#45
anishasthana merged 3 commits intoopendatahub-io:mainfrom
Gregory-Pereira:ocp-use-authorization-credentials-for-metrics

Conversation

@Gregory-Pereira
Copy link
Copy Markdown
Member

@Gregory-Pereira Gregory-Pereira commented Mar 27, 2026

OCP has deprecated use of bearerToken auth, see: https://docs.redhat.com/en/documentation/openshift_container_platform/4.17/html/monitoring_apis/servicemonitor-monitoring-coreos-com-v1#spec-endpoints-2
produces: https://access.redhat.com/solutions/7007873

Migrates from bearerToken to a secret backing the service account + authorization.credentials

What else :

cc @zdtsw @vivekk16 @pierDipi

Summary by CodeRabbit

  • Chores
    • Updated OpenShift deployment configuration to use token-based authentication for metrics collection, improving how Prometheus scrapes controller metrics.
    • Simplified default configuration by removing legacy namespace propagation logic that previously copied deployment namespace into the metrics monitor configuration.

Signed-off-by: greg pereira <grpereir@redhat.com>
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Mar 27, 2026

Warning

Rate limit exceeded

@Gregory-Pereira has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 5 minutes and 27 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 5 minutes and 27 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: d3d70633-e323-4ec9-a1c1-5b28cb052095

📥 Commits

Reviewing files that changed from the base of the PR and between 8647e6d and 4889964.

📒 Files selected for processing (2)
  • config/openshift/kustomization.yaml
  • config/openshift/prometheus-metrics-auth-binding.yaml
📝 Walkthrough

Walkthrough

Removed a replacements entry from the default Kustomization that propagated the controller-manager Deployment namespace into the ServiceMonitor. The OpenShift Kustomization adds a new Secret resource (workload-variant-autoscaler-metrics-reader-token) of type kubernetes.io/service-account-token and a ServiceMonitor patch (monitor-auth-patch.yaml) that switches the ServiceMonitor to use authorization.credentials (secret key token), scrapes the HTTPS /metrics endpoint every 10s, and sets tlsConfig.insecureSkipVerify: true.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Security & Design Issues

  • TLS verification disabled: tlsConfig.insecureSkipVerify: true disables certificate validation (CWE-295). Action: set to false and ensure Prometheus has the proper CA bundle, or document why validation cannot be enforced.

  • Secret type & placement risk: creating a kubernetes.io/service-account-token Secret exposes a bearer token (sensitive credential, CWE-200/CWE-276). Action: ensure the Secret is created in the same namespace the ServiceMonitor expects, restrict RBAC so only the monitoring system reads it, and consider using projected ServiceAccountToken or IRSA/Workload Identity alternatives if available.

  • ServiceAccount and RBAC assumptions: the Secret annotation binds to workload-variant-autoscaler-controller-manager SA; confirm that SA exists and has minimal privileges required for metrics access. Action: explicitly declare required Role/RoleBinding or document RBAC requirements.

  • Namespace binding removed: removal of the replacements propagation may cause namespace mismatch between Deployment/ServiceMonitor and the new Secret, breaking metric collection. Action: verify ServiceMonitor namespaceSelector and Secret namespace alignment; reintroduce a controlled namespace mapping if needed.

Address the above items before deploying to production.

🚥 Pre-merge checks | ✅ 2
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'use authorization.credentials instead of bearerToekn' directly reflects the main change across the PR: migrating from bearerToken to authorization.credentials for ServiceMonitor authentication.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@config/openshift/metrics-reader-token.yaml`:
- Around line 1-12: Create and add a ClusterRoleBinding that binds the existing
ClusterRole "workload-variant-autoscaler-metrics-reader" to the ServiceAccount
"workload-variant-autoscaler-controller-manager" (subject namespace:
"workload-variant-autoscaler-system"), using the ClusterRoleBinding name
"workload-variant-autoscaler-metrics-reader-rolebinding"; then include that new
manifest in the RBAC kustomization so the binding is applied alongside the
existing "metrics_reader" ClusterRole and ServiceAccount to allow Prometheus
access to /metrics.

In `@config/openshift/monitor-auth-patch.yaml`:
- Around line 16-17: The manifest sets tlsConfig.insecureSkipVerify: true which
disables TLS certificate validation; replace this by configuring a trusted CA
instead—use tlsConfig.ca (base64 CA data) or tlsConfig.caFile pointing to the
cluster CA so metrics scraping validates the controller-manager certificate;
update the tlsConfig block (the tlsConfig key and its insecureSkipVerify field)
to remove insecureSkipVerify and add the appropriate tlsConfig.ca or
tlsConfig.caFile entry referencing the in-cluster CA.
- Around line 18-21: Remove the insecureSkipVerify: true setting under the TLS
configuration and instead supply a trusted CA bundle or OpenShift service cert
for TLS verification; update the resource that configures prometheus scrape
authorization (the authorization.credentials block referencing name:
workload-variant-autoscaler-metrics-reader-token and key: token) to include a
proper tlsConfig.caFile (or equivalent OpenShift service-to-service cert
reference) so certificate validation is enforced rather than bypassed.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 1f59993f-e13d-4b1c-9c0b-6fbe389cd8f7

📥 Commits

Reviewing files that changed from the base of the PR and between 06862a9 and 79e2242.

📒 Files selected for processing (4)
  • config/default/kustomization.yaml
  • config/openshift/kustomization.yaml
  • config/openshift/metrics-reader-token.yaml
  • config/openshift/monitor-auth-patch.yaml
💤 Files with no reviewable changes (1)
  • config/default/kustomization.yaml

Comment on lines +1 to +12
# Long-lived SA token for Prometheus to authenticate to the WVA metrics endpoint.
# Required on OpenShift because user-workload-monitoring Prometheus rejects
# bearerTokenFile for security. The ServiceMonitor is patched (via
# monitor-auth-patch.yaml) to reference this Secret instead.
apiVersion: v1
kind: Secret
metadata:
name: workload-variant-autoscaler-metrics-reader-token
namespace: system
annotations:
kubernetes.io/service-account.name: workload-variant-autoscaler-controller-manager
type: kubernetes.io/service-account-token
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify whether a ClusterRoleBinding exists that binds metrics-reader to the controller-manager SA
echo "=== Searching for metrics-reader ClusterRoleBinding ==="
rg -n "metrics-reader" --type yaml -C5

echo ""
echo "=== Checking all ClusterRoleBindings for controller-manager SA ==="
rg -n "workload-variant-autoscaler-controller-manager" --type yaml -C5

Repository: opendatahub-io/workload-variant-autoscaler

Length of output: 18253


🏁 Script executed:

#!/bin/bash
# Check what resources are included in the openshift kustomization
echo "=== config/openshift/kustomization.yaml ==="
cat config/openshift/kustomization.yaml

echo ""
echo "=== Check if metrics_reader_role_binding.yaml exists in config/rbac/ ==="
ls -la config/rbac/metrics_reader_role_binding.yaml 2>&1 || echo "NOT FOUND"

echo ""
echo "=== Check what's in config/rbac/kustomization.yaml ==="
grep -n "metrics_reader" config/rbac/kustomization.yaml || echo "No metrics_reader reference"

Repository: opendatahub-io/workload-variant-autoscaler

Length of output: 1171


🏁 Script executed:

#!/bin/bash
# Check config/default/ to see if metrics_reader_role_binding is included there
echo "=== config/default/kustomization.yaml ==="
cat config/default/kustomization.yaml 2>&1 | head -50

echo ""
echo "=== Search for any metrics_reader_role_binding in entire config/ ==="
find config -name "*metrics_reader*binding*" -type f 2>/dev/null || echo "None found"

Repository: opendatahub-io/workload-variant-autoscaler

Length of output: 2191


Add ClusterRoleBinding for metrics-reader role to authorize controller-manager ServiceAccount.

The metrics-reader ClusterRole exists (defined in config/rbac/metrics_reader_role.yaml and included in the kustomize build), but there is no corresponding ClusterRoleBinding that grants this role to the workload-variant-autoscaler-controller-manager ServiceAccount. The token generated in this Secret will authenticate successfully but will receive HTTP 403 when Prometheus attempts to scrape /metrics due to missing authorization.

Add the following ClusterRoleBinding to config/rbac/:

metrics_reader_role_binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: workload-variant-autoscaler-metrics-reader-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: workload-variant-autoscaler-metrics-reader
subjects:
- kind: ServiceAccount
  name: workload-variant-autoscaler-controller-manager
  namespace: workload-variant-autoscaler-system

Then reference it in config/rbac/kustomization.yaml.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@config/openshift/metrics-reader-token.yaml` around lines 1 - 12, Create and
add a ClusterRoleBinding that binds the existing ClusterRole
"workload-variant-autoscaler-metrics-reader" to the ServiceAccount
"workload-variant-autoscaler-controller-manager" (subject namespace:
"workload-variant-autoscaler-system"), using the ClusterRoleBinding name
"workload-variant-autoscaler-metrics-reader-rolebinding"; then include that new
manifest in the RBAC kustomization so the binding is applied alongside the
existing "metrics_reader" ClusterRole and ServiceAccount to allow Prometheus
access to /metrics.

Comment on lines +16 to +17
tlsConfig:
insecureSkipVerify: true
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

insecureSkipVerify: true disables TLS certificate validation (CWE-295).

Not a regression (base manifest has same setting), but this allows MITM attacks on metrics scraping traffic. If the controller-manager serves metrics over a cert signed by an in-cluster CA, consider configuring tlsConfig.ca or tlsConfig.caFile instead.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@config/openshift/monitor-auth-patch.yaml` around lines 16 - 17, The manifest
sets tlsConfig.insecureSkipVerify: true which disables TLS certificate
validation; replace this by configuring a trusted CA instead—use tlsConfig.ca
(base64 CA data) or tlsConfig.caFile pointing to the cluster CA so metrics
scraping validates the controller-manager certificate; update the tlsConfig
block (the tlsConfig key and its insecureSkipVerify field) to remove
insecureSkipVerify and add the appropriate tlsConfig.ca or tlsConfig.caFile
entry referencing the in-cluster CA.

Comment on lines +18 to +21
authorization:
credentials:
name: workload-variant-autoscaler-metrics-reader-token
key: token
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🌐 Web query:

Prometheus Operator ServiceMonitor authorization credentials SecretKeySelector spec

💡 Result:

The Prometheus Operator ServiceMonitor CRD includes an authorization field under spec.endpoints[] for configuring Authorization header credentials. This field uses a credentials object that functions as a SecretKeySelector. Key specification details: - Path: spec.endpoints[].authorization.credentials - Fields: - key (string, required): The key within the Secret containing the credentials. - name (string): Name of the Secret (in the same namespace as the ServiceMonitor). - optional (boolean): Whether the Secret or key must exist. The type field (string) under authorization sets the authentication type (defaults to "Bearer"; "Basic" causes an error). This cannot be used simultaneously with basicAuth or oauth2. The Secret must be accessible by the Prometheus Operator in the ServiceMonitor's namespace. Example YAML structure: spec: endpoints: - authorization: credentials: name: my-secret key: token type: Bearer port: metrics For basic authentication, use basicAuth instead, which has separate username and password SecretKeySelectors: basicAuth: username: name: auth-secret key: username password: name: auth-secret key: password Sources confirm this schema across Prometheus Operator v0.50+, OpenShift docs, and official CRD definitions.

Citations:


🏁 Script executed:

cat -n config/openshift/monitor-auth-patch.yaml

Repository: opendatahub-io/workload-variant-autoscaler

Length of output: 890


🏁 Script executed:

# Search for the Secret resource
rg 'workload-variant-autoscaler-metrics-reader-token' --type yaml

Repository: opendatahub-io/workload-variant-autoscaler

Length of output: 289


🏁 Script executed:

# Check for other insecureSkipVerify instances in ServiceMonitor configs
rg 'insecureSkipVerify' --type yaml -C 2

Repository: opendatahub-io/workload-variant-autoscaler

Length of output: 3740


Remove insecureSkipVerify: true—enables TLS certificate verification bypass (CWE-295).

The authorization.credentials structure is correct per Prometheus Operator spec. However, line 17 disables all TLS certificate validation, allowing man-in-the-middle attacks against the metrics endpoint. Either provide a valid CA certificate via tlsConfig.caFile or use OpenShift's service-to-service certificate validation. Disabling TLS verification is not acceptable for production.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@config/openshift/monitor-auth-patch.yaml` around lines 18 - 21, Remove the
insecureSkipVerify: true setting under the TLS configuration and instead supply
a trusted CA bundle or OpenShift service cert for TLS verification; update the
resource that configures prometheus scrape authorization (the
authorization.credentials block referencing name:
workload-variant-autoscaler-metrics-reader-token and key: token) to include a
proper tlsConfig.caFile (or equivalent OpenShift service-to-service cert
reference) so certificate validation is enforced rather than bypassed.

Signed-off-by: greg pereira <grpereir@redhat.com>
Signed-off-by: greg pereira <grpereir@redhat.com>
interval: 10s
scheme: https
tlsConfig:
insecureSkipVerify: true
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lacking full context, do we have to do this?

Copy link
Copy Markdown
Member

@anishasthana anishasthana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@anishasthana anishasthana merged commit 71ddcba into opendatahub-io:main Mar 27, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants