Skip to content

LWS operator fails with ServiceMonitorCreateFailed on non-OpenShift clusters #38

@aneeshkp

Description

@aneeshkp

The Red Hat LWS operator (openshift/lws-operator) unconditionally creates a ServiceMonitor resource during reconciliation. On non-OpenShift Kubernetes clusters (AKS, CoreWeave) where the Prometheus Operator is not installed, the ServiceMonitor CRD does not exist, causing
continuous reconciliation failures.

Error

Event: type: 'Warning' reason: 'ServiceMonitorCreateFailed' Failed to create
ServiceMonitor.monitoring.coreos.com/lws-controller-manager-metrics-monitor -n
openshift-lws-operator: the server could not find the requested resource

"Unhandled Error" err="TargetConfigController reconciliation failed: the server
could not find the requested resource"

Root Cause

The operator's TargetConfigReconciler.manageServiceMonitor() (source) calls ApplyServiceMonitor() without checking if the ServiceMonitor CRD exists. The error
propagates up and fails the entire reconciliation loop.

Impact

  • The operator logs errors continuously on every reconciliation cycle
  • The LWS operand (controller-manager) still deploys successfully, but the operator reports a degraded state

Environment

  • Kubernetes: AKS (v1.33.5), CoreWeave
  • LWS Operator bundle: registry.redhat.io/leader-worker-set/lws-operator-bundle:1.0

Workaround

Install the ServiceMonitor CRD (from prometheus-operator) without the full monitoring stack:

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.78.2/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml

Suggested Fix (upstream)

In manageServiceMonitor(), check if the ServiceMonitor CRD exists before attempting to create the resource. If the CRD is not available, skip creation and log a warning instead of returning an error.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions