test: refactor operatorpolicy tests to run faster

[jan-law]

Notes

Aiming to reduce total e2e test time by 10min without adding more flakes

refactor tests ref: https://issues.redhat.com/browse/ACM-26529
test flake ref: https://issues.redhat.com/browse/ACM-23597

Warning

The commits "test: fix flaky reconcile counter" and "test: run crd tests with a different operator" reduce but do not eliminate these two test flakes. Any suggestions would be appreciated

Summary of changes

1. Removed top-level Ordered decorator. Tests will run in parallel test processes unless marked with Serial
2. Refactored setup/cleanup methods to run each Describe container with its own parent policy and namespace suffixed with a hash of the test name
3. Patched the yaml files with utils.KubectlJSONPatchToFile to update the namespaces and policy name suffixes before creating the resources
4. Nested 5 tests that create grc example-operator with AllNamespaces mode into an Ordered Describe container
5. Nested 4 tests that delete the same CRD into an Ordered Describe container. Replaced the operator in these tests with a different operator and CRD than the other tests
6. Marked 3 tests as Serial: 2 tests that break the catalog source (including ACM-23597), and 1 test that counts controller reconciles

Assisted by Claude 4.5

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jan-law

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [jan-law]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jan-law]

This test flakes occasionally. Both times have the same events, passed the Eventually() in checkFunc, but failed at Consistently(). The last It() step in the test intentionally breaks the spec/subscription/sourceNamespace, but then an install plan starts installing during Consistently(). The subscription status changes to Compliant instead of remaining at the expected ConstraintsNotSatisfiable.

Update: The previous step always installs the install plan and changes the subscription condition if the patch in the last step does not run fast enough. Thinking of ways to handle this

{
          "lastTransitionTime": "2025-12-04T19:14:57Z",
          "message": "the Subscription matches what is required by the policy",
          "reason": "SubscriptionMatches",
          "status": "True",
          "type": "SubscriptionCompliant"
        },
        {
          "lastTransitionTime": "2025-12-04T19:14:23Z",
          "message": "the policy spec is valid",
          "reason": "PolicyValidated",
          "status": "True",
          "type": "ValidPolicySpec"
        }
      ],
      "history": [
        {
          "lastTimestamp": "2025-12-04T19:14:58.015269Z", # unexpected
          "message": "NonCompliant; the policy spec is valid, the OperatorGroup matches what is required by the policy, the Subscription matches what is required by the policy, a relevant InstallPlan is actively installing, the ClusterServiceVersion required by the policy was not found, no CRDs were found for the operator, there are no relevant deployments because the ClusterServiceVersion is missing, CatalogSource 'operatorhubio-catalog' was not found"
        },
        {
          "lastTimestamp": "2025-12-04T19:14:55.701762Z", # this should be the last message
          "message": "NonCompliant; the policy spec is valid, the OperatorGroup matches what is required by the policy, constraints not satisfiable: refer to the Subscription for more details, there are no relevant InstallPlans in the namespace, the ClusterServiceVersion required by the policy was not found, no CRDs were found for the operator, there are no relevant deployments because the ClusterServiceVersion is missing, CatalogSource 'operatorhubio-catalog' was not found"
        },
...
  wanted related objects: [{Properties:<nil> Object:{Metadata:{Name:project-quay Namespace:operator-policy-testns-124d3267} APIVersion:operators.coreos.com/v1alpha1 Kind:Subscription} Compliant:NonCompliant Reason:ConstraintsNotSatisfiable}]
  wanted condition: {Type:SubscriptionCompliant Status:False ObservedGeneration:0 LastTransitionTime:0001-01-01 00:00:00 +0000 UTC Reason:ConstraintsNotSatisfiable Message:constraints not satisfiable: refer to the Subscription for more details}
...
[FAILED] Failed after 2.327s.
  The function passed to Consistently failed at /home/runner/work/config-policy-controller/config-policy-controller/test/e2e/case38_install_operator_test.go:169 with:
  Expected
      <string>: Compliant
  to equal
      <string>: NonCompliant

logs:
https://github.com/open-cluster-management-io/config-policy-controller/actions/runs/19911108613/job/57178128413?pr=419

https://github.com/open-cluster-management-io/config-policy-controller/actions/runs/19972532656/job/57280596312

[jan-law]

Going to skip the Consistently() check here

[jan-law]

On one test flake, the logs indicated the subscription wasn't patched, then timed out. Hopefully retrying helps. https://github.com/open-cluster-management-io/config-policy-controller/actions/runs/19972532656/job/57291714147#logs

2025-12-05T21:04:04.310Z	debug	controllers/operatorpolicy_controller.go:1711	..."controllerKind": "OperatorPolicy", "OperatorPolicy": {"name":"oppol-manual-upgrades-e8e934d1","namespace":"managed"}, ... "subscriptionConditionMessage": "constraints not satisfiable: no operators found with name strimzi-cluster-operator.v0.0.0.1337 in channel strimzi-0.36.x of package strimzi-kafka-operator in the catalog referenced by subscription strimzi-kafka-operator, subscription strimzi-kafka-operator exists"}

[jan-law]

Note: Anyone who add tests to this file needs to be very careful if they decide to create an APIcast or Authorino operator, as I've used those for cluster-scoped tests. As far as I know, the other operators in the tests can be reused in parallel