OCPBUGS-51267: Fix an issue in the ComplianceScan controller

[Vincent056]

We were doing two updates without re‐fetching. This fixes it by ensuring we re-fetch before going to do the next step.

This will help with the stability of the controller logic.

[github-actions]

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:685-b1b7b22590e4a5385c3cc0420a4744ebdf947125

[yuumasato]

@Vincent056 Did this issue manifest in any way? Did you notice any strange behavior?

[Vincent056]

@Vincent056 Did this issue manifest in any way? Did you notice any strange behavior?

yes, I noticed the issues in the log, something like we were not able to apply the changes to object because it has being modified

[openshift-ci-robot]

@Vincent056: This pull request references Jira Issue OCPBUGS-51267, which is invalid:

• expected the bug to target the "4.19.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

We were doing two updates without re‐fetching. This fixes it by ensuring we re-fetch before going to do the next step.

This will help with the stability of the controller logic.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[github-actions]

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:685-01df89bd9aad0b7ed0113917f7469c35deaf4b1a

[xiaojiey]

@Vincent056 Without this PR, I can see the "Cannot update the scan with the check count" error log. With this PR, the error log disappears.
However, there are a lot of "Reconciler error" still exist in the operator logs with this PR. Is it expected? Thanks.

##Some error log without this PR:
% cat co.log| grep -i error | grep -i count    
{"level":"error","ts":"2025-02-26T03:46:57.346Z","logger":"scanctrl","msg":"Cannot update the scan with the check count","Request.Namespace":"openshift-compliance","Request.Name":"ocp4-cis-node-worker","error":"Operation cannot be fulfilled on compliancescans.compliance.openshift.io \"ocp4-cis-node-worker\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancescan.(*ReconcileComplianceScan).phaseAggregatingHandler\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancescan/compliancescan_controller.go:671\ngithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancescan.(*ReconcileComplianceScan).Reconcile\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancescan/compliancescan_controller.go:196\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:319\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:279\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:240"}
{"level":"error","ts":"2025-02-26T03:47:17.153Z","logger":"scanctrl","msg":"Cannot update the scan with the check count","Request.Namespace":"openshift-compliance","Request.Name":"ocp4-cis","error":"Operation cannot be fulfilled on compliancescans.compliance.openshift.io \"ocp4-cis\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancescan.(*ReconcileComplianceScan).phaseAggregatingHandler\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancescan/compliancescan_controller.go:671\ngithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancescan.(*ReconcileComplianceScan).Reconcile\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancescan/compliancescan_controller.go:196\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:319\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:279\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:240"}

##Error logs with this PR.

% cat co_new.log| grep -i error | grep -i count
% cat co_new.log | grep -i error:
...
{"level":"error","ts":"2025-02-26T04:34:02.758Z","msg":"Reconciler error","controller":"compliancesuite-controller","controllerGroup":"compliance.openshift.io","controllerKind":"ComplianceSuite","ComplianceSuite":{"name":"cis-compliance","namespace":"openshift-compliance"},"namespace":"openshift-compliance","name":"cis-compliance","reconcileID":"8a746dc4-e9ab-4f40-bae6-bf1039c271a8","error":"Error setting processing status for suite: Operation cannot be fulfilled on compliancesuites.compliance.openshift.io \"cis-compliance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:332\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:279\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:240"}
{"level":"error","ts":"2025-02-26T04:34:38.571Z","logger":"scanctrl","msg":"Cannot retrieve pod","Request.Namespace":"openshift-compliance","Request.Name":"ocp4-cis","Pod.Name":"aggregator-pod-ocp4-cis","error":"Pod \"aggregator-pod-ocp4-cis\" not found","stacktrace":"github.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancescan.isPodRunning\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancescan/compliancescan_controller.go:936\ngithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancescan.isAggregatorRunning\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancescan/aggregator.go:143\ngithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancescan.(*ReconcileComplianceScan).phaseAggregatingHandler\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancescan/compliancescan_controller.go:618\ngithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancescan.(*ReconcileComplianceScan).Reconcile\n\tgithub.com/ComplianceAsCode/compliance-operator/pkg/controller/compliancescan/compliancescan_controller.go:196\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:319\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:279\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:240"}

[github-actions]

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:685-933d728990a75daa57ef942af0000ea41180fe7a

[github-actions]

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:685-07be31b904fd4cb63b0ea2921684137c16fcc789

[rhmdnd]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rhmdnd, Vincent056

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Vincent056,rhmdnd]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rhmdnd]

cc: @mkumku @sheriff-rh @xiaojiey @BhargaviGudi for px, qe, and doc ack.

[xiaojiey]

/test e2e-aws-parallel

[xiaojiey]

/label qe-approved
I failed to reproduce the issue. Run a regression test and re-trigger e2e-aws-serial job instead.

[openshift-ci-robot]

@Vincent056: Jira Issue OCPBUGS-51267: Some pull requests linked via external trackers have merged:

• ComplianceAsCode/compliance-operator#685

The following pull requests linked via external trackers have not merged:

• ComplianceAsCode/compliance-operator#693 is open

These pull request must merge or be unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-51267 has not been moved to the MODIFIED state.

In response to this:

We were doing two updates without re‐fetching. This fixes it by ensuring we re-fetch before going to do the next step.

This will help with the stability of the controller logic.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.