WLP was in failed state during runtime-enforcer upgrade from 0.5.0 to 0.6.0 upgrade on scale setup

[venkateshjayagopal]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- OS: Ubuntu 22.04.5 LTS 
- Architecture: x86
- Cluster: v1.34.2+rke2r1

Issue Description

• Expected behavior: WorkloadPolicy should be Active during upgrade.
• Current behavior: WorkloadPolicy was in failed state during upgrade.
• Steps to reproduce:

1. v0.5.0 was running in scale setup with 100+ nodes for 30 days
2. Upgraded to v0.6.0 today and observed the workloadPolicy was in 'failed' for few mins.

probably due to controller was waiting for all the runtime-enforcer-agent to upgrade?. Attached the console logs observed during upgrade.

runtime_enforcer_upgrade.txt

[holyspectral]

I'd probably have to look into this deeper, but the log indicates that

{"time":"2026-05-21T17:03:54.222784061Z","level":"ERROR","msg":"failed to get policies status","component":"controller","logger":"WorkloadPolicyStatusSync","err":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.42.4.110:50051: connect: connection refused\"","node":"stress-node-pool-10-08"}

I think it's related to how the current implementation discovers the active agent pod to connect to.  The pod list was probably stale.