Intermittent clusterIP service connectivity issue.

[mohd-shahalam]

I am facing intermittent connection drop while connecting to the clusterIP service.

Environmental Info:
RKE2 Version:
1.33.6

Node(s) CPU architecture, OS, and Version:
Nodes:
1 Master, 2 Worker.
CPU- 4 CPU each
OS- RHEL 9.6

Cluster Configuration:
1 server and 2 agents

Describe the bug:
Below is a snip of intermittent connectivity issue

Snip-1

~ # nslookup cluster-1-rw
;; communications error to 10.43.0.10#53: timed out
;; Got recursion not available from 10.43.0.10
Server: 10.43.0.10
Address: 10.43.0.10#53

Name: cluster-1-rw.staging.svc.cluster.local
Address: 10.43.150.88
;; Got recursion not available from 10.43.0.10

~ # nslookup cluster-1-rw
;; communications error to 10.43.0.10#53: timed out
;; communications error to 10.43.0.10#53: timed out
;; Got recursion not available from 10.43.0.10
Server: 10.43.0.10
Address: 10.43.0.10#53

Name: cluster-1-rw.staging.svc.cluster.local
Address: 10.43.150.88
;; Got recursion not available from 10.43.0.10

~ # nslookup cluster-1-rw
;; communications error to 10.43.0.10#53: timed out
;; Got recursion not available from 10.43.0.10
Server: 10.43.0.10
Address: 10.43.0.10#53

Name: cluster-1-rw.staging.svc.cluster.local
Address: 10.43.150.88
;; communications error to 10.43.0.10#53: timed out
;; Got recursion not available from 10.43.0.10

~ #

Snip-2
~ # nslookup cluster-2-rw
;; Got recursion not available from 10.43.0.10
Server: 10.43.0.10
Address: 10.43.0.10#53

Name: cluster-2-rw.staging.svc.cluster.local
Address: 10.43.20.38
;; communications error to 10.43.0.10#53: timed out
;; communications error to 10.43.0.10#53: timed out
;; communications error to 10.43.0.10#53: timed out
;; no servers could be reached

~ # nslookup cluster-2-rw
;; Got recursion not available from 10.43.0.10
Server: 10.43.0.10
Address: 10.43.0.10#53

Name: cluster-2-rw.staging.svc.cluster.local
Address: 10.43.20.38
;; Got recursion not available from 10.43.0.10

~ #

Steps To Reproduce:
Step-1 Installed rke2 cluster
Step-2 Installed openebs for storage
Step-3 Installed postgres cnpg customresources
Step-4 Create cnpg cluster
Step-5 Create busybox pod for connection test.

Expected behavior:

Actual behavior:

Additional context / logs:

[mohd-shahalam]

This issue I am getting in almost all services.

mongodb connection

~ # for i in $(seq 1 10); do nslookup mongodb-0.staging.svc.cluster.local || echo FAIL; done
;; Got recursion not available from 10.43.0.10
;; Got recursion not available from 10.43.0.10
;; communications error to 10.43.0.10#53: timed out
;; Got recursion not available from 10.43.0.10
;; Got recursion not available from 10.43.0.10
Server: 10.43.0.10
Address: 10.43.0.10#53

Name: mongodb-0.staging.svc.cluster.local
Address: 10.43.64.202
;; Got recursion not available from 10.43.0.10

;; Got recursion not available from 10.43.0.10
;; Got recursion not available from 10.43.0.10
;; communications error to 10.43.0.10#53: timed out
;; communications error to 10.43.0.10#53: timed out
;; communications error to 10.43.0.10#53: timed out
;; no servers could be reached

FAIL
;; Got recursion not available from 10.43.0.10
;; communications error to 10.43.0.10#53: timed out
;; communications error to 10.43.0.10#53: timed out
;; Got recursion not available from 10.43.0.10
;; communications error to 10.43.0.10#53: timed out
;; no servers could be reached

FAIL
;; communications error to 10.43.0.10#53: timed out
;; communications error to 10.43.0.10#53: timed out
;; communications error to 10.43.0.10#53: timed out
;; no servers could be reached

FAIL
;; communications error to 10.43.0.10#53: timed out
;; communications error to 10.43.0.10#53: timed out
;; Got recursion not available from 10.43.0.10
;; Got recursion not available from 10.43.0.10
;; communications error to 10.43.0.10#53: timed out
;; no servers could be reached

FAIL
;; communications error to 10.43.0.10#53: timed out
;; Got recursion not available from 10.43.0.10
;; communications error to 10.43.0.10#53: timed out
;; communications error to 10.43.0.10#53: timed out
;; no servers could be reached

FAIL
;; communications error to 10.43.0.10#53: timed out
;; Got recursion not available from 10.43.0.10
;; Got recursion not available from 10.43.0.10
;; Got recursion not available from 10.43.0.10
;; communications error to 10.43.0.10#53: timed out
;; communications error to 10.43.0.10#53: timed out
;; no servers could be reached

FAIL
;; Got recursion not available from 10.43.0.10
;; communications error to 10.43.0.10#53: timed out
;; communications error to 10.43.0.10#53: timed out
;; Got recursion not available from 10.43.0.10
;; communications error to 10.43.0.10#53: timed out
;; no servers could be reached

FAIL
;; communications error to 10.43.0.10#53: timed out
;; communications error to 10.43.0.10#53: timed out
;; Got recursion not available from 10.43.0.10
;; Got recursion not available from 10.43.0.10
;; communications error to 10.43.0.10#53: timed out
;; no servers could be reached

FAIL
;; Got recursion not available from 10.43.0.10
;; Got recursion not available from 10.43.0.10
;; Got recursion not available from 10.43.0.10
;; communications error to 10.43.0.10#53: timed out
;; Got recursion not available from 10.43.0.10
Server: 10.43.0.10
Address: 10.43.0.10#53

Name: mongodb-0.staging.svc.cluster.local
Address: 10.43.64.202
;; Got recursion not available from 10.43.0.10

~ # nslookup mongodb-0
;; communications error to 10.43.0.10#53: timed out
;; Got recursion not available from 10.43.0.10
Server: 10.43.0.10
Address: 10.43.0.10#53

Name: mongodb-0.staging.svc.cluster.local
Address: 10.43.64.202
;; Got recursion not available from 10.43.0.10

~ #