Submariner looks like it is not in working condition

[nikitarm1611]

ISSUE Description

The submariner does not looks fine with the show-all and diagnose all commands.

Setup:
Site 1: OCP 4.18.22
Site 2: OCP 4.18.22

Submariner version: v0.19.2

Output Collected

show all and diagnose all command outputs for Site 2

This is the error we are seeing in metrodr rack on site2 pod

sh-5.1$ /root/.local/bin/subctl diagnose all --kubeconfig /tmp/local-kubeconfig
Cluster "local-config"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.31.11" is supported

 ✓ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
 ✓ Checking DaemonSet "submariner-gateway"
 ✓ Checking DaemonSet "submariner-routeagent"
 ✓ Checking DaemonSet "submariner-metrics-proxy"
 ✓ Checking Deployment "submariner-lighthouse-agent"
 ✓ Checking Deployment "submariner-lighthouse-coredns"
 ⚠ Checking the status of all Submariner pods
 ⚠ Pod "submariner-gateway-7vn4h" has restarted 63 times
 ✓ Checking that gateway metrics are accessible from non-gateway nodes 

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OVNKubernetes") is supported
 ✓ Checking OVN version 
 ✓ The ovn-nb database version 7.6.0 is supported
 ✗ Checking gateway connections
 ✗ Connection to cluster "site1" is in progress
 ✗ Checking route agent connections
 ✗ Connection to cluster "site1" is not established. Connection details:
{
  "status": "error",
  "statusMessage": "Failed to successfully ping the remote endpoint IP \"10.129.0.2\"",
  "spec": {
    "cluster_id": "site1",
    "cable_name": "submariner-cable-site1-10-48-96-195",
    "healthCheckIP": "10.129.0.2",
    "hostname": "control-1-ru2.f27l003.fusion.tadn.ibm.com",
    "subnets": [
      "172.30.0.0/16",
      "10.128.0.0/14"
    ],
    "private_ip": "10.48.96.195",
    "public_ip": "129.41.86.6",
    "nat_enabled": true,
    "backend": "libreswan",
    "backend_config": {
      "natt-discovery-port": "4490",
      "preferred-server": "false",
      "udp-port": "4500"
    }
  }
}
 ✗ Connection to cluster "site1" is not established. Connection details:
{
  "status": "error",
  "statusMessage": "Failed to successfully ping the remote endpoint IP \"10.129.0.2\"",
  "spec": {
    "cluster_id": "site1",
    "cable_name": "submariner-cable-site1-10-48-96-195",
    "healthCheckIP": "10.129.0.2",
    "hostname": "control-1-ru2.f27l003.fusion.tadn.ibm.com",
    "subnets": [
      "172.30.0.0/16",
      "10.128.0.0/14"
    ],
    "private_ip": "10.48.96.195",
    "public_ip": "129.41.86.6",
    "nat_enabled": true,
    "backend": "libreswan",
    "backend_config": {
      "natt-discovery-port": "4490",
      "preferred-server": "false",
      "udp-port": "4500"
    }
  }
}
 ✗ Connection to cluster "site1" is not established. Connection details:
{
  "status": "error",
  "statusMessage": "Failed to successfully ping the remote endpoint IP \"10.129.0.2\"",
  "spec": {
    "cluster_id": "site1",
    "cable_name": "submariner-cable-site1-10-48-96-195",
    "healthCheckIP": "10.129.0.2",
    "hostname": "control-1-ru2.f27l003.fusion.tadn.ibm.com",
    "subnets": [
      "172.30.0.0/16",
      "10.128.0.0/14"
    ],
    "private_ip": "10.48.96.195",
    "public_ip": "129.41.86.6",
    "nat_enabled": true,
    "backend": "libreswan",
    "backend_config": {
      "natt-discovery-port": "4490",
      "preferred-server": "false",
      "udp-port": "4500"
    }
  }
}
 ✗ Connection to cluster "site1" is not established. Connection details:
{
  "status": "error",
  "statusMessage": "Failed to successfully ping the remote endpoint IP \"10.129.0.2\"",
  "spec": {
    "cluster_id": "site1",
    "cable_name": "submariner-cable-site1-10-48-96-195",
    "healthCheckIP": "10.129.0.2",
    "hostname": "control-1-ru2.f27l003.fusion.tadn.ibm.com",
    "subnets": [
      "172.30.0.0/16",
      "10.128.0.0/14"
    ],
    "private_ip": "10.48.96.195",
    "public_ip": "129.41.86.6",
    "nat_enabled": true,
    "backend": "libreswan",
    "backend_config": {
      "natt-discovery-port": "4490",
      "preferred-server": "false",
      "udp-port": "4500"
    }
  }
}
 ✗ Connection to cluster "site1" is not established. Connection details:
{
  "status": "error",
  "statusMessage": "Failed to successfully ping the remote endpoint IP \"10.129.0.2\"",
  "spec": {
    "cluster_id": "site1",
    "cable_name": "submariner-cable-site1-10-48-96-195",
    "healthCheckIP": "10.129.0.2",
    "hostname": "control-1-ru2.f27l003.fusion.tadn.ibm.com",
    "subnets": [
      "172.30.0.0/16",
      "10.128.0.0/14"
    ],
    "private_ip": "10.48.96.195",
    "public_ip": "129.41.86.6",
    "nat_enabled": true,
    "backend": "libreswan",
    "backend_config": {
      "natt-discovery-port": "4490",
      "preferred-server": "false",
      "udp-port": "4500"
    }
  }
}
 ✓ Checking Submariner support for the kube-proxy mode
 ✓ Cluster is running with "OVNKubernetes" CNI which internally implements kube-proxy functionality
 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic

 ✓ Checking that services have been exported properly

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

Also in the metrodr CR below is the output for the submariner output

✓ Showing Gateways
          NODE                             HA STATUS   SUMMARY                                  
          control-1-ru2.f27l003.fusion.t   passive     There are no connections                 
          control-1-ru3.f27l003.fusion.t   passive     There are no connections                 
          control-1-ru4.f27l003.fusion.t   active      0 connections out of 1 are established

These are below submariner log

F29-submariner-20260129072032.zip

F27-submariner-20260129073303.zip

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

⚠️ Possible Duplicate Issue

• The Submariner Gateway pods are in Running state, but the connection between the sites is not happening, unable to find the root cause. #3701

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[BhavaniYalamanchili]

Hi team, we have tried reinstallation of the Submariner; this time it worked. Can you please give some insight on the root cause.

[vthapar]

Just looking at the output, there was likely some issue with the installation. I couldn't download the logs but can check GW pod log for errors as to why it was restarting so many times. Since GW pod didn't come up, there were no connections setup. Root cause would be whatever was causing GW pod restarts.

[BhavaniYalamanchili]

Hi @vthapar, if you are facing difficulty in downloading the zip files, try this way I am uploading them as .txt. Once you download you change the extension to .zip and let me know if you can access the logs. If you need specific logs, I am ready to provide.

F27-submariner-20260129073303.txt
F29-submariner-20260129072032.txt

[yboaron]

It looks like the F29 Gateway pod is failing to resolve its public IP, whereas the F27 site resolved it successfully.

By default, Submariner queries external servers to discover public IPs and runs NAT-Discovery to decide whether to connect via public or private IPs (details here).

Suggested Fix:

• If your gateways can reach each other via private IPs (== Node IPs), you can skip the auto-public IP resolution by adding an annotation to the Gateway node. Check the "Double NAT Traversal (Scenario 1)" section in the NAT-T docs for the specific annotation.
• If gateway aren't reachable via private IPs, make sure that GW on cluster F29 can communicate with the internet for auto resolve public IP

[shankha95]

Hi team, we have tried reinstallation of the Submariner; this time it worked. Can you please give some insight on the root cause.

Hi , i am facing the same problem with the error -

Failed to successfully ping the remote endpoint IP >> It stopped working suddenly , nothing changed in the network or in the clusters and it was working until last week.
Also my gateway pods are not restarting as well , they seem to be running.

root@control-plane:~/.kube# subctl show connections
Cluster "cluster-ash"
✓ Showing Connections
GATEWAY CLUSTER REMOTE IP NAT CABLE DRIVER SUBNETS STATUS RTT avg.
worker-1 cluster-hazel 192.168.4.13 no libreswan 10.43.0.0/16, 10.42.0.0/16 error 0s

Cluster "cluster-hazel"
✓ Showing Connections
GATEWAY CLUSTER REMOTE IP NAT CABLE DRIVER SUBNETS STATUS RTT avg.
worker-1 cluster-ash 192.168.4.11 no libreswan 10.96.0.0/16, 10.144.0.0/16 error 0s

root@control-plane:~/.kube# kubectl describe gateways.submariner.io worker-1 -n submariner-operator
Name: worker-1
Namespace: submariner-operator
Labels:
Annotations: update-timestamp: 1770127248
API Version: submariner.io/v1
Kind: Gateway
Metadata:
Creation Timestamp: 2026-02-03T13:59:53Z
Generation: 3
Resource Version: 140203083
UID: 4b1fcdaa-f43b-49d3-9ba1-5d160a6aaaff
Status:
Connections:
Endpoint:
Backend: libreswan
backend_config:
Natt - Discovery - Port: 4490
Preferred - Server: false
Udp - Port: 4500
cable_name: submariner-cable-cluster-ash-192-168-4-11
cluster_id: cluster-ash
Health Check IP: 10.144.5.0
Health Check I Ps:
10.144.5.0
Hostname: worker-1
nat_enabled: false
Private I Ps:
192.168.4.11
private_ip: 192.168.4.11
Public I Ps:
155.245.93.5
public_ip: 155.245.93.5
Subnets:
10.96.0.0/16
10.144.0.0/16
Latency RTT:
Average: 0s
Last: 0s
Max: 0s
Min: 0s
Std Dev: 0s
Status: error
Status Message: Failed to successfully ping the remote endpoint IP "10.144.5.0"
Using IP: 192.168.4.11
Ha Status: active
Local Endpoint:
Backend: libreswan
backend_config:
Natt - Discovery - Port: 4490
Preferred - Server: false
Udp - Port: 4500
cable_name: submariner-cable-cluster-hazel-192-168-4-13
cluster_id: cluster-hazel
Health Check IP: 10.42.5.0
Health Check I Ps:
10.42.5.0
Hostname: worker-1
nat_enabled: false
Private I Ps:
192.168.4.13
private_ip: 192.168.4.13
Public I Ps:
155.245.93.5
public_ip: 155.245.93.5
Subnets:
10.43.0.0/16
10.42.0.0/16
Status Failure:
Version: release-0.21-556a38149341
Events:

Just wanted to ask did you reinstall submainer with the new version v.22.0 ?? or just recreated the tunnel again ?

[shankha95]

Hi team, we have tried reinstallation of the Submariner; this time it worked. Can you please give some insight on the root cause.

Hi , i am facing the same problem with the error -

Failed to successfully ping the remote endpoint IP >> It stopped working suddenly , nothing changed in the network or in the clusters and it was working until last week. Also my gateway pods are not restarting as well , they seem to be running.

root@control-plane:~/.kube# subctl show connections Cluster "cluster-ash" ✓ Showing Connections GATEWAY CLUSTER REMOTE IP NAT CABLE DRIVER SUBNETS STATUS RTT avg. worker-1 cluster-hazel 192.168.4.13 no libreswan 10.43.0.0/16, 10.42.0.0/16 error 0s

Cluster "cluster-hazel" ✓ Showing Connections GATEWAY CLUSTER REMOTE IP NAT CABLE DRIVER SUBNETS STATUS RTT avg. worker-1 cluster-ash 192.168.4.11 no libreswan 10.96.0.0/16, 10.144.0.0/16 error 0s

root@control-plane:~/.kube# kubectl describe gateways.submariner.io worker-1 -n submariner-operator Name: worker-1 Namespace: submariner-operator Labels: Annotations: update-timestamp: 1770127248 API Version: submariner.io/v1 Kind: Gateway Metadata: Creation Timestamp: 2026-02-03T13:59:53Z Generation: 3 Resource Version: 140203083 UID: 4b1fcdaa-f43b-49d3-9ba1-5d160a6aaaff Status: Connections: Endpoint: Backend: libreswan backend_config: Natt - Discovery - Port: 4490 Preferred - Server: false Udp - Port: 4500 cable_name: submariner-cable-cluster-ash-192-168-4-11 cluster_id: cluster-ash Health Check IP: 10.144.5.0 Health Check I Ps: 10.144.5.0 Hostname: worker-1 nat_enabled: false Private I Ps: 192.168.4.11 private_ip: 192.168.4.11 Public I Ps: 155.245.93.5 public_ip: 155.245.93.5 Subnets: 10.96.0.0/16 10.144.0.0/16 Latency RTT: Average: 0s Last: 0s Max: 0s Min: 0s Std Dev: 0s Status: error Status Message: Failed to successfully ping the remote endpoint IP "10.144.5.0" Using IP: 192.168.4.11 Ha Status: active Local Endpoint: Backend: libreswan backend_config: Natt - Discovery - Port: 4490 Preferred - Server: false Udp - Port: 4500 cable_name: submariner-cable-cluster-hazel-192-168-4-13 cluster_id: cluster-hazel Health Check IP: 10.42.5.0 Health Check I Ps: 10.42.5.0 Hostname: worker-1 nat_enabled: false Private I Ps: 192.168.4.13 private_ip: 192.168.4.13 Public I Ps: 155.245.93.5 public_ip: 155.245.93.5 Subnets: 10.43.0.0/16 10.42.0.0/16 Status Failure: Version: release-0.21-556a38149341 Events:

Just wanted to ask did you reinstall submainer with the new version v.22.0 ?? or just recreated the tunnel again ?

So apparently , i deleted and recreated the tunnel between the clusters but within 24 hours it broke again . So it'd be great to get a root cause for this and if it would be better to upgrade to the latest version -v.22.0 ?

I am working with 2 clustersets -
both of them are on -
subctl version: v0.21.1
subctl version: v0.21.2

respectively.

[yboaron]

Hi @shankha95 ,
Thanks for the update, this looks like a different problem, so let's move it to new issue, please file a new issue for this.

Make sure the title reflects the specific behavior you're seeing, include details on your environment, and attach subctl gather from both clusters so we can take a closer look.

[yboaron]

@BhavaniYalamanchili Any update on this issue?

[BhavaniYalamanchili]

@yboaron We don't want to use double NATs,

and yes private ips are reachable to gateways and also the gateways can communicate with the internet