Enhancement: Add Cross-Cluster Connectivity Validation to ramenctl

[BenamarMk]

Problem

We've encountered a customer issue where creating a ServiceExport exposed an underlying network problem that had not been detected by any existing validation.

The flow is illustrated below:

                         Create ServiceExport
                                 │
                                 ▼
                   Lighthouse Agent (managed cluster)
                                 │
                                 │ HTTPS (TLS)
                                 ▼
                     Broker Kubernetes API Server
                                 │
                      TLS ClientHello sent
                                 │
                                 ▼
                    Normal host network routing
                     (NOT Submariner IPsec tunnel)
                                 │
                                 ▼
                 Path MTU is smaller than 1500 bytes
                                 │
                  PMTU discovery is not working
              (or ICMP "Fragmentation Needed" is blocked)
                                 │
                                 ▼
                Large TLS ClientHello packet is dropped
                                 │
                                 ▼
                  TLS handshake hangs / eventually times out
                                 │
                                 ▼
             Lighthouse Agent cannot update the Broker
                                 │
                                 ▼
          Broker never receives the exported service metadata
                                 │
                                 ▼
      Remote clusters never create the corresponding ServiceImport

Although the Submariner IPsec tunnel was healthy and data-plane traffic was functional, communication between the Lighthouse Agent and the Broker API failed because it follows the host network rather than the Submariner tunnel. As a result, the cluster pair appeared healthy until a ServiceExport was created.

Proposed Enhancement

Enhance ramenctl to perform end-to-end validation of the communication paths required by Ramen before disaster recovery operations are configured.

The validation should include:

1. Control-plane validation

   • Verify communication from each managed cluster to the Broker API.
   • Validate that creating a temporary ServiceExport results in the corresponding ServiceImport being created on the peer cluster.
   • Detect failures such as TLS handshake timeouts, MTU/PMTU issues, routing problems, or other connectivity failures that prevent Lighthouse from communicating with the Broker.

2. Data-plane validation

   • Validate Submariner data-plane connectivity independently of the control plane.
   • Create temporary VolSync ReplicationSource and ReplicationDestination resources.
   • Synchronize a small test PVC between the clusters.
   • Verify that replication completes successfully and clean up all temporary resources.

Expected Benefits

• Detect cross-cluster communication issues before they impact disaster recovery workflows.
• Validate both the Lighthouse control plane and the Submariner data plane independently.
• Identify network problems, including MTU, routing, TLS, or firewall issues
• Provide administrators with actionable diagnostics rather than discovering connectivity problems only after DR configuration or failover operations begin.

[nirs]

@BenamarMk This validation sounds useful.

What if instead of replicating what ramen does, we use ramen to do this validation?

We can do:

• deploy cephfs test application
• protect the application and wait for first replication
• unprotected the application
• undeploy the application

This verifies the entire system. If this fails we can gather the application which includes all the submariner resources so we have all the info for debugging.

This is similar to ramenctl test run but much later - skipping the failover and relocate that can take up to 15 each.

Based on CI times this will take about 2-3 minutes.

Creating this special test is very quick - basically creating a new simpler flow with the same code. We can backport this to 4.22 if needed.

It can work like this:

# config.yaml
...
tests:
- deployer: disapp
  workload: deploy
  pvcSpec: cpehfs

$ ramenctl test run --flow connectivity -o protect

If user want to test both cephfs and red they can use different configuration:

# config.yaml
...
tests:
- deployer: disapp
  workload: deploy
  pvcSpec: cpehfs
- deployer: disapp
  workload: deploy
  pvcSpec: rbd

With this configuration we can test connectivity in parallel with 2 applications. On failures all failed applications are gathered for debugging (already implemented).

--flow will default to "full" (deploy, protect, failover, relocate, unprotect, undeploy). We want to add names flows for long time - this is another reason to add them. For example we can use it to test more complicated flows (multiple failovers, multiple relocate, unprotect after protect, etc).

Is this good enough for validation?

[nirs]

1. Control-plane validation

   • Verify communication from each managed cluster to the Broker API.
   • Validate that creating a temporary ServiceExport results in the corresponding ServiceImport being created on the peer cluster.
   • Detect failures such as TLS handshake timeouts, MTU/PMTU issues, routing problems, or other connectivity failures that prevent Lighthouse from communicating with the Broker.

If protecting an application fails, we gather all the resources int the application namespaces.
We can add all the volsync resources to the validation report and validate them like we do for drpc, vrg, and protected pvcs.

This is lot of work that we need to plan for future release.

2. Data-plane validation

   • Validate Submariner data-plane connectivity independently of the control plane.
   • Create temporary VolSync ReplicationSource and ReplicationDestination resources.
   • Synchronize a small test PVC between the clusters.
   • Verify that replication completes successfully and clean up all temporary resources.

This is best done using ramen and busybox application. Validation for this is already implemented.

[nirs]

When test application fail, we gather the data. We can instead run validate application for the failed application, and create HTML report for each failed application. This will make debugging failures easier.

Example flow

1. Run connectivity test
2. Test fail, validate test application
3. HTML report open, shows the failed application including all volsync resources, showing problems like missing service export/import, volsync errors, etc.

[nirs]

Ways to implement

Testing connectivity can be done in 2 ways:

• Emulate ramen setup: Create pod, service, serviceexport, serviceimport, pod and validated that the service cannot be accessed. This is similar to how drenv submariner addon self test:
  https://github.com/RamenDR/ramen/blob/main/test/drenv/addons/submariner/test.py

• Test with ramen: Create a tiny application, protect it with ramen, and wait for the first replication. This is part of the full e2e dr flow:
  https://github.com/RamenDR/ramen/blob/06f48e4854d1e9c57c863c7fe2187135680079c6/e2e/dr_test.go#L76

Emulating how ramen use submariner is faster (40s in e2e CI) - we create the resource directly and can verify connectivity faster since we do not depend on rbd mirroring or volsync schedule. It is more precise, checking only networking - it will not fail because of storage issue, or problem in ramen or ACM. Since this does not use ramen it can succeed when real application fail if ramen changes the way it configures submariner, or not using submariner at all.

Testing with ramen is slower (~150s in e2e CI) since we depend on ramen protecting the application, on storage providing PVCs, and on rbd-mirror or volsync replicating the data. It is less precise since it can fail because of other issues (ramen, ocm, storage). But it is more correct since it tests the exact flow and configuration used by ramen.

Testing with ramen is much less code - we already have test using the full flow. We need to create s new partial flow that deploy, protect, and purge the application.

Testing with ramen aligns with future plans for ramenctl: we need configurable test flows anyway.