Add retry logic for tunnel connection attempts using backoff

thallgren · thallgren · commit d091d191613a · 2026-01-12T21:42:04.000+01:00
Introduce backoff-based retry mechanism in `dialer` to handle cases
where the target IP isn't ready to accept requests immediately.

Signed-off-by: Thomas Hallgren &lt;thomas@tada.se&gt;
diff --git a/CHANGELOG.yml b/CHANGELOG.yml
@@ -60,6 +60,11 @@ items:
           The Traffic Agent's retry interval when it establishes its watcher for intercepts is now configurable using the Helm chart value
           `agent.watchRetryInterval`. The default retry interval was also increased from 2 seconds to 10 seconds to improve resilience when
           connections to the traffic manager are lost.
+      - type: bugfix
+        title: Add retry logic for tunnel connection attempts
+        body: >-
+          The tunnel dialer now uses a backoff-based retry mechanism when establishing connections. This ensures that dialing attempts for
+          both TCP and UDP protocols persist if the target IP is not immediately ready to receive requests.
       - type: bugfix
         title: Retry mechanism for client tunnel creation
         body: >-
diff --git a/docs/release-notes.md b/docs/release-notes.md
@@ -32,6 +32,12 @@ The watchable map has been refactored into a client/server model that supports d
 The Traffic Agent's retry interval when it establishes its watcher for intercepts is now configurable using the Helm chart value `agent.watchRetryInterval`. The default retry interval was also increased from 2 seconds to 10 seconds to improve resilience when connections to the traffic manager are lost.
 </div>
 
+## <div style="display:flex;"><img src="images/bugfix.png" alt="bugfix" style="width:30px;height:fit-content;"/><div style="display:flex;margin-left:7px;">Add retry logic for tunnel connection attempts</div></div>
+<div style="margin-left: 15px">
+
+The tunnel dialer now uses a backoff-based retry mechanism when establishing connections. This ensures that dialing attempts for both TCP and UDP protocols persist if the target IP is not immediately ready to receive requests.
+</div>
+
 ## <div style="display:flex;"><img src="images/bugfix.png" alt="bugfix" style="width:30px;height:fit-content;"/><div style="display:flex;margin-left:7px;">Retry mechanism for client tunnel creation</div></div>
 <div style="margin-left: 15px">
 
diff --git a/docs/release-notes.mdx b/docs/release-notes.mdx
@@ -38,6 +38,12 @@ The watchable map has been refactored into a client/server model that supports d
 The Traffic Agent's retry interval when it establishes its watcher for intercepts is now configurable using the Helm chart value `agent.watchRetryInterval`. The default retry interval was also increased from 2 seconds to 10 seconds to improve resilience when connections to the traffic manager are lost.
   </Body>
 </Note>
+<Note>
+	<Title type="bugfix">Add retry logic for tunnel connection attempts</Title>
+	<Body>
+The tunnel dialer now uses a backoff-based retry mechanism when establishing connections. This ensures that dialing attempts for both TCP and UDP protocols persist if the target IP is not immediately ready to receive requests.
+  </Body>
+</Note>
 <Note>
 	<Title type="bugfix">Retry mechanism for client tunnel creation</Title>
 	<Body>
diff --git a/pkg/tunnel/dialer.go b/pkg/tunnel/dialer.go
@@ -13,6 +13,7 @@ import (
 	"sync/atomic"
 	"time"
 
+	"github.com/cenkalti/backoff/v4"
 	"google.golang.org/grpc/codes"
 	"google.golang.org/grpc/status"
 
@@ -167,12 +168,19 @@ func (h *dialer) Start(ctx context.Context) {
 			dtoCtx, cancel := context.WithTimeout(ctx, dto)
 			defer cancel()
 			var conn net.Conn
-			var err error
-			if id.Protocol() == types.ProtoUDP {
-				conn, err = d.DialUDP(dtoCtx, netip.AddrPort{}, id.Destination())
-			} else {
-				conn, err = d.DialTCP(dtoCtx, id.Destination())
-			}
+
+			// A retry is needed here because the attempt to establish a Tunnel might arrive before
+			// the target IP is ready to receive requests. The target IP might well be intercepted
+			// (or in progress of switching to become intercepted).
+			err := backoff.Retry(func() error {
+				var err error
+				if id.Protocol() == types.ProtoUDP {
+					conn, err = d.DialUDP(dtoCtx, netip.AddrPort{}, id.Destination())
+				} else {
+					conn, err = d.DialTCP(dtoCtx, id.Destination())
+				}
+				return err
+			}, backoff.WithContext(backoff.NewConstantBackOff(time.Second), dtoCtx))
 			if err != nil {
 				dlog.Errorf(ctx, "!> %s %s, failed to establish connection: %v", tag, id, err)
 				if err = h.stream.Send(ctx, NewMessage(DialReject, nil)); err != nil {
