Option to verify backend connectivity before health check endpoints return 200

[dekokun]

Feature Description

When running alloydb-auth-proxy as a sidecar in Cloud Run Jobs with container-dependencies, there is no reliable way to ensure the proxy can actually reach AlloyDB before the dependent container starts.

Currently, both /startup and /readiness return 200 as soon as NotifyStarted() is called -- which happens right after the local TCP listener starts, before any actual connection to AlloyDB is established. In environments using PSC, the network path to AlloyDB can take 10+ seconds to become available after container startup, causing the dependent application container to fail with connection timeouts.

--run-connection-test partially addresses this but exits immediately on failure with no retry, making it unsuitable for sidecar use cases where transient delays are expected.

A flag (e.g., --startup-connection-check) that makes Serve() verify actual AlloyDB connectivity (with retries) before calling notify() would solve this problem. This would ensure /startup and /readiness only return 200 after the proxy can actually forward connections to AlloyDB.

Sample code

In internal/proxy/proxy.go, the change would be roughly:

func (c *Client) Serve(ctx context.Context, notify func()) error {
    // ... existing listener setup ...

    // New: if startup connection check is enabled, retry until AlloyDB is reachable
    if c.conf.StartupConnectionCheck {
        c.logger.Infof("Startup connection check: verifying backend connectivity...")
        for {
            if _, err := c.CheckConnections(ctx); err != nil {
                c.logger.Infof("Startup connection check: not ready, retrying in 1s: %v", err)
                select {
                case <-ctx.Done():
                    return ctx.Err()
                case <-time.After(1 * time.Second):
                    continue
                }
            }
            c.logger.Infof("Startup connection check: backend is reachable")
            break
        }
    }

    notify() // /startup and /readiness now return 200
    return <-exitCh
}

Alternatives Considered

Workaround	Limitation
--run-connection-test	Exits on first failure with no retry. Unusable as a sidecar when transient connectivity delays (e.g., PSC cold-start) are expected.
Switching startup probe from /startup to /readiness	No difference -- both return 200 at the same timing (NotifyStarted()). Note: the /readiness documentation says "when the proxy can connect to all registered instances" but the implementation does not actually check this.
Increasing application-side connect_timeout / pool_timeout	Band-aid that does not prevent the race condition. Requires changes in every application using the proxy.

Additional Details

• Running v1.13.7. Also reviewed the v1.14.1 source code and confirmed the same behavior.
• Connectivity method: PSC (--psc)
• Runtime: Cloud Run Jobs with container-dependencies annotation
• Auth: --auto-iam-authn with --impersonate-service-account
• The /readiness endpoint documentation states it checks "when the proxy can connect to all registered instances" but the implementation does not perform any backend connectivity check. This may be a separate documentation issue.

[nancynh]

Hi! Thanks for opening up a feature request! Usually we expect that so long as your AlloyDB instance is up and running (thus the Auth Proxy server would be as well), then the client would also be able to establish connections once it's ready. So, your point on "In environments using PSC, the network path to AlloyDB can take 10+ seconds to become available after container startup, causing the dependent application container to fail with connection timeouts" is interesting. Could you tell us more details on the 10s delay (such as any logs you see from the client during this, you can include more logs if you add --debug-logs flag for more info)? Also some information on how your network path is setup would also be helpful.

[enocom]

@dekokun any thoughts on the question above?

[dekokun]

@nancynh @enocom Hi, thanks for asking!

Environment Details

• Runtime: Cloud Run Jobs (gen2) with container-dependencies annotation
• Connectivity: PSC (--psc) via VPC network
• Auth: --auto-iam-authn with --impersonate-service-account
• Proxy version: v1.13.7 (at the time of the issue)
• Network: Cloud Run (Direct VPC egress) → VPC → PSC endpoint → AlloyDB
• Region: asia-northeast1

Observed Timeline (from production logs)

Here is a representative failure timeline from one execution (2026-03-19T04:25 UTC):

Time (UTC)	Event
04:25:03	Cloud Run Job execution created
04:25:20.556	alloydb-auth-proxy: Ignoring --disable-metrics... (proxy starting)
04:25:20.611	alloydb-auth-proxy: Listening on 127.0.0.1:5432
04:25:20.611	alloydb-auth-proxy: The proxy has started successfully and is ready for new connections!
04:25:24.924	STARTUP HTTP probe succeeded for alloydb-auth-proxy on port 9090 /startup → app container starts
04:25:44.824	app: Job starts, immediately attempts DB query via localhost:5432
04:25:45.034	alloydb-auth-proxy: accepted connection from 127.0.0.1:54310
04:25:54.948	app: ERROR — Timed out fetching a new connection from the connection pool (pool timeout: 10s, limit: 5)
04:25:55	app: Container called exit(1)

The proxy accepted the TCP connection at 04:25:45, but the actual TLS handshake + PSC network path to AlloyDB took longer than the application's 10-second connection pool timeout.

Scale of the issue

This was not an isolated incident. Over a ~7-hour window (2026-03-18T21:10Z – 2026-03-19T04:25Z), we observed 30+ distinct Cloud Run Job executions failing with the same pattern. The errors appeared across multiple job types:

• Can't reach database server at localhost:5432
• Timed out fetching a new connection from the connection pool (the proxy accepted the connection but couldn't complete the backend dial within the timeout)

All errors share the same root cause: the app container started (triggered by the proxy's /startup returning 200) and immediately tried to connect to localhost:5432, but the proxy had not yet established actual backend connectivity to AlloyDB via PSC.

Debug logs observation

We enabled --debug-logs later on the same day. With debug logs enabled, we could observe the proxy performing Connection info refresh operation and Dialing <PSC endpoint>:5433 during the first connection. Unfortunately, we were unable to capture --debug-logs output during a failure — the issue became less frequent after the redeployment required to enable debug logging. The timeline above is from logs captured before --debug-logs was enabled.

Network setup

• Cloud Run Job uses Direct VPC egress
• AlloyDB is accessed via a PSC endpoint (*.alloydb-psc.goog.:5433)

As described in the original issue's "Alternatives Considered" section, a --startup-connection-check flag would perfectly solve this by deferring notify() until the proxy can actually forward connections to AlloyDB.

[enocom]

The Auth Proxy taking 10 seconds to dial a remote server certainly isn't expected. Understanding why that happened is the most important detail here in my opinion.

As for the request, we historically had a downstream connection test in place as part of the health endpoints (maybe in /readiness, I can't remember), but ultimately removed it because there's a general rule to avoid adding dependencies on downstream networked services as part of health checks given how they can cause unpredictable behavior in a service.

As a side note, I also wonder what kind of CPU usage you see in the job when you observe this behavior -- it sounds like you're hitting a resource limit which slows down what is otherwise a fast operation. Catching a failure case when debug logging will at least narrow down what is slow (the AlloyDB API interaction to retrieve connection info, or the dataplane connection).

[dekokun]

@enocom I investigated the debug logs further and found a clear example of the issue.

10:20:21.171 INFO The proxy has started successfully and is ready for new connections!
10:20:21.171 INFO Starting health check server at 0.0.0.0:9090
10:20:48.449 INFO accepted connection from 127.0.0.1:47960
10:20:48.449 DEBUG Connection info refresh operation started
10:20:49.122 DEBUG Connection info refresh operation complete # 0.67s - OK
10:20:49.122 DEBUG Cert is valid = true
10:20:49.122 DEBUG Dialing :5433
10:29:49.292 INFO client closed the connection # ← 9 MINUTES later
10:29:49.292 INFO accepted connection from 127.0.0.1:59550 # 2nd attempt
10:29:49.292 DEBUG Cert is valid = true
10:29:49.292 DEBUG Dialing :5433
10:29:50.618 INFO client closed the connection # ← 1.3s - OK

For comparison, two other executions that started at the same time completed their Dialing in ~0.1s — likely reusing an already-warm PSC path.

Key findings:

• Connection info refresh is not the bottleneck (~0.5-0.7s across all executions).
• The first Dialing (PSC TCP connect) hangs for ~9 minutes until the client times out.
• The second Dialing to the same PSC endpoint completes in 1.3s, suggesting the PSC network path was initialized by then.

This points to the PSC network path initialization being the actual source of latency, not the connection info refresh.

[enocom]

This is not an expected behavior and is likely a problem either with Cloud Run's networking or something about the container's use of Direct VPC Egress. Have you raised this as a support case?

I think we don't want to add a new feature to resolve this problem -- there's something broken here and it's likely not the Proxy.

Are you using Managed Connection Pooling? One possible way for this to manifest would be if your instance had Managed Connection Pooling enabled but the server pool was maxed out. The 10 minute duration looks like you're hitting a limit somewhere (number of egress sockets from your container, number of available server connections in a managed pool, etc).

[dekokun]

@enocom Thanks for the analysis.

• We have not raised a support case yet — will do.
• We are not using Managed Connection Pooling.
• We are using Direct VPC Egress (private-ranges-only) to reach the AlloyDB PSC endpoint.

You're right that the ~9 minute hang on Dialing is likely a networking layer issue rather than the proxy itself. I'll raise a support case with Google Cloud to investigate the Cloud Run / Direct VPC Egress / PSC path.

Thanks for helping narrow this down.

[enocom]

Sounds good, @dekokun. I'll close this issue for now. If you have any further suggestions here (once the 10 minute networking issue is sorted out), we can re-open and discuss.

[dekokun]

@enocom Following up — I raised a support case with Google Cloud, and they confirmed this is a known limitation of Direct VPC Egress:

▎ When using Direct VPC Egress, connection establishment may be delayed by 1+ minutes at instance startup.

This is documented here: https://cloud.google.com/run/docs/configuring/vpc-direct-vpc#limitations

Google Cloud Support recommended configuring HTTP startup probes to handle this delay. However, the Auth Proxy's /startup endpoint currently returns 200 as soon as the proxy process is listening — it does not verify actual backend connectivity.
This means the startup probe cannot detect whether the Direct VPC Egress network path is ready.

Since this is a known infrastructure limitation (not a bug), I believe the Auth Proxy should support a way to defer readiness until backend connectivity is confirmed — for example, a --startup-connection-check flag as originally proposed.

Without this, users on Cloud Run Jobs with Direct VPC Egress + PSC have no way to use probes to handle this delay, which is the approach Google Cloud Support themselves recommend.

Would you be open to reconsidering this feature request?

[enocom]

I'm surprised to hear about the Direct VPC Egress taking more than a minute. Nonetheless, I think we can solve this here with an optional downstream connectivity check. We're stretched thin this week, but will take another look at this next week.