How can I debug the local workload connection?

[xolott]

I have an app running on port 3000. Doing http://localhost:3000 on my browser works as expected.

Then I run these commands:

signadot local connect --cluster=my-cluster 

signadot sandbox apply -f ./sandbox.yaml --set user=xolott --set branch=feature-branch --set deployment=app --set namespace=app --set port=8080 --set localPort=3000

Where sandbox.yaml contains:

name: "@{user}-@{branch}-sandbox"
spec:
  cluster: "my-cluster"
  ttl:
    duration: 1d
  description: "@{user} local sandbox"
  labels:
    feature: "@{branch}"
  local:
    - name: "local-@{user}-@{deployment}"
      from:
        kind: Deployment
        namespace: "@{namespace}"
        name: "@{deployment}"
      mappings:
        - port: "@{port}"
          toLocal: "localhost:@{localPort}"

It fails with:

Waiting (up to --wait-timeout=3m0s) for sandbox to be ready...
✗ Sandbox status: Not Ready: connection is not ready

Dashboard page: https://app.signadot.com/sandbox/id/z0sbzckn4qk3n


The sandbox was applied, but it may not be ready yet. To check status, run:

  signadot sandbox get xolott-main-sandbox

Error: timed out after 3m0s

Is there a way to debug the connection?
Creating a sandbox in the cluster work as expected with our initial tests

[scott-cotton]

Hi @xolott

Thanks for the question!

Since the output you posted does not capture signadot local connect, but rather signadot sandbox apply, I'll assume that signadot local connect succeeded, and the problem is specific to the sandbox.

The first step to debugging the connection is to check the status of the sandbox: signadot sandbox get xolott-main-sandbox -o yaml will provide more details, including any errors relating to the installation such as whether the routing is ready and if not, why.

That might give a first (and easiest) clue.

Other things you can do to debug this are:

1. look at ~/.signadot/sandbox-manager.log and ~/.signadot/root-manager.log for any errors
2. run signadot local connect with debug (signadot --debug local connect ...). This increases the verbosity in the above logs.
3. run curl agent-metrics.signadot.svc:9090/metrics after signadot local connect, it should succeed giving something like this:

# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
# HELP signadot_agent_connection_attempts Count of attempts to connect
# TYPE signadot_agent_connection_attempts counter
signadot_agent_connection_attempts 2

If not, there may be a problem with the connection needed for the local sandbox to function.
4. use kubectl to check on the readiness of the signadot components in your cluster: kubectl -n signadot get pods.

Hope that helps!