feat(reachability): Network-path diagnosis

[hisco]

Reachability · Network Path — honest, DevOps-grade path diagnosis

TL;DR

A new Reachability view for the kinds that carry traffic — Service, Ingress, HTTPRoute, GRPCRoute, Gateway — that answers one operator question: "can traffic get to my app, and if not, exactly where and why does it break?" It reads like a DevOps engineer built it for someone who isn't a Kubernetes expert, and its defining constraint is honesty: it never says "broken" when traffic flows, and never says "healthy/reachable" for something it couldn't actually verify.

Two modes: Config check (passive, zero traffic - reads the declared path + config breaks) and Live test (on demand - DNS/TCP/TLS/HTTP probes, reported per-route and per-vantage). Surfaces: a full Reachability tab (a node/edge Diagram + a per-route/per-vantage matrix), a passive drawer glance, and the MCP diagnose tool extended to network kinds.

---

How to review this (suggested reading order)

It's a large diff but it's one feature with a clear spine. Read in this order:

1. internal/trace/coverage.go — the honesty spine. RouteResult{Outcome, Confidence}, CoverageVerdict, CoverageHeadline. Understand real vs indirect confidence here and the rest follows.
2. pkg/probe/probe.go — the probe primitives + the two vantages (real path vs apiserver-proxy) + proxyResult / isClusterUnreachable (couldn't-reach ≠ unreachable).
3. internal/trace/trace.go — computeVerdict / reviseVerdictWithProbes (how findings + probes become healthy/degraded/broken/unknown).
4. internal/trace/entries.go — how each subject's static path graph (hops + findings) is built.
5. UI: packages/k8s-ui/src/components/trace/reachVerdict.ts (operator-facing verdict), traceToSubgraph.ts (node/edge coloring), TraceSummary.tsx (drawer glance).

What to scrutinize: the honesty invariants below — each is load-bearing and each has a test that pins it. If you can find a case that false-condemns (says broken when traffic flows) or overclaims (says reachable/healthy without real verification), that's the bug that matters most here.

---

High-level architecture

                         ┌──────────────────────────────────────────────┐
   K8s cache (informers) │  STATIC TRACE            internal/trace        │
   Services, Endpoints,  │  build the path graph: entry → backends → pods │
   Pods, Ingress, GW,    │  attach findings (missing ref, no-ready,       │
   Routes, NetPol,       │  targetPort, scaled-to-0, ingress-controller,  │
   IngressClass          │  TLS, NetworkPolicy prediction)                │
                         └───────────────┬──────────────────────────────┘
                                         │ Hops + Findings
                    ┌────────────────────┼────────────────────┐
                    │ (optional)         │                     │
            ┌───────▼────────┐   ┌───────▼────────┐    ┌───────▼─────────┐
            │ ACTIVE PROBES  │   │ IN-CLUSTER TEST│    │  COVERAGE        │
            │ pkg/probe      │   │ internal/      │    │  PROJECTION      │
            │ DNS/TCP/TLS/   │   │ reachability   │    │  coverage.go     │
            │ HTTP           │   │ short-lived    │    │  per-route       │
            │ 2 vantages:    │   │ Jobs/pods,     │    │  outcome +       │
            │ • real path    │   │ RBAC-gated,    │    │  confidence      │
            │ • apiserver    │   │ self-deleting  │    │  (real|indirect) │
            │   proxy        │   │ (MUTATING)     │    │                  │
            └───────┬────────┘   └───────┬────────┘    └───────┬─────────┘
                    └────────────────────┴─────────────────────┘
                                         │ Routes + Coverage
                            ┌────────────▼────────────┐
                            │  VERDICT                 │
                            │  computeVerdict +        │
                            │  reviseVerdictWithProbes │
                            └────────────┬─────────────┘
                                         │  Trace (JSON contract)
              ┌──────────────────────────┼───────────────────────────┐
        ┌─────▼──────┐          ┌─────────▼─────────┐         ┌────────▼────────┐
        │ Reach tab  │          │  Drawer glance    │         │  MCP diagnose   │
        │ Diagram +  │          │  TraceSummary     │         │  (agents)       │
        │ matrix     │          │  passive, calm    │         │                 │
        └────────────┘          └───────────────────┘         └─────────────────┘

The two ideas that do the heavy lifting

1. Vantage confidence. Every probe is tagged by how it learned a result. real = tested the way traffic flows (direct dial / in-cluster). indirect = reached only via the API-server proxy — it localizes a problem but never sets the headline and never condemns the real path (the proxy bypasses NetworkPolicy and can fail for vantage-only reasons). The one exception: an authoritative cache fact (e.g. 0 ready endpoints) is definitive regardless of vantage and is promoted to a real failure.
2. Node = own health, edge = the path. A node's color is the resource's own health (did it answer), never how we reached it or whether a downstream route broke. Per-route reachability lives on the edges.

---

Honesty invariants (each pinned by a test)

Invariant	Where	Pinned by
A Service with some ready pods still serves → degraded, never broken	trace.go hopReachSeverity; traceToSubgraph.ts	TestComputeVerdict_PartialReadyBackendIsDegradedNotBroken; traceToSubgraph "partial-ready backend → amber"
0 ready endpoints is definitive → red, not soft "via the proxy" amber	coverage.go upgradeDefinitiveBackendDown	TestUpgradeDefinitiveBackendDown (incl. multi-port + uncertain-warning-stays-soft)
Couldn't reach the cluster ≠ unreachable → skip	pkg/probe/probe.go isClusterUnreachable	TestIsClusterUnreachable
A failed probe never reads as "reached a server"	trace.go anyProbeReached	TestAnyProbeReached_FailedProbeIsNotReached
An apiserver-proxy-only failure never condemns the real path	coverage.go coverageBannerTone, singleRouteHeadline	reachVerdict / coverage suites
No-healthy-backend headline blocked when a sibling backend is healthy / unreadable / selectorless / external	reachVerdict.ts backendDown	reachVerdict "multi-backend", "RBAC-redacted sibling", "selectorless sibling"
Scaled-to-0 is deliberate dormancy, not an outage	coverage.go, TraceSummary.tsx	TraceSummary "scaled-to-0" + coverage benign-softening
Node color = own health, not route outcome or probe vantage	traceToSubgraph.ts	traceToSubgraph "node = own health" suite

---

Refinements from review

Later review rounds tightened the honesty model (all test-pinned):

• One verdict, one source. The shipped verdict is collapsed to a single coverage-honest value stored on the trace and read identically by REST, the UI, and MCP - no per-surface drift. The in-cluster test still upgrades an apiserver-only ("unknown") trace to real once it confirms the data path.
• Network path, not app health. Any HTTP response (including an app's own 5xx) reads as reached - the app answered, the path works. Only a proxy's own 502/504 (its upstream is unreachable) or the apiserver failing to reach the backend is a network fault. A valid cert that is merely expiring soon stays reachable (advisory), never a "TLS failure".
• Per-layer failures. A degraded route names the exact layer that broke via failedLayer (tcp / tls / http, or upstream for a 502/504) instead of a generic "server error".
• UI: route rows are pills (Verified · 200 / Reached · 404 / Not tested) with the reason on a hover tooltip; a single diagram view (the tree/diagram toggle and ReachabilityTree were removed).

---

Screenshots

Captured against a local kind cluster of fixtures (healthy, crashlooping, partially-ready, image-pull, ingress with/without controller, TLS, multi-port, scaled-to-zero, external-name).
1. Backend down - names the cause, links the culprit, doesn't blame the network. The headline leads with the real cause ("container 'c' is crashlooping"), credits the wiring ("the break is in the pods, not the network"), and the node panel links straight to the failing Pod + the exact kubectl logs. The Service node stays neutral (wired fine); the red is on the Pod.

2. Coverage matrix + the honest verdict. Each route × FROM RADAR · YOUR MACHINE / VIA API SERVER / IN-CLUSTER, with per-route pills (Verified · 200). Reached only via the API-server proxy is never a confident green - the verdict reads "Backend answered via API server - the real network path isn't confirmed".

3. Per-port truth. A multi-port Service is reported per port - :80 verified, :9090 unreachable (a targetPort mismatch - nothing listens on 9090) - instead of collapsing to one verdict.

4. HTTP 404 is reached, not broken. We diagnose the network path, not app health: any HTTP response means traffic reached the app, so a 404 reads green-dashed "reached", never a false "unreachable".

5. Resource drawer. The passive glance in the resource drawer surfaces the operational issue ("No endpoints - 0/1 ready") with one CTA into the live test. It never nags.

---

File map (80 files)

• internal/trace/ (24) — the engine: trace.go (verdict), coverage.go (route projection), entries.go (path graph), probes.go (probe orchestration), findings.go, netpol.go (NetworkPolicy evaluator), ingress_controller.go (controller tier), egress.go. The cycleN_fixes_test.go files are test-only regression suites from review rounds.
• pkg/probe/ (2) — probe primitives (DNS/TCP/TLS/HTTP, apiserver-proxy) + classification.
• internal/reachability/ (4) — the in-cluster probe runner (short-lived Jobs).
• packages/k8s-ui/src/components/trace/ (21) - the UI: ReachabilityView (the node/edge diagram), ReachabilityExplainer (per-route/per-vantage matrix + pills), reachVerdict, traceToSubgraph, TraceSummary (drawer glance), probe-display. (ReachabilityTree was removed - single diagram view; TracePanel now provides shared helpers, not the rendered tab.)
• internal/mcp/ (5), internal/server/ (4) — MCP diagnose extension + the /api/trace endpoint.
• internal/k8s/ (6) — cache wiring for the kinds the trace reads.
• web/, cmd/, docs/, deploy/helm, README — wiring, flags, docs.

Test plan

• Go: go test ./... (incl. internal/trace, pkg/probe) — green.
• Frontend: tsc clean, full vitest suite green (incl. reachVerdict / traceToSubgraph / TraceSummary / probe-display).
• Verified end-to-end against a live kind cluster across the full fixture matrix, including cluster-down behavior (probes skip honestly instead of condemning).
• Hardened via multi-round cross-model review; all findings resolved or skipped-with-evidence.

Known limitations (deliberately deferred)

Surfaced by an independent critic pass and consciously left for a follow-up — each would be net-negative or out-of-scope to fix now:

• Core-NetworkPolicy "would-deny" on CNI-CRD clusters (Cilium/Calico). The static evaluator reads core NetworkPolicy only. A path a CNI CRD policy allows could still show an amber "a rule would block traffic". Not capped: core NP is enforced on those CNIs too, so suppressing it would under-report a real deny — and the finding is hedged + clearable by the in-cluster test (subject to the real CNI).
• Backend IsNotFound during informer sync. A just-applied or cross-scope backend can briefly read as a missing-ref break until the cache syncs. Narrow + self-healing; a scope-aware guard risks a worse false-clear.
• Succeeded/Job pods behind a Service. The finding is already softened at the detection layer; a rare metadata-level amber remains for the Service-fronting-Jobs case. Excluding terminal pods from selection risks a broad semantic change.

Scope / risk notes

• The live test emits real traffic against the declared path (bounded to a 3s budget; a timeout returns an honest partial).
• The in-cluster option is mutating — it creates up to 5 short-lived, self-deleting probe pods under the caller's RBAC; the only diagnose option with a side effect, gated on create jobs + list/get pods.
• Custom probe path supported (method fixed to GET; per-request headers deferred).

---

Note

High Risk
Adds mutating in-cluster probe Jobs, broad REST/MCP surface, and verdict logic where proxy-only failures must not false-condemn paths—security and honesty regressions are the main risk.

Overview
Introduces network path diagnosis for Service, Ingress, HTTPRoute, GRPCRoute, and Gateway: a static hop chain (upstream / subject / downstream) built from the informer cache, with existing detections attached per hop and coverage-style verdicts that distinguish real datapath results from indirect API-server-proxy probes.

API & MCP: GET /api/trace/{kind}/{ns}/{name} with optional ?probe=true (and path); POST routes for single-target and whole-subject in-cluster tests plus a capability check. MCP diagnose gains the same shape for network kinds (probe, mutating inCluster), with Cloud Member gating aligned to REST.

Probing: New radar probe subcommand (pkg/probe) and internal/reachability runner that creates restricted, short-lived Jobs under the caller’s RBAC; image resolution via --reachability-image / config, self-read (MY_POD_NAME), RADAR_IMAGE (Helm), or version-tagged default. make kind-load-probe supports local kind dev.

Detection tweaks: Scale-to-zero recognition extended to Argo Rollouts (with ErrDynamicNotReady for ambiguous dynamic cache state); Service “0 ready” stays critical when Rollout lookup is uncertain.

^{Reviewed by Cursor Bugbot for commit d1face5. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Wrong cross-namespace service DNS

Medium Severity

Cross-namespace in-cluster reachability dials backends as name.namespace.svc:port, but cluster DNS expects name.namespace (or the full name.namespace.svc.cluster.local). The extra .svc segment is not a valid Kubernetes service DNS label, so TCP probes can fail with NXDOMAIN or hit the wrong name while the backend is healthy.

Additional Locations (1)

• packages/k8s-ui/src/components/trace/TracePanel.tsx#L677-L682

 

^{Reviewed by Cursor Bugbot for commit 211460f. Configure here.}

[cursor]

Rollout sync softens real outages

Medium Severity

When a Service has zero ready pods but a matching Deployment or StatefulSet still has replicas greater than zero, the new Rollout cache path can still return uncertain, so detection emits a warning instead of the critical no-ready-endpoints finding until the Rollout informer syncs or the list succeeds.

Additional Locations (2)

• internal/k8s/detect.go#L1828-L1830
• internal/k8s/detect.go#L640-L659

 

^{Reviewed by Cursor Bugbot for commit efecce5. Configure here.}

[cursor]

Unsanitized HTTP probe paths

Low Severity

The trace API accepts a custom HTTP probe path via ?path= and the in-cluster trace body, but unlike handleProbeInCluster it never normalizes that value with path.Clean. httpPath only trims and adds a leading slash, so segments like .. can reach the probe Job and proxy requests as a different path than the operator intended.

Additional Locations (2)

• internal/server/reachability_run.go#L240-L241
• internal/trace/probes.go#L31-L40

 

^{Reviewed by Cursor Bugbot for commit 4280fb3. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 4 total unresolved issues (including 3 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7785c6a. Configure here.}