Reachability: honest, DevOps-grade network-path diagnosis

Author: hisco

CLAUDE.md

@@ -137,6 +137,7 @@ Use `/visual-test` command for the full workflow (cluster check, Playwright MCP,
 - GitOps detail data: `/api/gitops/tree/{kind}/{ns}/{name}` (resource tree + ownership edges), `/api/gitops/insights/{kind}/{ns}/{name}` (curated diagnosis: summary + issues + drift + events + plan + history + capabilities)
 - Nodes: `/api/nodes/{name}/...` (cordon, uncordon, drain, debug)
 - Audit: `/api/audit`, `/api/audit/resource/{kind}/{ns}/{name}`, `/api/settings/audit` (GET/PUT)
+- Network trace: `/api/trace/{kind}/{ns}/{name}` (path-shaped diagnosis for Service/Ingress/HTTPRoute/GRPCRoute/Gateway; `?probe=true` runs DNS/TCP/TLS/HTTP probes against the declared path — direct TCP in-cluster, K8s API server proxy from a laptop, gated by the user's `services/proxy` + `pods/proxy` RBAC).
 - CAPI: `/api/capi/clusters/{ns}/{name}/kubeconfig` (GET), `/api/capi/clusters/{ns}/{name}/connect` (POST)
 - RBAC reverse-lookup: `/api/rbac/subject/{kind}/{namespace}/{name}` (ServiceAccount) and `/api/rbac/subject/{kind}/{name}` (User/Group) return direct + group-inherited bindings + flattened effective rules. SA subjects also get a `usedByPods` list (Pods whose `spec.serviceAccountName` matches — closes the loop on the SA detail page). `/api/rbac/role/{kind}/{namespace}/{name}` (use `_` for ClusterRole's empty namespace) returns the inverse — bindings that reference the role + their subjects. `/api/rbac/namespace/{namespace}` returns RoleBindings in the namespace + ClusterRoleBindings with at least one SA subject in it + a ServiceAccount count (backs the NamespaceRenderer's RBAC section; group-only ClusterRoleBindings like `system:authenticated` grants are deliberately excluded — they'd appear in every namespace and would be noise). `/api/rbac/whoami?namespace=...` is a pass-through of `SelfSubjectRulesReview` for the current user. Backed by `pkg/rbac/` (pure index + 5s TTL memo); endpoints gate on `list rolebindings` AND `list clusterrolebindings` (403 when either is denied — silent partial views would mislead operators).
 

Makefile

@@ -28,6 +28,23 @@ deploy-test: frontend embed
 	kubectl rollout status deploy/$(CLUSTER_DEPLOY) -n $(CLUSTER_NS) --timeout=60s
 	@echo "=== Done. Tail logs: kubectl logs -n $(CLUSTER_NS) -l app.kubernetes.io/name=$(CLUSTER_DEPLOY) -f ==="
 
+# Build a probe image from the CURRENT code and load it into a kind cluster, for
+# developing the in-cluster reachability probe. A -dirty local build's version is
+# not a published tag, so the default image won't exist — load a local one and run
+# radar with --reachability-image $(PROBE_IMAGE). Binary path /radar + ENTRYPOINT
+# match the official image so the probe Job's `["/radar","probe",...]` command works.
+KIND_CLUSTER ?= test
+PROBE_IMAGE  ?= radar-probe:dev
+kind-load-probe:
+	@echo "Building probe binary for linux/$$(go env GOARCH)..."
+	GOOS=linux GOARCH=$$(go env GOARCH) CGO_ENABLED=0 go build -ldflags "$(LDFLAGS)" -o /tmp/radar-probe-bin ./cmd/explorer
+	@echo 'FROM gcr.io/distroless/static-debian12:nonroot' > /tmp/Dockerfile.probe
+	@echo 'COPY radar-probe-bin /radar' >> /tmp/Dockerfile.probe
+	@echo 'ENTRYPOINT ["/radar"]' >> /tmp/Dockerfile.probe
+	docker build -t $(PROBE_IMAGE) -f /tmp/Dockerfile.probe /tmp
+	kind load docker-image $(PROBE_IMAGE) --name $(KIND_CLUSTER)
+	@echo "Loaded $(PROBE_IMAGE) into kind/$(KIND_CLUSTER). Run radar with: --reachability-image $(PROBE_IMAGE)"
+
 ## Build targets
 
 # Build the complete application (frontend + embedded binary)

README.md

@@ -362,6 +362,18 @@ Proactive best-practices scanner with 31 checks across security, reliability, an
 - Framework labels: NSA/CISA, CIS benchmarks
 - MCP tool (`get_cluster_audit`) for AI-assisted cluster analysis
 
+### Network Path Diagnose
+
+Hop-ordered diagnosis for Service, Ingress, HTTPRoute, GRPCRoute, and Gateway — answering "if traffic is sent toward this resource, does it reach a healthy process, and if not which hop breaks first?"
+
+- Composes the detections Radar already runs (missing backend Service, port mismatches, no-ready-endpoints, route not Accepted by parent Gateway, readiness probe targeting the wrong port) into a path shape ordered along the traffic flow
+- Upstreams (Ingresses / Routes pointing at a Service) are judged independently — one broken Ingress doesn't condemn the other delivery paths
+- First critical hop is named explicitly so the operator can localize the break without reading the whole list; each finding ships a kubectl reproducer
+- **Optional one-shot reachability test** runs DNS / TCP / TLS / HTTP probes against the declared path — direct TCP when Radar is in-cluster, K8s API server proxy when running from a laptop — so the same button works regardless of where Radar runs. Probes never override the static verdict; they add evidence.
+- NetworkPolicies that select the subject's pods are statically evaluated for their caller-independent ingress rules: a "would block" **WARNING prediction** when no rule admits the path's port, a source-restricted advisory, or an outbound egress note. It's a prediction, never a verdict — the CNI is the only enforcement authority, so the live in-cluster probe confirms or downgrades it
+- Static trace is pure functions over the in-memory informer cache. Active probing from a laptop uses the cluster's normal RBAC (`get services/proxy`, `get pods/proxy`); in-cluster mode goes directly to the data path.
+- Exposed via the **Diagnose** tab in the resource detail view and via the MCP `diagnose` tool for AI consumers — see [docs/diagnose.md](docs/diagnose.md)
+
 ### Access Control (RBAC visibility)
 
 Inspect what any ServiceAccount can actually do — without three `kubectl describe` calls.

cmd/explorer/main.go

@@ -21,6 +21,7 @@ import (
 	"github.com/skyhook-io/radar/internal/config"
 	"github.com/skyhook-io/radar/internal/k8s"
 	mcppkg "github.com/skyhook-io/radar/internal/mcp"
+	"github.com/skyhook-io/radar/internal/reachability"
 	"github.com/skyhook-io/radar/internal/server"
 	"golang.org/x/net/http/httpguts"
 	_ "k8s.io/client-go/plugin/pkg/client/auth" // Register all auth provider plugins (OIDC, GCP, Azure, etc.)
@@ -34,13 +35,27 @@ var (
 func main() {
 	startupStart := time.Now()
 
+	// Subcommand dispatch: `radar probe` runs a one-shot reachability probe and
+	// exits — no server, no cluster client. The in-cluster reachability runner
+	// executes this binary in that mode. Intercept before the server flag
+	// parsing, since "probe" is a positional argument, not a flag.
+	if len(os.Args) > 1 && os.Args[1] == "probe" {
+		os.Exit(runProbeCommand(os.Args[2:]))
+	}
+
 	// Propagate the build-time version to the cloud dialer so the agent
 	// advertises the real version (e.g. "1.5.5") on the tunnel handshake
 	// instead of the "dev" default. Dockerfile + Makefile inject
 	// `-X main.version=...`; mirror it here rather than adding a second
 	// ldflags target so there's a single source of truth.
 	cloud.Version = version
 
+	// The in-cluster reachability probe Job runs the version-matched published
+	// Radar image by default (overridable via --reachability-image / RADAR_IMAGE).
+	// The flag override is applied just after it is parsed (see below) so BOTH the
+	// REST handler and the MCP diagnose(inCluster) path resolve the same image.
+	reachability.DefaultImageRef = "ghcr.io/skyhook-io/radar:" + version
+
 	// Load persistent config (~/.radar/config.json) for flag defaults.
 	// CLI flags override config file values.
 	fileCfg := config.Load()
@@ -62,6 +77,7 @@ func main() {
 	disableLocalTerminal := flag.Bool("disable-local-terminal", false, "Disable local terminal feature")
 	podShellDefault := flag.String("pod-shell-default", "", "Override the default pod exec shell command (runs as 'sh -c <value>'; empty = built-in bash -il → ash → sh cascade)")
 	debugImage := flag.String("debug-image", fileCfg.DebugImage, "Image for ephemeral debug containers and node debug pods (empty = busybox:latest; point at a mirror for air-gapped/private-registry clusters)")
+	reachabilityImage := flag.String("reachability-image", fileCfg.ReachabilityImage, "Image for the in-cluster reachability probe Job (empty = RADAR_IMAGE env, then the version-matched published Radar image; point at a mirror for air-gapped clusters)")
 	listPageSize := flag.Int64("list-page-size", 0, "Paginate the initial LIST of high-cardinality kinds (Pods, ReplicaSets) at this page size on clusters without WatchList streaming. 0 = off (single LIST). Try 2000 if a very large cluster fails to sync.")
 	// Timeline storage options
 	timelineStorage := flag.String("timeline-storage", fileCfg.TimelineStorageOr("memory"), "Timeline storage backend: memory or sqlite")
@@ -118,6 +134,16 @@ func main() {
 	maxScopeCandidates := flag.Int("max-scope-candidates", k8s.EnvIntOr("RADAR_MAX_SCOPE_CANDIDATES", 20), "Cap on the namespace-fallback probe fanout for users who can list namespaces cluster-wide but not list a specific kind cluster-wide (default: 20). Raise for clusters with more than 20 namespaces to avoid silently marking kinds as denied in dropped namespaces. Env: RADAR_MAX_SCOPE_CANDIDATES")
 	flag.Parse()
 
+	// An explicit --reachability-image override applies to BOTH probe paths. It must
+	// win over the in-cluster self-read, so record it as the configured override
+	// (checked ABOVE self-read), not as DefaultImageRef (the last-resort fallback,
+	// below self-read — which would let radar's own pod image beat the operator's
+	// explicit choice on the MCP path). REST passes the config as the override arg;
+	// MCP has no server config, so it relies on this. Air-gapped mirrors work on both.
+	if *reachabilityImage != "" {
+		reachability.SetConfiguredImage(*reachabilityImage)
+	}
+
 	// Cloud-mode: Radar runs inside a customer cluster and fronts Radar
 	// Cloud. Under cloud-mode the tunnel is the only path to this listener
 	// and it delivers Cloud-authenticated identity headers on every request.
@@ -211,6 +237,7 @@ func main() {
 		DisableLocalTerminal:     *disableLocalTerminal,
 		PodShellDefault:          *podShellDefault,
 		DebugImage:               *debugImage,
+		ReachabilityImage:        *reachabilityImage,
 		ListPageSize:             *listPageSize,
 		TimelineStorage:          *timelineStorage,
 		TimelineDBPath:           *timelineDBPath,

cmd/explorer/probe_cmd.go

@@ -0,0 +1,94 @@
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"net"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/skyhook-io/radar/pkg/probe"
+)
+
+// runProbeCommand implements `radar probe` — run reachability probes against ONE
+// target from wherever this process runs and print the results as JSON. When the
+// process runs inside a cluster pod, the probes traverse the real dataplane
+// (subject to NetworkPolicy + mesh mTLS), which is exactly what the laptop's
+// apiserver-proxy view cannot confirm. No HTTP server, no Kubernetes client:
+// probe-and-exit. This is the binary the in-cluster reachability runner executes
+// (the Radar image invoked as `radar probe ...`).
+func runProbeCommand(args []string) int {
+	fs := flag.NewFlagSet("probe", flag.ContinueOnError)
+	target := fs.String("target", "", "host:port to probe (host alone for a dns-only check)")
+	host := fs.String("host", "", "TLS SNI / HTTP Host header (default: the host part of --target)")
+	scheme := fs.String("scheme", "http", "http or https (https adds a TLS check and uses an https URL)")
+	path := fs.String("path", "/", "request path for the HTTP probe")
+	layersFlag := fs.String("layers", "tcp,http", "comma-separated layers: dns,tcp,tls,http")
+	timeout := fs.Duration("timeout", 3*time.Second, "overall probe budget")
+	jsonOut := fs.Bool("json", true, "emit JSON (false = one human line per probe)")
+	if err := fs.Parse(args); err != nil {
+		return 2
+	}
+	if *target == "" {
+		fmt.Fprintln(os.Stderr, "radar probe: --target host:port is required")
+		return 2
+	}
+
+	h := *host
+	if h == "" {
+		if hp, _, err := net.SplitHostPort(*target); err == nil && hp != "" {
+			h = hp
+		} else {
+			h = *target
+		}
+	}
+
+	ctx, cancel := context.WithTimeout(context.Background(), *timeout)
+	defer cancel()
+	vantage := probe.DetectVantage()
+
+	want := map[string]bool{}
+	for _, l := range strings.Split(*layersFlag, ",") {
+		if t := strings.TrimSpace(strings.ToLower(l)); t != "" {
+			want[t] = true
+		}
+	}
+
+	var results []probe.Result
+	if want["dns"] {
+		results = append(results, probe.DNS(ctx, h, vantage))
+	}
+	if want["tcp"] {
+		results = append(results, probe.TCP(ctx, *target, vantage))
+	}
+	if want["tls"] || strings.EqualFold(*scheme, "https") {
+		results = append(results, probe.TLS(ctx, *target, h, vantage))
+	}
+	if want["http"] {
+		reqPath := *path
+		if !strings.HasPrefix(reqPath, "/") {
+			reqPath = "/" + reqPath
+		}
+		results = append(results, probe.HTTP(ctx, fmt.Sprintf("%s://%s%s", *scheme, *target, reqPath), h, vantage))
+	}
+
+	if *jsonOut {
+		enc := json.NewEncoder(os.Stdout)
+		enc.SetIndent("", "  ")
+		if err := enc.Encode(results); err != nil {
+			return 1
+		}
+		return 0
+	}
+	for _, r := range results {
+		detail := r.Detail
+		if detail == "" {
+			detail = r.Error
+		}
+		fmt.Printf("%-5s %-22s ok=%-5v tone=%-9s %s\n", r.Layer, r.Target, r.OK, r.Tone, detail)
+	}
+	return 0
+}

deploy/helm/radar/templates/deployment.yaml

@@ -179,11 +179,21 @@ spec:
               {{- end }}
             {{- end }}
             {{- end }}
-            {{- if .Values.rbac.selfUpgrade }}
+            - name: MY_POD_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.name
             - name: MY_POD_NAMESPACE
               valueFrom:
                 fieldRef:
                   fieldPath: metadata.namespace
+            # Pin the in-cluster reachability probe to radar's own deployed image.
+            # This is the no-RBAC fallback for when the runtime self-read of radar's
+            # pod can't run; an explicit --reachability-image / config still wins,
+            # and the self-read (radar's live pod image) is preferred when available.
+            - name: RADAR_IMAGE
+              value: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
+            {{- if .Values.rbac.selfUpgrade }}
             - name: MY_DEPLOYMENT_NAME
               value: {{ include "radar.fullname" . }}
             {{- end }}