[BUG] Agents page cannot load: controller-v2 never wires the system service, so ListNamespaces always returns INTERNAL

[Charlesthebird]

Summary

The UI's Agents page fails to load on every install, showing Could not load agents — internal server error. The page is fine; SystemService/ListNamespaces returns INTERNAL on every call, and the agents list treats the namespace read as its own failure because templates are read one namespace at a time.

$ curl -s -D - -o /dev/null -X POST \
    localhost:8080/api/kagent.api.v1alpha1.SystemService/ListNamespaces \
    -H 'Content-Type: application/grpc-web+proto' --data-binary @empty.grpcweb
Grpc-Message: internal server error
Grpc-Status: 13

Reproduce

1. Stand up a cluster with scripts/setup-cluster/setup-cluster.sh.
2. Open Agents. The page shows Could not load agents — internal server error and lists nothing, although kubectl get agenttemplates,harnesses -n kagent returns the harness and template the script created.

Root cause

systemservice.WithInventory is the only thing that gives the system service a Kubernetes client, and it has no caller:

core/internal/service/system/service.go:111   func WithInventory(...)   ← declaration only

It used to be called from go/core/pkg/app/app.go, which 221674d1 — refactor: remove legacy controller runtime (#2595) — deleted:

-	systemService := systemservice.NewService(systemservice.WithInventory(
-		mgr.GetClient(),
-		watchNamespacesList,
-		extensionCfg.Authorizer,
-		substrateAteClient,
-	))

core/cmd/controller-v2/main.go never carried that over, so grpcserver.New falls back to its own default:

if config.SystemService == nil {
    config.SystemService = systemservice.NewService()   // no options → no kube client
}

and every namespace read stops at the guard in ListNamespaces:

if s.kubeClient == nil {
    return nil, serviceerrors.NewInternal("Failed to list namespaces", fmt.Errorf("kubernetes client is not configured"))
}

The silent default is what let this ship: a missing service becomes a runtime INTERNAL on first use rather than a failure to start.

Same deletion, still open

Probed against the running controller, grpc-status 12 unknown service:

• ModelService, ToolService, PromptTemplateService, FeedbackService, MemoryService are not registered at all — controller-v2 never constructs them. Models, MCP Servers and Prompts are unreachable for the same reason. (Distinct from [BUG] MCP servers never appear in the list — nothing writes the toolserver table since the legacy reconciler was removed #2603, which is about the toolserver table having no writer; these RPCs do not exist on the server.)
• SystemService/GetSubstrateStatus fails even once the inventory is wired: managerScheme in core/cmd/controller-v2/main.go registers only clientgoscheme and kagentv1alpha3, while the deleted app.go also registered atev1alpha1. Listing worker pools and actor templates has no kind registered for the type.

Why nothing appears in the controller logs

Nothing in go/ calls ctrl.SetLogger/log.SetLogger. controller-runtime prints

[controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.

and then discards every ctrllog write — the RPC completed lines, and the cause behind each redacted internal server error. The failure above is invisible server-side, which is most of why it reads as a broken local setup.

Version

Observed on 9e246fd3.

---

🤖 written by Claude

[Charlesthebird]

#2602 covers most of this: it wires systemservice.WithInventory, registers ModelService, ToolService and PromptTemplateService, adds atev1alpha1 to the manager scheme, calls ctrl.SetLogger, and adds a RemoteMCPServer discovery controller that writes the toolserver table (#2603). #2614 is closed in its favour.

Not covered there: FeedbackService and MemoryService are still unregistered, so their RPCs return unknown service. Leaving this open for those two.

---

🤖 written by Claude

[Charlesthebird]

Resolved in #2602