bug(operator): deleted MCPServer StatefulSet is not recreated

[RaviTharuma]

Bug description

For stdio MCPServer objects the operator creates two workloads:

1. a Deployment (<name>) — proxy-runner (thv proxy / HTTP listener)
2. a StatefulSet (<name>) — the actual MCP process

If the StatefulSet is deleted (mistaken “orphan” cleanup, kubectl delete sts, disk-pressure recovery, etc.), the operator does not recreate it.

• MCPServer stays Ready / “MCP server is running”
• the proxy Deployment stays up
• the Service still points at the proxy
• clients get a dead or empty backend until a human deletes the MCPServer CR and lets GitOps/the operator recreate the whole stack

Annotating the MCPServer or restarting the operator does not bring the StatefulSet back.

Steps to reproduce

1. Deploy a stdio MCPServer (image or npx/uvx command) so both the proxy Deployment and the workload StatefulSet exist.
2. kubectl delete statefulset <name> -n <ns>
3. Wait several reconcile intervals. Restart the operator pod if you want.
4. Observe: StatefulSet stays gone. MCPServer phase remains Ready.

Recovery that works:

kubectl delete mcpserver <name> -n <ns>
# if GitOps: reconcile the kustomization / re-apply the CR

Recovery that does not work: operator restart, MCPServer annotation, kubectl rollout restart deploy/<name>.

Expected behavior

The operator owns the StatefulSet. If it is missing, the next reconcile should create it again (same as a missing Service or proxy Deployment).

Actual behavior

Missing StatefulSet is not healed. The only reliable fix is deleting the MCPServer CR so the operator rebuilds proxy + workload.

This is easy to hit because kubectl get pods shows two pods per stdio server. People (and audit scripts) assume the STS pod is leftover.

Environment (if relevant)

• OS/version: Talos Linux, Kubernetes
• ToolHive version: originally hit on earlier 0.3x; still the dual Deployment+StatefulSet model on v0.43.0

Additional context

We previously deleted 15 StatefulSets after a shallow ownerRef/endpoints check labeled them orphans. All 15 stdio backends broke. Recreating the CRs restored them.

Please also document the dual-pod model in the operator runbook (Deployment = proxy, StatefulSet = workload; never delete the STS).

Related: #3329 (operator reverts STS replica patches — it does watch the STS for drift, but not for deletion). #3179 / #4877 (proxy ↔ STS lifecycle).

No credentials or private hostnames in this report.