Bug description
VirtualMCPServer can report healthy while a client-facing initialize hangs long enough to look down.
On v0.43.0 we still see:
- CR
Ready=True, status.discoveredBackends[].status=ready, circuit breaker closed, message=Healthy
GET /health → 200 {"status":"ok"} (when probed in-cluster / via the proxy)
POST /mcp initialize from a real client: one request timed out at 12s with 0 bytes; the next request to the same path returned 200 in ~3s
- The same aggregate on a prefixed path (
/<name>/mcp) answered immediately
So “Ready” / /health / backend Healthy are not a reliable signal that a new MCP session will succeed within a normal gateway/client timeout.
#5861 (slow backend coin-flips the tenant) was closed as fixed by #6162 (“skip known-bad backends when opening sessions”). This looks like a leftover of the same class: health is not the same as “session open is fast and succeeds”.
Steps to reproduce
- Run a
VirtualMCPServer with several backends (we used 5), exposed through an HTTP gateway.
- Confirm
kubectl get virtualmcpserver Ready and every discoveredBackends[].status=ready.
- From outside the cluster,
POST initialize with a 12s client timeout, several times in a row, including after a period of idle.
We are not claiming a deterministic 50% fail rate anymore (that was #5861 on v0.34.0). We are claiming Ready/health can be green while initialize still sometimes exceeds 12s or returns nothing.
Expected behavior
/health and the Ready condition should fail (or go Degraded) if a new session cannot initialize within the configured timeout.
- Alternatively, document that Ready only means “process up + last backend probe ok”, and ship a separate
SessionReady / latency SLO.
Actual behavior
Ready + Healthy + /health 200 while a client initialize can still time out. A retry a few seconds later succeeds. Easy to page the wrong layer (gateway vs vMCP vs backend).
Environment (if relevant)
- OS/version: Talos Linux, Kubernetes, Cilium Gateway API in front of the vMCP Service
- ToolHive version: operator + vmcp v0.43.0
Additional context
Related: #5861 (closed, #6162), #6339 (degraded for unadvertised capabilities — inverse: status too pessimistic), #6276 (GET /metrics returns MCP 406; /health is the one unauthenticated JSON endpoint that does answer).
If the remaining hang is “first session after idle aggregates all backends synchronously”, a bounded initialize budget + best_effort default would match what #5861 asked for.
No credentials or private hostnames in this report.
Bug description
VirtualMCPServercan report healthy while a client-facinginitializehangs long enough to look down.On v0.43.0 we still see:
Ready=True,status.discoveredBackends[].status=ready, circuit breakerclosed,message=HealthyGET /health→ 200{"status":"ok"}(when probed in-cluster / via the proxy)POST /mcpinitializefrom a real client: one request timed out at 12s with 0 bytes; the next request to the same path returned 200 in ~3s/<name>/mcp) answered immediatelySo “Ready” /
/health/ backendHealthyare not a reliable signal that a new MCP session will succeed within a normal gateway/client timeout.#5861 (slow backend coin-flips the tenant) was closed as fixed by #6162 (“skip known-bad backends when opening sessions”). This looks like a leftover of the same class: health is not the same as “session open is fast and succeeds”.
Steps to reproduce
VirtualMCPServerwith several backends (we used 5), exposed through an HTTP gateway.kubectl get virtualmcpserverReady and everydiscoveredBackends[].status=ready.POST initializewith a 12s client timeout, several times in a row, including after a period of idle.We are not claiming a deterministic 50% fail rate anymore (that was #5861 on v0.34.0). We are claiming Ready/health can be green while
initializestill sometimes exceeds 12s or returns nothing.Expected behavior
/healthand the Ready condition should fail (or go Degraded) if a new session cannotinitializewithin the configured timeout.SessionReady/ latency SLO.Actual behavior
Ready + Healthy +
/health200 while a clientinitializecan still time out. A retry a few seconds later succeeds. Easy to page the wrong layer (gateway vs vMCP vs backend).Environment (if relevant)
Additional context
Related: #5861 (closed, #6162), #6339 (degraded for unadvertised capabilities — inverse: status too pessimistic), #6276 (
GET /metricsreturns MCP 406;/healthis the one unauthenticated JSON endpoint that does answer).If the remaining hang is “first session after idle aggregates all backends synchronously”, a bounded
initializebudget +best_effortdefault would match what #5861 asked for.No credentials or private hostnames in this report.