Skip to content

bug(vmcp): Ready and /health stay OK while initialize can hang past client timeouts #6345

Description

@RaviTharuma

Bug description

VirtualMCPServer can report healthy while a client-facing initialize hangs long enough to look down.

On v0.43.0 we still see:

  • CR Ready=True, status.discoveredBackends[].status=ready, circuit breaker closed, message=Healthy
  • GET /health → 200 {"status":"ok"} (when probed in-cluster / via the proxy)
  • POST /mcp initialize from a real client: one request timed out at 12s with 0 bytes; the next request to the same path returned 200 in ~3s
  • The same aggregate on a prefixed path (/<name>/mcp) answered immediately

So “Ready” / /health / backend Healthy are not a reliable signal that a new MCP session will succeed within a normal gateway/client timeout.

#5861 (slow backend coin-flips the tenant) was closed as fixed by #6162 (“skip known-bad backends when opening sessions”). This looks like a leftover of the same class: health is not the same as “session open is fast and succeeds”.

Steps to reproduce

  1. Run a VirtualMCPServer with several backends (we used 5), exposed through an HTTP gateway.
  2. Confirm kubectl get virtualmcpserver Ready and every discoveredBackends[].status=ready.
  3. From outside the cluster, POST initialize with a 12s client timeout, several times in a row, including after a period of idle.

We are not claiming a deterministic 50% fail rate anymore (that was #5861 on v0.34.0). We are claiming Ready/health can be green while initialize still sometimes exceeds 12s or returns nothing.

Expected behavior

  • /health and the Ready condition should fail (or go Degraded) if a new session cannot initialize within the configured timeout.
  • Alternatively, document that Ready only means “process up + last backend probe ok”, and ship a separate SessionReady / latency SLO.

Actual behavior

Ready + Healthy + /health 200 while a client initialize can still time out. A retry a few seconds later succeeds. Easy to page the wrong layer (gateway vs vMCP vs backend).

Environment (if relevant)

  • OS/version: Talos Linux, Kubernetes, Cilium Gateway API in front of the vMCP Service
  • ToolHive version: operator + vmcp v0.43.0

Additional context

Related: #5861 (closed, #6162), #6339 (degraded for unadvertised capabilities — inverse: status too pessimistic), #6276 (GET /metrics returns MCP 406; /health is the one unauthenticated JSON endpoint that does answer).

If the remaining hang is “first session after idle aggregates all backends synchronously”, a bounded initialize budget + best_effort default would match what #5861 asked for.

No credentials or private hostnames in this report.

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triageIssue needs initial triage by a maintainer

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions