bug(vmcp): HTTP 429 listing resource templates marks a backend unavailable

[RaviTharuma]

Bug description

On vmcp v0.43.0, a backend health check that lists resource templates treats an upstream HTTP 429 as backend unavailable. The VirtualMCPServer then reports that backend as unknown / degraded and sets Ready=False (BackendsDegraded) even when:

• partialFailureMode: best_effort is set
• MCP initialize on the aggregate still succeeds
• the same backend’s dedicated proxy answers initialize in well under a second

This fired when four VirtualMCPServers started at the same time and all probed the same remote MCP. Two of four aggregates stayed Degraded until the next health interval; the other two stayed Ready.

Steps to reproduce

1. Run operator + vmcp v0.43.0.
2. Point several VirtualMCPServers at the same remote MCP that implements Modern resource templates and rate-limits resources/templates (or any 429).
3. Restart all of those vMCP pods together.
4. Inspect logs and CR status.

Observed log line (backend name redacted):

WARN backend initialized with failure
  error: health check failed: backend unavailable: failed to list resource templates for backend <name>: modern backend returned a transient error: HTTP 429

Later:

INFO backend recovering from failures
  previous_status: unknown
  status: degraded

Expected behavior

• A transient 429 on an optional Modern call (resources/templates / resources/list / prompts/list) should not mark the backend unavailable.
• Health should retry with backoff, or treat template listing as non-fatal when tools/list / initialize succeeded.
• Ready should not flip False for a single degraded optional probe while best_effort is on and initialize works.

Actual behavior

• One 429 on resource templates → backend unknown, then degraded.
• Aggregate Ready=False / phase Degraded even though clients can still initialize.
• Recovery waits for the next healthCheckInterval (5m in our config).

Environment

• ToolHive operator + vmcp v0.43.0
• Kubernetes (Talos)

Additional context

Related: #6345 (Ready/health OK while initialize can hang — inverse symptom). #5861 (slow backend affecting the tenant; residual hang).

No credentials or private hostnames in this report.