Prometheus metrics queries over Skupper link result in EOF errors

[knuurr]

We’re running a setup where self-managed Grafana in one k8s cluster queries Prometheus / VMSelect metrics from another k8s cluster over a Skupper link.

When Grafana (or alerts/dashboards) tries to query the remote Prometheus/VMSelect, the request (often, but not always) fails with an EOF error. Example:

Post "http://****-vmselect-vm.monitoring.svc.cluster.local:8481/select/0/prometheus/api/v1/query-range": EOF

This doesn’t appear to happen when we connect between EKS clusters directly (without Skupper). The problem is specific to traffic routed through Skupper. That's why I don't think it might be only related to Grafana config.

Questions:

• How can I confirm whether this issue is indeed Skupper-related, vs. something in the Prometheus/VMSelect or network configuration?
• If it is Skupper-related, what could be the underlying reason for an EOF error on HTTP requests? (e.g., connection resets, timeout?)
• Where should I look to diagnose this — logs, connection configs, Skupper router settings?
• Once the cause is found, what remediation steps would be possible?

Additional details

• Works fine when querying between our different EKS clusters directly (no skupper link involved)
• Fails when queries are routed across the Skupper link.
• Affects dashboards and alerts (not just ad-hoc queries).

Any guidance on how to debug this (or pointers to known limitations) would be greatly appreciated!

[ted-ross]

It would be helpful if you would include what version of Skupper you are using.

[knuurr]

@ted-ross apologies for not including this info.

-> % skupper version -n monitoring
client version                 1.9.4
transport version              quay.io/skupper/skupper-router:2.7.6 (sha256:c7429db4cabd)
controller version             quay.io/skupper/service-controller:1.9.4 (sha256:01997ceac66d)
config-sync version            quay.io/skupper/config-sync:1.9.4 (sha256:a3ec698ca2c1)
flow-collector version         not-found

One more observation, not sure if important here, is that skipper-router (well one of the pods, since we have more than 1 replica at a time) is always full of these messages in logs:

2025/10/15 09:54:47 LOG_EVENT: {"recType":"LOGEVENT","identity":"tx827:466573","parent":"tx827:0","startTime":1760522087899555,"endTime":1760522087899555,"source":"tx827:0.logs","severity":3,"logText":"LOG_SERVER: [C466557] Connection from 172.22.99.206:20194 (to :55671) failed: amqp:connection:framing-error No protocol header found (connection aborted)","fileName":"/build/src/server.c","lineNumber":1109}
2025/10/15 09:54:48 Bridge config is synchronised
2025/10/15 09:54:48 LOG_EVENT: {"recType":"LOGEVENT","identity":"r76rp:466490","parent":"r76rp:0","startTime":1760522088645916,"endTime":1760522088645916,"source":"r76rp:0.logs","severity":3,"logText":"LOG_SERVER: [C466504] Connection from 172.22.123.108:43308 (to :45671) failed: amqp:connection:framing-error No protocol header found (connection aborted)","fileName":"/build/src/server.c","lineNumber":1109}
2025/10/15 09:54:48 LOG_EVENT: {"recType":"LOGEVENT","identity":"tx827:466574","parent":"tx827:0","startTime":1760522088773971,"endTime":1760522088773971,"source":"tx827:0.logs","severity":3,"logText":"LOG_SERVER: [C466558] Connection from 172.22.66.236:14670 (to :55671) failed: amqp:connection:framing-error No protocol header found (connection aborted)","fileName":"/build/src/server.c","lineNumber":1109}

Not sure what more info I could provide. Happy to provide whatever is needed!

[ajssmith]

Hi @knuurr apologize for the delay in responding. Those log events indicate that the inter-router links between sites is failing and then likely getting re-established. The prometheus traffic that traverses the links would then be affected. Can you check on the ingress that you use and see if there are any events that might correspond to the inter router connections being impacted?

Also, we would suggest that you try running single routers and not multiple replicas in the site.

If you continue to experience, router logs from both sides would be helpful to review.