Summary
When a Prometheus scrape connection closes before lndmon finishes writing the
response, the log fills with dozens of identical lines:
promhttp ... http.go:192: error encoding and sending metric family: write tcp 10.0.0.166:8990->10.0.1.221:39822: write: broken pipe
promhttp ... http.go:192: error encoding and sending metric family: write tcp 10.0.0.166:8990->10.0.1.221:39822: write: broken pipe
... (repeated once per metric family)
Why it happens
promhttp encodes the scrape response one metric family at a time. lndmon
configures the handler with promhttp.ContinueOnError and a plain log.Logger
(collectors/prometheus.go, the HandlerOpts/errorLogger around lines
198-212). Under ContinueOnError, each family that fails to write is logged
individually and the loop continues to the next family. So once the client
connection is gone, every remaining Encode call fails with broken pipe and
emits its own line — the count is just how many metric families were left to
write.
The connection is gone for one of two reasons:
- Slow scrape overruns Prometheus's
scrape_timeout. This is the standalone
trigger and needs no crash. lndmon's default --lnd.rpctimeout is 30s
(config.go), longer than a typical Prometheus scrape_timeout of ~10s. If
any collector RPC is slow, the scraper gives up and closes the connection
while lndmon is still gathering/encoding, so the eventual write hits a dead
socket.
- lndmon shuts down mid-scrape. If the exporter is torn down while a scrape
is in flight, the same broken-pipe burst appears. Because the burst is emitted
during shutdown, it follows the log line reporting why lndmon exited and can
bury it under dozens of identical entries, making the actual cause of the
shutdown hard to find.
Related issues
Summary
When a Prometheus scrape connection closes before lndmon finishes writing the
response, the log fills with dozens of identical lines:
Why it happens
promhttpencodes the scrape response one metric family at a time. lndmonconfigures the handler with
promhttp.ContinueOnErrorand a plainlog.Logger(
collectors/prometheus.go, theHandlerOpts/errorLoggeraround lines198-212). Under
ContinueOnError, each family that fails to write is loggedindividually and the loop continues to the next family. So once the client
connection is gone, every remaining
Encodecall fails withbroken pipeandemits its own line — the count is just how many metric families were left to
write.
The connection is gone for one of two reasons:
scrape_timeout. This is the standalonetrigger and needs no crash. lndmon's default
--lnd.rpctimeoutis 30s(
config.go), longer than a typical Prometheusscrape_timeoutof ~10s. Ifany collector RPC is slow, the scraper gives up and closes the connection
while lndmon is still gathering/encoding, so the eventual write hits a dead
socket.
is in flight, the same broken-pipe burst appears. Because the burst is emitted
during shutdown, it follows the log line reporting why lndmon exited and can
bury it under dozens of identical entries, making the actual cause of the
shutdown hard to find.
Related issues
/metricslatency, the condition that triggers the slow-scrapepath above. This issue covers the log-flood symptom; those track the latency
itself.
promhttplog-noise line (superfluous response.WriteHeader);sibling logging-noise report, likely already addressed by the client_golang
upgrade.