Skip to content

[inputs.prometheus] kubernetes informer got stuck on reload #18303

@redbaron

Description

@redbaron

Relevant telegraf.conf

[[inputs.prometheus]]
monitor_kubernetes_pods = true
metric_version = 1
monitor_kubernetes_pods_method = "settings+annotations"
monitor_kubernetes_pods_port = 7354
pod_scrape_interval = 60
content_length_limit = "4096KB"

Logs from Telegraf

2026-02-04T14:08:41.094Z | 2026-02-04T14:08:41Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"", Flush Interval:10s
2026-02-04T14:08:41.097Z | 2026-02-04T14:08:41Z W! [inputs.prometheus::myapp] Error while attempting to watch pod: handler {0x452a5a0 0x452a380 0x452a320} was not added to shared informer because it has stopped already

System info

Telegraf 1.37

Docker

No response

Steps to reproduce

  1. have prometheus plugin in monitor_kubernetes_pods = true mode
  2. (possibly) have many (we have hundreds) pods that plugin would be targeting
  3. have telegraf with --watch-config poll (I suspect manual reloading with a signal will trigger it too)

Expected behavior

reload succeeds and metric collection continues

Actual behavior

plugin instances with kubernetes connection stop collecting metrics

Additional info

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugunexpected problem or unintended behavior

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions