Panic in chrony input plugin

[IngmarStein]

Relevant telegraf.conf

[[inputs.chrony]]
  server = "udp://[::1]:323"
  dns_lookup = true
  metrics = ["activity", "tracking", "sources", "sourcestats"]

Logs from Telegraf

2026-02-05T16:22:40Z E! FATAL: [inputs.chrony] panicked: runtime error: index out of range [256] with length 256, Stack:
goroutine 116018 [running]:
github.com/influxdata/telegraf/agent.panicRecover(0x140017f4600)
     github.com/influxdata/telegraf/agent/agent.go:1202 +0x60
panic({0x10a704a60?, 0x14001bde2b8?})
     runtime/panic.go:783 +0x120
encoding/binary.(*encoder).uint8(...)
     encoding/binary/binary.go:792
encoding/binary.(*encoder).value(0x14001cb5ce0, {0x109572b20?, 0x14003ada2c0?, 0x1090eeab3?})
     encoding/binary/binary.go:953 +0x89c
encoding/binary.(*encoder).value(0x14001cb5ce0, {0x1096ebfc0?, 0x14003ada1dc?, 0x10129cc2c?})
     encoding/binary/binary.go:919 +0x804
encoding/binary.(*encoder).value(0x14001cb5ce0, {0x10a506300?, 0x14003ada1c0?, 0x1005522cc?})
     encoding/binary/binary.go:928 +0x714
encoding/binary.Write({0x10ae62ac0, 0x14001be1440}, {0x10af3ec40, 0x110aacd20}, {0x109bc6780, 0x14003ada1c0})
     encoding/binary/binary.go:431 +0x254
github.com/facebook/time/ntp/chrony.(*Client).Communicate(0x14000410c18, {0x10ae9fec8, 0x14003ada1c0})
     github.com/facebook/time@v0.0.0-20250903103710-a5911c32cdb9/ntp/chrony/client.go:38 +0x84
github.com/influxdata/telegraf/plugins/inputs/chrony.(*Chrony).gatherSources(0x140015edd90, {0x10af57dc0, 0x1400073e100})
     github.com/influxdata/telegraf/plugins/inputs/chrony/chrony.go:417 +0x1a0
github.com/influxdata/telegraf/plugins/inputs/chrony.(*Chrony).Gather(0x140015edd90, {0x10af57dc0, 0x1400073e100})
     github.com/influxdata/telegraf/plugins/inputs/chrony/chrony.go:170 +0x130
github.com/influxdata/telegraf/models.(*RunningInput).Gather(0x140017f4600, {0x10af57dc0, 0x1400073e100})
     github.com/influxdata/telegraf/models/running_input.go:264 +0x210
github.com/influxdata/telegraf/agent.(*Agent).gatherOnce.func1()
     github.com/influxdata/telegraf/agent/agent.go:590 +0x50
created by github.com/influxdata/telegraf/agent.(*Agent).gatherOnce in goroutine 186
     github.com/influxdata/telegraf/agent/agent.go:588 +0xb4

goroutine 1 [sync.WaitGroup.Wait, 86 minutes]:
sync.runtime_SemacquireWaitGroup(0x100529930?, 0x10?)
     runtime/sema.go:114 +0x38
sync.(*WaitGroup).Wait(0x140003fb760)
     sync/waitgroup.go:206 +0xa8
git
2026-02-05T16:22:40Z E! PLEASE REPORT THIS PANIC ON GITHUB with stack trace, configuration, and OS information: https://github.com/influxdata/telegraf/issues/new/choose

System info

Telegraf 1.37.2, chrony 4.8

Docker

No response

Steps to reproduce

1. Add a chrony input to telegraf.conf
2. Start chronyd and telegraf
3. Wait till the panic occurs
   I'm unable to reproduce this on demand. I just see the infrequent panics in the log files. It can take a long while until it happens.

Expected behavior

No panic.

Actual behavior

Telegraf panics

Additional info

I've reported this previously as #17453 and #17757, but it looks like the root cause it not yet fixed. I've switched from a unix socket to UDP in the meantime to see if that makes a difference, but it doesn't.

Mentioning here just for completeness as I don't know whether it's related or not, but I occasionally see failures when writing batches to outputs.postgres like this (referring to various tables like temp, disk, diskio, http_response and more):

2026-02-05T13:48:38Z E! [outputs.postgresql] PG CopyFrom - map[columnNames:[time host sensor temp] err:ERROR: invalid byte sequence for encoding "UTF8": 0x00 (SQLSTATE 22021) pid:57850 tableName:[public temp] time:604.292µs         ]
2026-02-05T13:48:38Z E! [outputs.postgresql] write error (permanent, dropping sub-batch): ERROR: invalid byte sequence for encoding "UTF8": 0x00 (SQLSTATE 22021)

When running Telegraf in test mode, I never saw NUL bytes in the output and I now filter them with a regex processor, so maybe there is some memory corruption happening in the Telegraf process.

[skartikey]

@IngmarStein Thanks for reporting this again. I've investigated all three reports (#17453, #17757, this one) and the mutex fix from PR #17765 does not address the root cause.

The panic shows encoding/binary.Write allocating a 256-byte buffer, but all chrony request structs are 416–420 bytes; no code path can produce that size. Combined with other evidence across the three issues:

• NUL byte corrupting the plugin name in logs: [inputs.^@hrony] (Panic in chrony input plugin #17453)
• NUL bytes in PostgreSQL output data (this issue)
• fatal error: traceback did not unwind completely - Go runtime stack corruption (Panic in chrony input plugin #17757)

This points to memory corruption in the process rather than a bug in the chrony plugin or the facebook/time library.

To help narrow down the source, could you try:

1. Run https://support.apple.com/en-us/102550 to rule out a hardware memory issue
2. Run with GODEBUG=invalidptr=1 to surface corruption earlier
3. Test with a minimal config - only the chrony input and a file output - to see if another plugin (e.g., the PostgreSQL output which uses CGo) might be involved

[IngmarStein]

@skartikey maybe it's a side-effect of #18309. This could cause spooky actions at a distance almost everywhere in the Telegraf process…

[skartikey]

Good catch, the gopsutil CFString bug (shirou/gopsutil#1992) is very likely the root cause here. On macOS, CFStringGetLength returns UTF-16 code units instead of UTF-8 byte count, causing buffer underallocation in CGo. That heap corruption would explain both the spurious NUL bytes across plugins and the chrony panic (corrupted type metadata -> wrong buffer size in encoding/binary.Write).

PR #18293 bumps gopsutil to v4.26.1 which includes this fix. Once merged, it should resolve this as well.

[IngmarStein]

It's funny that when I finally thought to have found the root cause in gopsutil after struggling with the instability for a year that someone had already posted the fix just last month :) I hope you're able to sort the CI issues with the version bump because I think this is a critical bug fix for everyone on macOS who uses any of the sensors from gopsutil.