Optimizing LoadSources to support "Soft" configuration updates and to avoid full restart (mentioned as TODO) #1301
nadashaban11
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I’ve been looking into the TODO in reaper.go about optimizing LoadSources to avoid full restarts on minor configuration changes (adding/deleting one metric or changing a single interval). I’d like to share my logic and get your thoughts before I finalize the implementation.
Current Behavior:
Currently, any change in a source (like adding/deleting one metric or changing a single interval) triggers a full restart of the connection pool and all metric gatherers, which is resource-intensive.
My Solution:
I'm planning to split the update logic into two categories:
1- Hard Change: Changes to
ConnStr,Name,Group,kind,IsEnabled, andOnlyIfMaster. These will still trigger a full restart (close connection, cancel all workers).2- Soft Change: Changes to Metrics, Intervals, CustomTags, or Presets.
Instead of closing the pool, we only cancel() the context of the specifically affected metrics either deleted metrics or modified metrics,then the
Reaperloop naturally picks up "new" or "restarted" metrics in the next iteration because theirdbMetrickey will no longer exist in ther.cancelFuncsmap after a soft cancel.For example I've implemented a helper like this in
sources/types.go.My Question:
Is this approach is acceptable? I want to ensure that keeping the connection pool open during these changes doesn't conflict with any internal state management I might have missed. If this looks okay to you, I'm ready to finalize the code and open a PR.
I want your opinion about making
CustomTagsandPresetMetricsas a soft change, then all metrfirstics will be cancelled/restarted but without closing the connection.Beta Was this translation helpful? Give feedback.
All reactions