metrics: introduce custom registry

[mohammed90]

Module developers who add custom metrics are used to wrapping the registration with a sync.Once to avoid duplicate registration panic and control the registration. However, the flow of caddy reload doesn't play nice with this idiom.

Since the introduction of registry-per-context, we've told some users to ignore the duplicate registration error if received. However, checking for this particular error is annoying. I wonder if owning the sync.Once and tracking the registration is better. I don't know if this is the best flow.

[mholt]

I don't fully understand this (the metrics stuff in general) but it looks good to me -- no obvious red flags 😅

[mohammed90]

The key I'm using to track uniqueness isn't optimal because a single Caddy module may register multiple collectors. Extracting the unique identifiers using the Describe method is hard because theDesc type only has String method, and it's hard to extract both fqName and the const labels (unique combo) from the string. So I changed it to just swallow the duplicate registration error but bubble the rest. I doubt it's best-practice but cannot think of a better way.

[mholt]

That does sound like a tricky one. That's probably fine? We can see if it causes any issues but maybe it's not too likely, or is at least preventable.