PMM-15228 Enrich AuthServer with performance metrics

[maxkondr]

Ticket number: PMM-15228

Feature build: Percona-Lab/pmm-submodules#4484

This pull request adds detailed Prometheus metrics to the AuthServer component in order to monitor authentication request flows, cache usage, and request durations. It also ensures that metrics are accurately labeled, especially in error scenarios, and improves code clarity around request path handling.

Prometheus Metrics Integration:

• Added a new authMetrics struct to AuthServer that defines and registers Prometheus counters and histograms for tracking authentication requests, cache hits/misses, Grafana backend requests, cache size, and operation durations. (managed/services/grafana/auth_server.go)
• Implemented the prom.Collector interface for AuthServer, allowing it to be registered with Prometheus and report its metrics. (managed/services/grafana/auth_server.go)
• Registered the AuthServer as a Prometheus collector in the main application entrypoint. (managed/cmd/pmm-managed/main.go)

Metrics Collection in Auth Flow:

• Updated the authentication flow to increment the appropriate counters and observe durations for total request time, Grafana backend requests, database lookups, cache hits, and cache misses. (managed/services/grafana/auth_server.go) [1] [2] [3]
• Ensured that metrics for failed requests (e.g., bad requests) use cleaned and normalized route labels for consistency and security. (managed/services/grafana/auth_server.go)

Testing and Path Handling Improvements:

• Added tests to verify that metrics for bad requests use the cleaned route or fall back to the raw route if cleaning fails. (managed/services/grafana/auth_server_test.go)
• Centralized and clarified the use of cleaned request paths by ensuring path normalization occurs early and is consistently used throughout the authentication logic. (managed/services/grafana/auth_server.go) [1] [2]

These changes provide better observability into authentication operations and improve the maintainability and correctness of metrics reporting.

[codecov]

Codecov Report

❌ Patch coverage is 62.01550% with 49 lines in your changes missing coverage. Please review.
✅ Project coverage is 43.39%. Comparing base (31318c7) to head (607d9e6).
⚠️ Report is 65 commits behind head on main.

Files with missing lines	Patch %	Lines
managed/services/grafana/auth_server.go	62.50%	45 Missing and 3 partials ⚠️
managed/cmd/pmm-managed/main.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5670      +/-   ##
==========================================
- Coverage   43.59%   43.39%   -0.20%     
==========================================
  Files         415      433      +18     
  Lines       43134    35106    -8028     
  Branches        0      592     +592     
==========================================
- Hits        18804    15236    -3568     
+ Misses      22454    18322    -4132     
+ Partials     1876     1548     -328     

Flag	Coverage Δ
admin	34.94% <ø> (+0.15%)	⬆️
agent	?
managed	45.03% <62.01%> (+2.05%)	⬆️
unittests	41.29% <ø> (?)
vmproxy	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[maxkondr]

Another general PromQL gotcha is the use of rate(sum(...)), while the best practice is sum(rate(...)). Let's update the formulas.

updated

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (2)

managed/services/grafana/auth_server.go:588

• extractOriginalRequest now fails hard when cleanPath can't unescape the original URI (e.g., invalid % escapes), which makes ServeHTTP return 400. NGINX auth_request treats any non-401/403 response as 500, so malformed paths can turn into server errors. Consider falling back to the raw path (without query) instead of returning an error here.

	cleanedOrigURI, err := cleanPath(origURI)
	if err != nil {
		return fmt.Errorf("failed to unescape path %q: %w", origURI, err)
	}

managed/services/grafana/auth_server.go:399

• The route label value is currently the full (cleaned) request path (req.URL.Path). Even without query parameters this can still be very high-cardinality (IDs, UIDs, etc.), which risks unbounded time-series growth and increased memory/CPU in Prometheus and in-process metric vectors. A bounded label (e.g., the matched rule prefix returned by resolveRule) would avoid this.

		status := httpStatusForAuthError(authErr.code)
		s.incAuthRequests(req.Method, req.URL.Path, status)
		s.returnError(rw, status, m, l)

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

managed/services/grafana/auth_server.go:374

• In the bad-request branch, if cleanPath(route) fails you fall back to using the raw (user-controlled) X-Original-Uri value as the "route" label. That allows an attacker to generate unbounded label cardinality (and in-process memory growth inside CounterVec) by sending many distinct invalid-escape paths. Consider using a bounded fallback label (e.g. "/invalid" or "") and/or capping route length when cleaning fails, rather than emitting the raw value.

		route := req.Header.Get("X-Original-Uri")
		if route == "" {
			route = req.URL.Path
		} else if i := strings.IndexByte(route, '?'); i >= 0 {
			route = route[:i]
		}
		cleaned, cleanErr := cleanPath(route)
		if cleanErr == nil {
			route = cleaned
		}

managed/services/grafana/auth_server.go:748

• New metrics behavior is added for cache hits/misses and Grafana backend request outcomes, but only the bad-request route labeling is covered by tests. Adding unit tests around getAuthUser/retrieveRole to assert cache hit/miss counters and Grafana status_code counters would help prevent regressions (e.g., ensuring miss is recorded on stale/absent cache and Grafana counters increment on success/clientError/unknown error).

	if ok && time.Since(item.created) < cacheInvalidationInterval {
		s.incCacheHit()
		return &item.u, nil
	}

	s.incCacheMiss()
	return s.retrieveRole(ctx, hash, authHeaders, l)