docs: document exported metric names and coverage gaps

[HayimShaul]

Fixes #1745

Problem

Panurus exports 54 metrics and documented none of them under the name Prometheus actually serves.

Nothing in the SDK spells those names out. The fully-qualified name is assembled at registration time
from the Go package of the caller, and the TMS-scoped provider in token/core/common/metrics adds a
stack frame — so metrics created through it are exported under that package, not their own. Docs
written from the bare Name field therefore list names no query matches. That is why #1749 was closed.

What this PR does

1. Documents the exported names — docs/development/metrics.md: how a name is derived, one table
per subsystem (exported name, type, labels, meaning, source), PromQL examples, and a ranked
## Coverage gaps section on what a node still cannot report.

2. Guards the page — token/services/metricsdoc instantiates all 16 metrics constructors the way
production wires them, reads the names back out of a Prometheus registry, and checks them against
testdata/metrics.golden and against the page in both directions. It also pins the wiring the names
depend on: the two token drivers are asserted to be the only places a TMS-scoped provider is built, so
dropping that wrapper fails the build instead of silently renaming 21 metrics.

3. Adds a Grafana dashboard — docs/monitoring/grafana/token-sdk.json, revived from #1749 with the
correct names, plus a row for the four identity_* metrics that had no panel. All 54 metrics covered,
19 panels over 9 rows. Its queries are guarded too: every metric must be registered, every name must
carry its package prefix, every label filtered or grouped on must be declared, every $variable must
exist. These are the failures Grafana renders as "No data" rather than as an error.

4. Corrects docs/drivers/metrics.md, which listed declared names as if they were queryable, and
makes docs/development/monitoring.md the entry point.

NewMetrics/NewClientMetrics are exported in the auditor, certification and Fabric-X queue packages
so the guard can build their instrumentation without constructing the surrounding service.

Verification

• All 54 names read out of a live Prometheus registry, not transcribed from source.
• Each guard check proven to fire by mutating the source: renamed metric, dropped TMS wrapper, stale
  wiring entry, truncated name in the page, absent label, undeclared variable.
• Run against docs: add Token SDK metrics reference and identify coverage gaps #1749's original dashboard, the query guard reports 53 findings — that PR's defect would
  have failed the build.
• make checks and make lint exit 0; package green under -count=3 -race -shuffle=on.

Known limitation

The dashboard guard checks names, not rendering. PromQL is not parsed (it would need a new
dependency, the exprs contain Grafana variables that are not valid PromQL, and it is the one failure
mode Grafana surfaces itself), and the file has not been imported into a live Grafana against a live
Prometheus — so panel presentation is unverified.

Follow-up

The metric-hygiene items in the gap analysis (package attribution, stuttering names, missing
network/channel/namespace labels) all change exported names, so they belong in a single
deliberate follow-up with a release note rather than here.

[github-actions]

📊 Token Validation Benchmark

Comparison of this PR against the base branch. 🟢 improvement · 🔴 regression · ➖ within ±1.0% noise.

Variant	Benchmark	Params	Workers	TPS (base → PR)	Δ TPS
csp	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	117 → 117	➖ -0.1%
csp	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	170 → 172	➖ +0.8%
ipa	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	90 → 91	➖ +0.1%
ipa	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	76 → 76	➖ +0.2%

[github-actions]

📊 Token Validation Benchmark

Comparison of this PR against the base branch. 🟢 improvement · 🔴 regression · ➖ within ±1.0% noise.

Variant	Benchmark	Params	Workers	TPS (base → PR)	Δ TPS
csp	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	117 → 117	➖ +0.1%
csp	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	104 → 104	➖ +0.0%
ipa	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	70 → 70	➖ +0.0%
ipa	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	81 → 81	➖ +0.2%

[github-actions]

📊 Token Validation Benchmark

Comparison of this PR against the base branch. 🟢 improvement · 🔴 regression · ➖ within ±1.0% noise.

Variant	Benchmark	Params	Workers	TPS (base → PR)	Δ TPS
csp	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	117 → 118	➖ +0.1%
csp	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	103 → 104	➖ +0.2%
ipa	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	78 → 78	➖ -0.0%
ipa	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	76 → 76	➖ -0.2%

[AkramBitar]

Reviewed the docs against the code: the content is accurate. Every metric name, label value and scope claim matches production wiring today (traced the NewTMSProvider wrapper in token/core/{fabtoken,zkatdlog}/v1/driver/driver.go through ws.go into identity/role/idemix, and the DI-container providers for ttx / auditor / sherdlock / certifier / fabricx). All 15 files that declare metric opts are covered by the group list, the 54 table rows match the 54 golden lines, the monitoring.md URL change fixes a real 404, and go vet / gofmt / goimports / misspell / golangci-lint plus go test -count=3 -race -shuffle=on are clean.

The comments below are all about the new reference_test.go guard rather than the documentation. Two of them I'd suggest addressing before merge:

• The guard cannot detect the one change that would actually invalidate the whole page (the scope field is unverified metadata).
• The unconditional histogram-suffix trim will fail on a correct page as soon as someone adds a metric whose name ends in _count, _sum or _bucket.

The remaining four are polish — confusing failure output, a gap in the reverse check, a dead field, and a global that is only safe while the tests stay sequential.

[AkramBitar]

Follow-up: bring back token-sdk.json, with the names this PR pins.

#1749 shipped a Grafana dashboard next to its metrics doc — https://github.com/LFDT-Panurus/panurus/pull/1749/files (permalink) — 17 panels over 9 rows, with network/channel/namespace/method template variables. The panel layout and label filters are sound; the one defect is the one I flagged there: #1749 (comment) — every expr used the bare option name from the Go source, so no query matched anything in a real Prometheus.

That PR closed unmerged, and docs/monitoring/ is in neither this PR nor main, so the dashboard went with it. Worth reviving, because this PR's testdata/metrics.golden is exactly the input needed to fix it — the substitution is mechanical:

expr in #1749's dashboard	Correct name
issue_service_operations_total	panurus_core_common_metrics_issue_service_operations_total
endorsed_transactions	panurus_services_ttx_endorsed_transactions
finality_listener_confirmed_total	panurus_services_ttx_finality_finality_listener_confirmed_total
ttx_envelope_sent_total	panurus_services_utils_json_session_ttx_envelope_sent_total
auditor_audit_duration_seconds	panurus_services_auditor_auditor_audit_duration_seconds
unspent_tokens_invocations	panurus_services_selector_sherdlock_unspent_tokens_invocations
certified_tokens	panurus_services_certifier_interactive_certified_tokens
cache_level	panurus_core_common_metrics_cache_level
finality_queue_pending_events	panurus_services_network_fabricx_finality_queue_finality_queue_pending_events

Two things beyond the rename:

• the four panurus_core_common_metrics_identity_* signer-resolution metrics postdate docs: add Token SDK metrics reference and identify coverage gaps #1749 and have no panel yet;
• the metricsdoc package added here could guard the dashboard too — extract every metric name from the panel expr fields and assert each one is registered. Without that, the queries can drift out of sync again exactly the way docs: add Token SDK metrics reference and identify coverage gaps #1749's did, and the failure mode is a silent "No data" rather than an error.

Do we want the dashboard in this PR, or as a follow-up? I have no strong preference. Adding it here keeps the dashboard and the names it depends on in one changeset, which is nice for review — but this PR is already sizeable and purely documentation, so a separate PR is equally reasonable. A follow-up would need to be based on this branch rather than main, since the metricsdoc guard package doesn't exist on main yet. Happy either way — let me know which you prefer and I can open an issue to track it.

Dismissing my own review: all six findings are fixed in 83d60d7 and each thread is answered and resolved.

Not converting this to an approval on purpose — I wrote most of the code now (the guard hardening and the Grafana dashboard), so approving it would be a self-review. This PR still wants a fresh pair of eyes from another maintainer.

[AkramBitar]

@HayimShaul will be out of the office for the next two weeks. I'll be covering for him during that time, so I won't be able to continue reviewing this PR. Someone else will need to take over the review.