docs: document exported metric names and coverage gaps

The SDK never spells out the metric names Prometheus exports. The
fully-qualified name is assembled at registration time from the Go package
of the caller, and the TMS-scoped provider in token/core/common/metrics adds
a stack frame, so metrics created through it are exported under that
package rather than their own. Documentation written from the bare Name
field of the options therefore lists names no query matches.

Add docs/development/metrics.md: the derivation rules, every metric under
the name Prometheus actually serves, its type, labels, meaning and source,
example queries, and a ranked analysis of what a node still cannot report.

Guard the page with token/services/metricsdoc, which instantiates every
metrics constructor the way production wires it, reads the resulting names
back out of a Prometheus registry, and compares them against a golden file
and against the page itself. Renaming, adding or relocating a metric fails
the test until the documentation is updated.

Which provider a constructor receives is as much a part of the exported
name as the opts are, so it is checked rather than assumed. The token
drivers are pinned as the only files in the repository that build a
TMS-scoped provider, and as building it from the container's provider, so
the scope of each group follows by exhaustion; dropping the wrapper fails
the test instead of quietly renaming twenty-one metrics. Every production
call site a group claims must also still contain that call.

Two ways the guard could reject a correct page are fixed: the PromQL suffix
fold now applies only onto registered histogram families, so a metric whose
own name ends in _count, _sum or _bucket survives it; and the reverse check
uses an explicit prose allowlist rather than accepting any documented name
that is a prefix of a registered one, which had let truncations through.
Each group is instantiated against its own provider and registry, so the
declared and exported metrics cannot be mispaired by the provider's
deduplication, and a genuine clash is reported by name.

Add the Grafana overview dashboard, revived from #1749 with the names this
change pins. That dashboard closed unmerged because every one of its 51
panel queries and 4 variable queries named the bare Name field from the Go
source and so matched nothing. The substitution here is derived from the
golden file rather than transcribed. A row is added for the four identity
signer-resolution metrics that had no panel, so every exported metric is
now covered.

The dashboard's queries are guarded too. Each check covers a failure
Grafana does not report as an error - it renders "No data", which is
indistinguishable from an idle node: every metric named must be registered,
every name must carry its package prefix, every label filtered or grouped
on must be declared by that metric, and every variable must be a Grafana
built-in or declared by the dashboard. Run against #1749's original file,
the guard reports 53 findings. PromQL syntax is deliberately not parsed: it
would need a new dependency, the exprs contain Grafana variables that are
not valid PromQL, and it is the one failure mode Grafana surfaces itself.
The guard checks names, not rendering; the file has not been imported into
a live Grafana, so panel presentation is unverified.

Export NewMetrics/NewClientMetrics in the auditor, certification and
Fabric-X queue packages so the guard can build their instrumentation
without constructing the surrounding service.

Correct docs/drivers/metrics.md, which listed declared names as if they
were queryable, and make docs/development/monitoring.md the entry point.

Signed-off-by: Hayim.Shaul@ibm.com <hayimsha@fhe03.vpc.cloud9.ibm.com>
Signed-off-by: AkramBitar <akram@il.ibm.com>
Signed-off-by: Hayim.Shaul@ibm.com <hayimsha@fhe03.vpc.cloud9.ibm.com>

Author: Hayim.Shaul@ibm.com

.golangci.yml

@@ -120,6 +120,7 @@ linters:
         - github.com/hyperledger/fabric-lib-go/common/metrics.Gauge
         - github.com/hyperledger/fabric-lib-go/common/metrics.Histogram
         - github.com/hyperledger-labs/fabric-smart-client/platform/view/services/metrics.Gauge
+        - github.com/hyperledger-labs/fabric-smart-client/platform/view/services/metrics.Histogram
         - github.com/hyperledger-labs/fabric-smart-client/platform/common/driver.ConfigService
         - github.com/hyperledger-labs/fabric-smart-client/integration/nwo/api.ViewClient
         - github.com/hyperledger-labs/fabric-smart-client/integration/nwo/api.Platform

docs/README.md

@@ -35,6 +35,8 @@ If you are developing *using* Panurus or contributing *to* Panurus, check out th
 *   [General Guidelines](development/general.md)
 *   [Idiomatic Go](development/idiomatic.md)
 *   [Testing](development/testing.md)
+*   [Monitoring](development/monitoring.md), the [Metrics Reference](development/metrics.md) and the
+    [Grafana dashboards](monitoring/grafana/README.md)
 *   [Test & Benchmark Profiler](../cmd/profiler/README.md)
 
 ## Evolution

docs/development/development.md

@@ -8,6 +8,8 @@ This page contains link to the development guidelines and more.
 - [Development Tools](./tools.md)
 - [Linting](./linting.md)
 - [Monitoring](./monitoring.md)
+- [Metrics Reference](./metrics.md)
+- [Grafana dashboards](../monitoring/grafana/README.md)
 - [Mock Files Generation](./mock.md)
 - [AI Agents Best Practices](./ai_agents.md)
 - [Tools: tokengen](./tokengen.md)

docs/development/metrics.md

@@ -1,7 +1,19 @@
 # Monitoring
 
-We adopt the monitoring infrastructure provided by the [`Fabric Smart Client`](https://github.com/hyperledger-labs/fabric-smart-client/blob/main/docs/platform/view/monitoring.md).
+We adopt the monitoring infrastructure provided by the [`Fabric Smart Client`](https://github.com/hyperledger-labs/fabric-smart-client/blob/main/docs/platform/view/services/monitoring.md).
 
 We use the following two methods to monitor the performance of the application:
 * **Metrics** provide an overview of the overall system performance using aggregated results, e.g. total requests, requests per second, current state of a variable, average duration, percentile of duration
 * **Traces** help us analyze single requests by breaking down their lifecycles into smaller components
+
+## Where to look next
+
+* [Metrics Reference](./metrics.md) — every metric Panurus exports, under the exact name Prometheus
+  serves it, plus how those names are derived, example queries, and the current coverage gaps.
+* [Grafana dashboards](../monitoring/grafana/README.md) — an importable overview dashboard covering
+  every exported metric, and what to run after editing a query.
+* [Driver Metrics](../drivers/metrics.md) — how the driver service wrappers are built and which
+  methods they instrument.
+* [Fabric Smart Client monitoring](https://github.com/hyperledger-labs/fabric-smart-client/blob/main/docs/platform/view/services/monitoring.md)
+  — the platform metrics and traces Panurus inherits (views, sessions, gRPC, process), and how to
+  enable the Prometheus endpoint and the tracing exporter.

docs/development/monitoring.md

@@ -20,6 +20,14 @@ implementation, and records three metrics per method invocation:
 | **Histogram** (`*_duration_seconds`) | Execution duration of each call |
 | **Counter** (`*_errors_total`) | Total number of calls that returned an error |
 
+The metric names in this page are the **declared** names, as written in the wrapper sources.
+Prometheus exports them under a prefix derived from the package that creates them; because the driver
+wrappers receive a TMS-scoped provider, every metric below is exported as
+`panurus_core_common_metrics_<declared name>` — for instance `issue_service_operations_total` is
+queried as `panurus_core_common_metrics_issue_service_operations_total`. See
+[Metrics Reference](../development/metrics.md) for the derivation rules and the exported names of
+every metric in the SDK.
+
 All metrics carry four labels for multi-TMS filtering:
 
 | Label | Description |
@@ -136,27 +144,12 @@ Metrics emitted:
 
 ## Metric Reference
 
-The full list of metrics emitted by the driver wrappers:
-
-| Metric Name | Type | Description |
-|-------------|------|-------------|
-| `issue_service_operations_total` | Counter | Total `IssueService` method invocations |
-| `issue_service_duration_seconds` | Histogram | Duration of `IssueService` method calls |
-| `issue_service_errors_total` | Counter | Total `IssueService` method errors |
-| `transfer_service_operations_total` | Counter | Total `TransferService` method invocations |
-| `transfer_service_duration_seconds` | Histogram | Duration of `TransferService` method calls |
-| `transfer_service_errors_total` | Counter | Total `TransferService` method errors |
-| `auditor_service_operations_total` | Counter | Total `AuditorService` method invocations |
-| `auditor_service_duration_seconds` | Histogram | Duration of `AuditorService` method calls |
-| `auditor_service_errors_total` | Counter | Total `AuditorService` method errors |
-| `tokens_service_operations_total` | Counter | Total `TokensService` method invocations |
-| `tokens_service_duration_seconds` | Histogram | Duration of `TokensService` method calls |
-| `tokens_service_errors_total` | Counter | Total `TokensService` method errors |
-| `tokens_upgrade_service_operations_total` | Counter | Total `TokensUpgradeService` method invocations |
-| `tokens_upgrade_service_duration_seconds` | Histogram | Duration of `TokensUpgradeService` method calls |
-| `tokens_upgrade_service_errors_total` | Counter | Total `TokensUpgradeService` method errors |
-
-All metrics use labels: `network`, `channel`, `namespace`, `method`.
+The exported names, types and labels of the fifteen driver metrics are listed in
+[Metrics Reference — Driver services](../development/metrics.md#driver-services). That page is kept in
+step with the code by `token/services/metricsdoc`, so it is the authoritative list; this page describes
+only how the wrappers work and which methods they instrument.
+
+All driver metrics use labels: `network`, `channel`, `namespace`, `method`.
 
 ## Source
 

docs/drivers/metrics.md

@@ -0,0 +1,58 @@
+# Grafana dashboards
+
+## `panurus.json` — Panurus Overview
+
+A single overview dashboard covering every metric Panurus exports: 20 panels across 9 rows, one row per
+subsystem (driver services, transaction lifecycle, finality listener, envelope sessions, auditor, token
+selection, certification and identity caches, signer resolution and cache provisioning, Fabric-X finality
+queue).
+
+### Import
+
+1. Grafana → **Dashboards** → **New** → **Import** → *Upload JSON file*.
+2. Pick the Prometheus data source that scrapes the node's metrics endpoint when prompted for
+   `DS_PROMETHEUS`.
+
+The dashboard declares four template variables — `network`, `channel`, `namespace` and `method` — whose
+values are discovered with `label_values` against
+`panurus_core_common_metrics_transfer_service_operations_total`. A node that has never issued or
+transferred a token exports no series for that metric, so the pickers stay empty until the first
+transaction; the unfiltered panels still work.
+
+Requires Grafana 9.0 or later (`schemaVersion` 37).
+
+### Not covered
+
+- **FSC platform metrics** (views, sessions, gRPC, process) — these come from
+  [Fabric Smart Client](https://github.com/hyperledger-labs/fabric-smart-client/blob/main/docs/platform/view/services/monitoring.md)
+  and are exported under `fsc_*`, not `panurus_*`.
+- **Traces.** The dashboard is metrics-only.
+- Panels are built from metric *names*, so they show what a node reports, not whether the reported
+  numbers are healthy: there are no thresholds or alert rules here.
+
+### Changing it
+
+Every query in this file is checked by `token/services/metricsdoc`, which asserts that
+
+- each metric a query names is one the SDK registers, under the name Prometheus actually exports;
+- each metric name carries its package prefix, so a bare `Name` from the Go source fails the build
+  rather than rendering an empty panel;
+- each label a query filters or groups on is declared by the metric it is applied to;
+- each `$variable` a query interpolates is either a Grafana built-in or declared in this dashboard.
+
+These are the failure modes a dashboard cannot report itself: Grafana does not error on an unknown
+metric or an absent label, it renders **No data**, which is indistinguishable from an idle node. An
+earlier version of this dashboard ([#1749](https://github.com/LFDT-Panurus/panurus/pull/1749)) had every
+one of its 51 panel queries and 4 variable queries written against bare option names from the Go source,
+so not one of them matched a series; it closed unmerged.
+
+So: after editing a query, run
+
+```bash
+go test ./token/services/metricsdoc/...
+```
+
+If you add a panel for a metric that does not exist yet, add the metric first — see
+[Metrics Reference](../../development/metrics.md) for the exported names and
+[`testdata/metrics.golden`](../../../token/services/metricsdoc/testdata/metrics.golden) for the
+machine-readable list.