Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@ linters:
- github.com/hyperledger/fabric-lib-go/common/metrics.Gauge
- github.com/hyperledger/fabric-lib-go/common/metrics.Histogram
- github.com/hyperledger-labs/fabric-smart-client/platform/view/services/metrics.Gauge
- github.com/hyperledger-labs/fabric-smart-client/platform/view/services/metrics.Histogram
- github.com/hyperledger-labs/fabric-smart-client/platform/common/driver.ConfigService
- github.com/hyperledger-labs/fabric-smart-client/integration/nwo/api.ViewClient
- github.com/hyperledger-labs/fabric-smart-client/integration/nwo/api.Platform
Expand Down
2 changes: 2 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ If you are developing *using* Panurus or contributing *to* Panurus, check out th
* [General Guidelines](development/general.md)
* [Idiomatic Go](development/idiomatic.md)
* [Testing](development/testing.md)
* [Monitoring](development/monitoring.md), the [Metrics Reference](development/metrics.md) and the
[Grafana dashboards](monitoring/grafana/README.md)
* [Test & Benchmark Profiler](../cmd/profiler/README.md)

## Evolution
Expand Down
2 changes: 2 additions & 0 deletions docs/development/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ This page contains link to the development guidelines and more.
- [Development Tools](./tools.md)
- [Linting](./linting.md)
- [Monitoring](./monitoring.md)
- [Metrics Reference](./metrics.md)
- [Grafana dashboards](../monitoring/grafana/README.md)
- [Mock Files Generation](./mock.md)
- [AI Agents Best Practices](./ai_agents.md)
- [Tools: tokengen](./tokengen.md)
Expand Down
413 changes: 413 additions & 0 deletions docs/development/metrics.md

Large diffs are not rendered by default.

14 changes: 13 additions & 1 deletion docs/development/monitoring.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,19 @@
# Monitoring

We adopt the monitoring infrastructure provided by the [`Fabric Smart Client`](https://github.com/hyperledger-labs/fabric-smart-client/blob/main/docs/platform/view/monitoring.md).
We adopt the monitoring infrastructure provided by the [`Fabric Smart Client`](https://github.com/hyperledger-labs/fabric-smart-client/blob/main/docs/platform/view/services/monitoring.md).

We use the following two methods to monitor the performance of the application:
* **Metrics** provide an overview of the overall system performance using aggregated results, e.g. total requests, requests per second, current state of a variable, average duration, percentile of duration
* **Traces** help us analyze single requests by breaking down their lifecycles into smaller components

## Where to look next

* [Metrics Reference](./metrics.md) — every metric Panurus exports, under the exact name Prometheus
serves it, plus how those names are derived, example queries, and the current coverage gaps.
* [Grafana dashboards](../monitoring/grafana/README.md) — an importable overview dashboard covering
every exported metric, and what to run after editing a query.
* [Driver Metrics](../drivers/metrics.md) — how the driver service wrappers are built and which
methods they instrument.
* [Fabric Smart Client monitoring](https://github.com/hyperledger-labs/fabric-smart-client/blob/main/docs/platform/view/services/monitoring.md)
— the platform metrics and traces Panurus inherits (views, sessions, gRPC, process), and how to
enable the Prometheus endpoint and the tracing exporter.
35 changes: 14 additions & 21 deletions docs/drivers/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,14 @@ implementation, and records three metrics per method invocation:
| **Histogram** (`*_duration_seconds`) | Execution duration of each call |
| **Counter** (`*_errors_total`) | Total number of calls that returned an error |

The metric names in this page are the **declared** names, as written in the wrapper sources.
Prometheus exports them under a prefix derived from the package that creates them; because the driver
wrappers receive a TMS-scoped provider, every metric below is exported as
`panurus_core_common_metrics_<declared name>` — for instance `issue_service_operations_total` is
queried as `panurus_core_common_metrics_issue_service_operations_total`. See
[Metrics Reference](../development/metrics.md) for the derivation rules and the exported names of
every metric in the SDK.

All metrics carry four labels for multi-TMS filtering:

| Label | Description |
Expand Down Expand Up @@ -136,27 +144,12 @@ Metrics emitted:

## Metric Reference

The full list of metrics emitted by the driver wrappers:

| Metric Name | Type | Description |
|-------------|------|-------------|
| `issue_service_operations_total` | Counter | Total `IssueService` method invocations |
| `issue_service_duration_seconds` | Histogram | Duration of `IssueService` method calls |
| `issue_service_errors_total` | Counter | Total `IssueService` method errors |
| `transfer_service_operations_total` | Counter | Total `TransferService` method invocations |
| `transfer_service_duration_seconds` | Histogram | Duration of `TransferService` method calls |
| `transfer_service_errors_total` | Counter | Total `TransferService` method errors |
| `auditor_service_operations_total` | Counter | Total `AuditorService` method invocations |
| `auditor_service_duration_seconds` | Histogram | Duration of `AuditorService` method calls |
| `auditor_service_errors_total` | Counter | Total `AuditorService` method errors |
| `tokens_service_operations_total` | Counter | Total `TokensService` method invocations |
| `tokens_service_duration_seconds` | Histogram | Duration of `TokensService` method calls |
| `tokens_service_errors_total` | Counter | Total `TokensService` method errors |
| `tokens_upgrade_service_operations_total` | Counter | Total `TokensUpgradeService` method invocations |
| `tokens_upgrade_service_duration_seconds` | Histogram | Duration of `TokensUpgradeService` method calls |
| `tokens_upgrade_service_errors_total` | Counter | Total `TokensUpgradeService` method errors |

All metrics use labels: `network`, `channel`, `namespace`, `method`.
The exported names, types and labels of the fifteen driver metrics are listed in
[Metrics Reference — Driver services](../development/metrics.md#driver-services). That page is kept in
step with the code by `token/services/metricsdoc`, so it is the authoritative list; this page describes
only how the wrappers work and which methods they instrument.

All driver metrics use labels: `network`, `channel`, `namespace`, `method`.

## Source

Expand Down
58 changes: 58 additions & 0 deletions docs/monitoring/grafana/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Grafana dashboards

## `panurus.json` — Panurus Overview

A single overview dashboard covering every metric Panurus exports: 20 panels across 9 rows, one row per
subsystem (driver services, transaction lifecycle, finality listener, envelope sessions, auditor, token
selection, certification and identity caches, signer resolution and cache provisioning, Fabric-X finality
queue).

### Import

1. Grafana → **Dashboards** → **New** → **Import** → *Upload JSON file*.
2. Pick the Prometheus data source that scrapes the node's metrics endpoint when prompted for
`DS_PROMETHEUS`.

The dashboard declares four template variables — `network`, `channel`, `namespace` and `method` — whose
values are discovered with `label_values` against
`panurus_core_common_metrics_transfer_service_operations_total`. A node that has never issued or
transferred a token exports no series for that metric, so the pickers stay empty until the first
transaction; the unfiltered panels still work.

Requires Grafana 9.0 or later (`schemaVersion` 37).

### Not covered

- **FSC platform metrics** (views, sessions, gRPC, process) — these come from
[Fabric Smart Client](https://github.com/hyperledger-labs/fabric-smart-client/blob/main/docs/platform/view/services/monitoring.md)
and are exported under `fsc_*`, not `panurus_*`.
- **Traces.** The dashboard is metrics-only.
- Panels are built from metric *names*, so they show what a node reports, not whether the reported
numbers are healthy: there are no thresholds or alert rules here.

### Changing it

Every query in this file is checked by `token/services/metricsdoc`, which asserts that

- each metric a query names is one the SDK registers, under the name Prometheus actually exports;
- each metric name carries its package prefix, so a bare `Name` from the Go source fails the build
rather than rendering an empty panel;
- each label a query filters or groups on is declared by the metric it is applied to;
- each `$variable` a query interpolates is either a Grafana built-in or declared in this dashboard.

These are the failure modes a dashboard cannot report itself: Grafana does not error on an unknown
metric or an absent label, it renders **No data**, which is indistinguishable from an idle node. An
earlier version of this dashboard ([#1749](https://github.com/LFDT-Panurus/panurus/pull/1749)) had every
one of its 51 panel queries and 4 variable queries written against bare option names from the Go source,
so not one of them matched a series; it closed unmerged.

So: after editing a query, run

```bash
go test ./token/services/metricsdoc/...
```

If you add a panel for a metric that does not exist yet, add the metric first — see
[Metrics Reference](../../development/metrics.md) for the exported names and
[`testdata/metrics.golden`](../../../token/services/metricsdoc/testdata/metrics.golden) for the
machine-readable list.
Loading