Skip to content

feat(node): expose attestation-freshness metrics on /metrics #3951

Description

@barakeinav1

Background

In the 3.13.x incident, collateral fetches failed for hours and every TEE node was ~24h from being silently dropped from the participant set. Nothing alerted — we found it by hand-querying get_attestation. The node publishes no signal for how stale its attestation is.

Node side only; the Grafana/Prometheus half is https://github.com/near/mpc-private/issues/559, which needs upgraded nodes first.

Proposed design. Two gauges, written where the node already confirms a submission on chain (#3736):

  • the expiry the contract stores for our TLS key — how long until eviction;
  • when a submission last landed — how long re-attestation has been failing.

Both as absolute Unix timestamps rather than remaining durations, so a gauge frozen by a node that stopped re-attesting keeps decaying towards now and still trips its threshold. A single failed attempt is absorbed by the next hourly retry, so only sustained failure moves the signal. Documented thresholds stay relative to the configured expiry window rather than hard-coded, so they survive changes to it (cf. #3947).

Acceptance Criteria

  • The node exposes attestation freshness on /metrics.
  • The signal distinguishes a sporadic failure from a sustained one.
  • Metrics and suggested alert thresholds are documented for operators.

Resources & Additional Notes

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions