Background
In the 3.13.x incident, collateral fetches failed for hours and every TEE node was ~24h from being silently dropped from the participant set. Nothing alerted — we found it by hand-querying get_attestation. The node publishes no signal for how stale its attestation is.
Node side only; the Grafana/Prometheus half is https://github.com/near/mpc-private/issues/559, which needs upgraded nodes first.
Proposed design. Two gauges, written where the node already confirms a submission on chain (#3736):
- the expiry the contract stores for our TLS key — how long until eviction;
- when a submission last landed — how long re-attestation has been failing.
Both as absolute Unix timestamps rather than remaining durations, so a gauge frozen by a node that stopped re-attesting keeps decaying towards now and still trips its threshold. A single failed attempt is absorbed by the next hourly retry, so only sustained failure moves the signal. Documented thresholds stay relative to the configured expiry window rather than hard-coded, so they survive changes to it (cf. #3947).
Acceptance Criteria
Resources & Additional Notes
Background
In the 3.13.x incident, collateral fetches failed for hours and every TEE node was ~24h from being silently dropped from the participant set. Nothing alerted — we found it by hand-querying
get_attestation. The node publishes no signal for how stale its attestation is.Node side only; the Grafana/Prometheus half is https://github.com/near/mpc-private/issues/559, which needs upgraded nodes first.
Proposed design. Two gauges, written where the node already confirms a submission on chain (#3736):
Both as absolute Unix timestamps rather than remaining durations, so a gauge frozen by a node that stopped re-attesting keeps decaying towards now and still trips its threshold. A single failed attempt is absorbed by the next hourly retry, so only sustained failure moves the signal. Documented thresholds stay relative to the configured expiry window rather than hard-coded, so they survive changes to it (cf. #3947).
Acceptance Criteria
/metrics.Resources & Additional Notes
near/mpc-privateresearch/dcap-collateral-freshness.mpc_tee_attestation_attempts_total(feat: add metrics for TEE attestation generation attempts #2727) andmpc_own_image_hash_*(feat(metrics): expose the TEE image-hash allowlist expiry metrics #3752) already exist — the freshness family is what's missing.