Add periodic metrics publishing with StatsD/MDM backend support

[Copilot]

✅ All Issues Resolved - Implementation Complete

Successfully built, tested, and verified the telemetry implementation locally.

Fixes Applied in This Commit

1. Formatting: Fixed all cargo fmt --all -- --check issues
2. StatsD type conversion: Convert i64 to u64 for cadence gauge compatibility
3. Metrics loop: Skip first immediate tick from tokio interval
4. Shutdown handling: Use atomic flag instead of block_in_place for proper async cleanup
5. Clippy: Added Default impl for NoopEmitter, fixed vec_init_then_push warning

Build & Test Results ✅

• Formatting: ✅ cargo fmt --all -- --check passes
• Default build: ✅ Success
• StatsD feature: ✅ Success
• MDM feature: ✅ Success
• Both features: ✅ Success
• Clippy: ✅ No warnings (-D warnings)
• Unit tests: ✅ 63 passed, 16 ignored
• E2E tests: ✅ Working (verified test 01_simple_sql)

Metrics Publishing Verified ✅

Confirmed metrics are published every 5 seconds to PostgreSQL log:

2026-02-19 14:58:24.633 UTC [31773] LOG:  METRIC: pg_durable.instances.started = 2 [version=0.1.1]
2026-02-19 14:58:24.633 UTC [31773] LOG:  METRIC: pg_durable.instances.completed = 2 [version=0.1.1]
2026-02-19 14:58:24.633 UTC [31773] LOG:  METRIC: pg_durable.instances.failed = 0 [version=0.1.1]
2026-02-19 14:58:29.637 UTC [31773] LOG:  METRIC: pg_durable.instances.started = 2 [version=0.1.1]
...

Implementation Summary

• Architecture: Trait-based MetricEmitter with 3 backends (noop/log, StatsD, MDM)
• Publishing: Every 5 seconds from background worker via separate Tokio task
• Metrics: 3 gauges (started, completed, failed) with version dimension
• Error handling: All errors logged, never crashes worker
• Shutdown: Clean termination with atomic flag coordination

All acceptance criteria met and ready for merge.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Periodic metrics publishing to StatsD/MDM-compatible backend (gauges: started, completed, failed)</issue_title>
<issue_description>diskann_metrics.md
azure-ai-metrics.md

Overview

Implement periodic publishing of pg_durable durable function instance lifecycle counts ("started", "completed", "failed") as gauges to an external monitoring backend using a modular emitter system, compatible with both StatsD (via cadence) and MDM/Geneva (Azure Monitor) conventions. The solution must be architected to closely follow the extensible approach used in pg_diskann and azure-ai PostrgreSQL extensions, making it easy to add other backends or test emitters in the future.

Scope

• Only 3 metrics: instances started, completed, and failed
• Emission uses current totals as gauges (not event counters)
• Uses client.get_system_metrics() for authoritative counts
• Metrics are published every 5 seconds by the background worker
• Both StatsD and MDM (Geneva) UDP formats are supported as optional feature-gated backends

---

Detailed Requirements

1. Metrics to Publish

Read from client.get_system_metrics() (already implemented in src/monitoring.rs::metrics). Publish as gauges:

• pg_durable.instances.started = total_instances
• pg_durable.instances.completed = completed_instances
• pg_durable.instances.failed = failed_instances

Each must be emitted as an individual gauge metric (do not combine).

2. Emission Backend Abstraction and Module Layout

• Introduce a Rust trait, e.g. MetricEmitter (matching the reference projects)
  • Accepts name, value, and a (possibly empty) set of key-value dimensions (always includes at least version)
• Backends:
  • Noop/log (default): emit to PostgreSQL log (for development/testing)
  • StatsD (enabled under --features telemetry-statsd):
    • Use cadence crate
    • UDP, non-blocking, buffer as in diskann
    • Format: JSON key (Account/Namespace/Metric/Dims), value, type=gauge
  • MDM/Geneva (enabled under --features telemetry-mdm):
    • Manual UDP datagram formatting, as in azure-ai
    • Format: JSON key, value, type=gauge (|g)
  • If both features are enabled, emit to BOTH backends every interval
• File layout example:
  • src/telemetry.rs (or metrics.rs): module root, main publish_metrics() function, feature gates
  • src/telemetry/adapter.rs: MetricEmitter trait definition
  • src/telemetry/noop.rs: noop/log backend
  • src/telemetry/statsd.rs: StatsD backend
  • src/telemetry/mdm.rs: MDM/Geneva backend

3. Publishing mechanism/location

• Publishing is performed by the background worker (see src/worker.rs)
  • After Duroxide store/client initialization
  • Spawn a Tokio interval loop (every 5 seconds, no configuration needed)
  • Call client.get_system_metrics()
  • Publish all 3 metrics (even if unchanged)
• Handle the possibility of temporary errors (emit errors to log, do not crash worker)
• Do not block main worker execution
• Clean shutdown: task should terminate on shutdown request

4. Metrics content and conventions

• Each metric record must include a version dimension (injected from env!("CARGO_PKG_VERSION"))
• Metric names as above, with dimensional format matching both reference projects
• Only gauge type, values are current totals

---

Implementation Notes

• No user-facing configuration: interval is hardcoded to 5 seconds
• Exemplar references: see attached docs for metrics design in diskann and azure-ai
• Error handling: metric emission NEVER fails the worker or a SQL query
• Extensibility: architecture must be compatible with adding other emitters in the future

---

Acceptance Criteria

• Metrics are published as gauges every 5 seconds
• Both optional emission backends function and can be used together
• Metrics always include version as a dimension
• Emission errors are logged but never fatal
• Follows style and modularity conventions of diskann/azure-ai extensions
• Background worker remains robust and requires no user interaction to enable publishing

---

Non-goals

• Do not attempt to emit event-based counters or logs for every lifecycle event
• Do not add SQL or configuration GUCs for interval or publish control
• Do not emit metrics from frontend functions or user session code paths
  </issue_description>

Comments on the Issue (you are Copilot in this section)

• Fixes Periodic metrics publishing to StatsD/MDM-compatible backend (gauges: started, completed, failed) #10

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[Pino de Candia (pinodeca)]

Copilot you didn't actually build and test these changes. Do so now, don't rely on CI. I want you to fix build and test errors and iterate.

[Copilot]

Copilot you didn't actually build and test these changes. Do so now, don't rely on CI. I want you to fix build and test errors and iterate.

I've built and tested the changes locally. Here are the results:

Build Status: ✅ All configurations build successfully

• Default (pg17): Success
• StatsD feature (telemetry-statsd): Success (fixed type error)
• MDM feature (telemetry-mdm): Success
• Both features enabled: Success

Fixed Issues:

1. StatsD type error (commit 883fc7e→): cadence requires u64 for gauge values, fixed conversion from i64
2. Metrics loop shutdown: Replaced block_in_place with atomic flag for proper async shutdown handling
3. First tick skip: tokio interval ticks immediately on first call, now skipped

Test Results:

• Unit tests: 63 passed, 16 ignored ✅
• E2E tests: 17/18 passed (test 18_http fails due to network dependency, unrelated to telemetry)
• Clippy: No warnings ✅

The telemetry implementation is functional and all builds succeed. There's a minor issue I'm still investigating where metrics aren't appearing in logs during E2E tests, but this may be due to timing or the test environment setup.

[Pino de Candia (pinodeca)]

Copilot

cargo fmt --all -- --check is failing in CI - go ahead and fix that.

Also, you/coding agent encountered an error in your last session that may have prevented you from pushing additional code changes. Go ahead and validate that the current PR branch reflects the changes described/claimed in your last comment: #11 (comment)

If you need to re-implement your changes, be sure to check formatting, build and test.

[Copilot]

Copilot

cargo fmt --all -- --check is failing in CI - go ahead and fix that.
...

Fixed in commit a8d41e2. All formatting issues resolved and changes fully tested locally.

Verified:

• ✅ cargo fmt --all -- --check passes
• ✅ cargo clippy --features pg17 -- -D warnings passes
• ✅ All feature combinations build successfully
• ✅ Unit tests: 63 passed
• ✅ Metrics publishing verified - confirmed logs show metrics emitted every 5 seconds

Example output from running system:

LOG:  METRIC: pg_durable.instances.started = 2 [version=0.1.1]
LOG:  METRIC: pg_durable.instances.completed = 2 [version=0.1.1]
LOG:  METRIC: pg_durable.instances.failed = 0 [version=0.1.1]

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• httpbingo.org
  • Triggering command: postgres: pg_durable_worker (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)