Skip to content

feat: Add a per-DAG "current status" gauge metric #2351

Description

@dbascoules

Is your feature request related to a problem?

/api/v1/metrics exposes dagu_dag_runs_total_by_dag{dag, status} as a counter of today's runs, plus separate gauges for currently-running/queued. There's no single metric giving "the current status of DAG X". Concretely, after a DAG fails once and succeeds on a retry the same day, both show up:

dagu_dag_runs_total_by_dag{dag="x",status="failed"} 1
dagu_dag_runs_total_by_dag{dag="x",status="succeeded"} 1
Both values are individually correct, but there's no way to tell from Prometheus data alone which one reflects the DAG's current state — a dashboard/alert built on this counter can't answer "is this DAG healthy right now?".

Describe the solution you'd like

Expose a one-hot gauge for the current status of each DAG's latest run, e.g.:

# HELP dagu_dag_run_status Current status of the most recent DAG-run for each DAG (one-hot gauge)
# TYPE dagu_dag_run_status gauge
dagu_dag_run_status{dag="x", status="succeeded"} 1
dagu_dag_run_status{dag="x", status="failed"} 0
dagu_dag_run_status{dag="x", status="running"} 0
dagu_dag_run_status{dag="x", status="queued"} 0
This mirrors the pattern already used for dagu_worker_health_status, and would let Grafana build a "current DAG health" panel/alert directly, independent of dagu_dag_runs_total_by_dag's daily-counter scoping.

Describe alternatives you've considered

Create a custom metric endpoint based on Dagu API.

Use case

Monitoring Dagu.

Additional context

DAGRunStore.LatestAttempt(ctx, name) already returns the most recent attempt per DAG name. This looks implementable by extending collectDAGRunMetrics in internal/cmn/telemetry/collector.go to call LatestAttempt per DAG (from dagStore.List) and emit the one-hot gauge, the same way collectWorkerRecordMetrics already does for worker health.

Happy to test a PR if useful.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions