[flyte2] Emit Prometheus metrics from v2 services (runs service + app framework)

[pingsutw]

Summary

The v2 services built on the flytestdlib/app framework (starting with the runs service) currently expose no Prometheus metrics. This is an umbrella issue tracking the work to add observability, broken into small, independent tasks that are great for first-time contributors.

Background

When you run the runs service (runs/cmd/main.go → app.App.Run()), the only HTTP endpoints served are /healthz and /readyz (flytestdlib/app/app.go:94-109). There is no /metrics endpoint, and nothing emits Prometheus metrics:

• app.SetupContext has a Scope promutils.Scope field (flytestdlib/app/context.go:62-63), but it is never assigned.
• runs/cmd/main.go:43 passes promutils.NewTestScope() into the DataStore — a throwaway scope that registers nothing.
• runs/setup.go registers no counters/gauges/timers and no RPC interceptors.
• The old v1 pattern that served /metrics (flytestdlib/profutils/server.go:118, promhttp.Handler()) was not ported into the v2 app framework.

So today a Prometheus scrape of the runs service returns nothing useful.

Goal

Give every v2 service that uses the app framework a Prometheus /metrics endpoint, a real metrics Scope, and meaningful instrumentation on the runs service hot paths.

Tasks

This work is split so multiple contributors can pick up pieces in parallel. Task 1 is the foundation and must land first; tasks 2–4 depend on it and can then proceed independently.

• [flyte2] Add /metrics endpoint and initialize metrics Scope in the app framework #7446 (foundation): Add a /metrics endpoint and initialize the metrics Scope in the app framework — blocker for the rest
• [flyte2] Add gRPC/Connect RPC metrics interceptor to the runs service #7447: Add gRPC/Connect RPC metrics interceptor to the runs service (depends on [flyte2] Add /metrics endpoint and initialize metrics Scope in the app framework #7446)
• [flyte2] Instrument the runs service DB repository layer with Prometheus metrics #7448: Instrument the runs DB repository layer with Prometheus metrics (depends on [flyte2] Add /metrics endpoint and initialize metrics Scope in the app framework #7446)
• [flyte2] Instrument the runs service reconcilers (abort-reconciler) with Prometheus metrics #7449: Instrument the runs reconcilers (abort-reconciler) with Prometheus metrics (depends on [flyte2] Add /metrics endpoint and initialize metrics Scope in the app framework #7446)
• [flyte2] Instrument the actions service (watcher metrics + dropped-updates counter) #7450: Instrument the actions service (watcher metrics + dropped-updates counter) (depends on [flyte2] Add /metrics endpoint and initialize metrics Scope in the app framework #7446)

Executor (controller-runtime based — already has a /metrics endpoint, but gaps remain):

• [flyte2] Executor: expose promutils (default-registry) metrics on the controller-runtime metrics endpoint #7453: Expose promutils (default-registry) metrics on the controller-runtime metrics endpoint — scoped metrics are collected but not scraped
• [flyte2] Executor: add custom domain metrics (TaskAction reconcile, cache, GC, plugins) #7455: Add custom domain metrics (TaskAction reconcile, cache, GC, plugins)
• [flyte2] Executor: dedupe the two promutils.NewScope("executor") constructions in setup.go #7456: Dedupe the two promutils.NewScope("executor") constructions in setup.go

How to claim a task

Comment on the specific sub-issue you'd like to work on and a maintainer will assign it. Please land the foundation issue (#7446) first, since the others build on it. Each sub-issue is self-contained with file references and acceptance criteria.

[runllm]

It seems likely that, as of now, v2 services using the flytestdlib/app framework (such as the runs service) do not expose a Prometheus /metrics endpoint or emit any Prometheus metrics. The Scope promutils.Scope is present but not properly initialized, and the legacy v1 pattern for serving /metrics was not ported to v2. The issue proposes adding a /metrics endpoint, initializing a real metrics scope, and instrumenting key service components for observability, with the foundational task being to add the /metrics endpoint and metrics scope initialization in the app framework. This foundational work is required before further instrumentation can proceed.

My understanding is that the data sources confirm the absence of Prometheus metrics in the current v2 app framework and outline the need for explicit integration, but do not provide implementation details or commands for adding the /metrics endpoint or initializing the metrics scope in v2. Would you like a step-by-step breakdown of how to approach implementing this foundational task?

Sources:

• https://github.com/flyteorg/flyte/blob/master/flytestdlib/promutils/scope_test.go
• https://github.com/flyteorg/flyte/blob/master/flyteplugins/go/tasks/pluginmachinery/internal/webapi/metrics.go
  Note: I wasn’t able to find highly relevant data sources, but above are a few potentially relevant links.

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!