[flyte2] Executor: add custom domain metrics (TaskAction reconcile, cache, GC, plugins)

[pingsutw]

Part of #7445. Best paired with #7453 (which makes promutils-scoped metrics actually scrapeable on the executor endpoint).

Summary

The executor emits no custom domain metrics. It gets controller-runtime built-ins for free (reconcile counts/errors/duration, workqueue depth/latency), but there's no visibility into the executor's own behavior: TaskAction reconcile outcomes, cache effectiveness, garbage-collection activity, or plugin execution latency. Add these using the metrics Scope that's already plumbed through.

Background

The plumbing exists but is unused: executor/pkg/plugin/setup_context.go already exposes MetricsScope() promutils.Scope, and executor/setup.go constructs promutils.NewScope("executor"). But a grep for metric instruments (MustNewCounter/MustNewGauge/MustNewStopWatch, .Inc()/.Observe()/.Set()/.Start()) across the executor's own code returns nothing — the controllers don't emit any.

What to do

Add metrics (under dedicated sub-scopes, e.g. scope.NewSubScope("taskaction"), "cache", "gc") to the core executor logic:

• TaskAction controller (executor/pkg/controller/taskaction_controller.go): reconcile outcomes labeled by result/phase (success/error/requeue), and per-reconcile latency. (Note: controller-runtime already gives generic reconcile totals — add only what the generic metrics don't cover, e.g. terminal phase counts.)
• TaskAction cache (executor/pkg/controller/taskaction_cache.go): cache hit / miss / eviction counters, and current size gauge.
• Garbage collector (executor/pkg/controller/garbage_collector.go): objects deleted (counter), deletion errors (counter), and GC sweep duration.
• Plugin execution (via the registry / setupContext.MetricsScope()): per-plugin execution latency and error counts, if not already covered.

Acceptance criteria

• The above metrics appear on the executor metrics endpoint (depends on [flyte2] Executor: expose promutils (default-registry) metrics on the controller-runtime metrics endpoint #7453 being resolved so default-registry metrics are exposed; until then they can be verified via the default registry in a unit test).
• Metrics use dedicated sub-scopes and are created once (no duplicate-registration panics).
• Unit tests assert the relevant counters/gauges update (e.g. cache hit increments on a hit; GC deletion counter increments on delete).

Pointers

• executor/pkg/plugin/setup_context.go:44 — MetricsScope() accessor.
• executor/pkg/controller/taskaction_controller.go, taskaction_cache.go, garbage_collector.go — instrumentation targets.
• flytestdlib/promutils/scope.go — Scope helpers (MustNewCounter, MustNewGauge, MustNewStopWatch, NewSubScope).

Notes for contributors

• Keep label cardinality bounded — label by phase/result/plugin-type, never by action/run IDs.
• This can be split among contributors by component (controller vs cache vs gc vs plugins) — comment on which piece you're taking.

[runllm]

My understanding is that the executor currently lacks custom domain metrics, even though the plumbing for metrics scopes exists. To address this, you should use promutils.Scope and its helpers (MustNewCounter, MustNewGauge, MustNewStopWatch, NewSubScope) to add metrics under dedicated sub-scopes (e.g., scope.NewSubScope("taskaction"), "cache", "gc"), as described in the issue. This includes counters and timers for TaskAction outcomes, cache effectiveness, GC activity, and plugin execution latency. Make sure metrics are registered once per scope to avoid panics, and use bounded label cardinality (e.g., by phase/result/plugin-type only) as recommended.

It seems likely that you can follow the patterns in flytestdlib/promutils/scope.go for metric creation and registration, and verify updates in unit tests as shown in the test file. For example, use MustNewCounter for event counts, MustNewGauge for current values (like cache size), and MustNewStopWatch for durations. See flytestdlib/promutils/scope.go and flytestdlib/promutils/scope_test.go for concrete usage examples.

Would you like a step-by-step breakdown or example code for a specific component?

Sources:

• flytestdlib/promutils/scope.go
• flytestdlib/promutils/scope_test.go

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[davidlin20dev]

#take

[davidlin20dev]

Hi! I'd love to pick this one up and work on all four components (controller, cache, GC, and plugins).