Add OS-level resource metric collection to flow run subprocesses#21071
Merged
desertaxle merged 9 commits intomainfrom Mar 11, 2026
Merged
Conversation
Add OpenTelemetry-based CPU and memory metric collection inside flow run subprocesses, exporting via OTLP HTTP to the Cloud telemetry endpoint. - Add TelemetrySettings model with enable_resource_metrics and resource_metrics_interval_seconds settings - Add opentelemetry-instrumentation-system-metrics to otel extra and dev deps - Add RunMetrics context manager that starts SystemMetricsInstrumentor filtered to process.cpu.utilization, process.memory.usage, process.memory.virtual with proper resource attributes - Wrap run_flow() in engine.py __main__ block with RunMetrics Closes: OSS-7694 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merging this PR will not alter performance
Comparing 
|
…and type annotation - Add PREFECT_TELEMETRY_ENABLE_RESOURCE_METRICS and PREFECT_TELEMETRY_RESOURCE_METRICS_INTERVAL_SECONDS to SUPPORTED_SETTINGS - Fix test_noop_when_import_fails to use builtins module instead of __builtins__ dict - Fix test_instruments_and_shuts_down to patch OTel classes at their source modules instead of at the import site - Add type annotation to logger and use get_logger() Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…collection-to-flow-run
chrisguidry
approved these changes
Mar 11, 2026
Collaborator
chrisguidry
left a comment
chrisguidry
left a comment
There was a problem hiding this comment.
🥵 This is going to be real!
The OTLPMetricExporter needs the Prefect API key to authenticate with Cloud's telemetry ingestion endpoint. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prevents a 30s stall on shutdown when the endpoint is unreachable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 219e28a67c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…tion - Only send API key to Cloud-derived endpoints, not user-overridden ones - Pass settings into _resolve_metrics_endpoint to avoid double call - Set export_timeout_millis=5000 on the reader to prevent 30s shutdown stall - Add ge=1 validation on resource_metrics_interval_seconds Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3453cf9dd0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
zzstoatzz
approved these changes
Mar 11, 2026
Comment on lines
+122
to
+128
| with RunMetrics(flow_run, flow): | ||
| if flow.isasync: | ||
| run_coro_as_sync( | ||
| run_flow(flow, flow_run=flow_run, error_logger=run_logger) | ||
| ) | ||
| else: | ||
| run_flow(flow, flow_run=flow_run, error_logger=run_logger) |
…rt, cloud auth on override - Wrap telemetry setup in try/except so initialization errors degrade to no-op instead of aborting the flow run - Honor standard OTEL_EXPORTER_OTLP_ENDPOINT env var as fallback when the metrics-specific variable is not set - Preserve Cloud auth headers when endpoint is overridden via env var (is_cloud now derived from connected_to_cloud, not endpoint source) - Only pass headers kwarg to OTLPMetricExporter for Cloud endpoints so non-cloud exporters can use OTEL_EXPORTER_OTLP_HEADERS env vars Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
User-overridden endpoints (OTEL_EXPORTER_OTLP_METRICS_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT) now always return is_cloud=False, so the Prefect API key is only attached to the auto-derived Cloud endpoint. This prevents leaking credentials to third-party collectors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prevents double-slash in derived endpoint when PREFECT_API_URL ends with /, which can cause export failures behind strict proxies. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
desertaxle
deleted the
alexs/oss-7694-add-os-level-resource-metric-collection-to-flow-run
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TelemetrySettingsmodel withenable_resource_metrics(default:True) andresource_metrics_interval_seconds(default:10) settings underPREFECT_TELEMETRY_env prefixopentelemetry-instrumentation-system-metricsto theoteloptional extra anddevdependency groupRunMetricscontext manager that creates an OTelMeterProviderwithOTLPMetricExporter, startsSystemMetricsInstrumentorfiltered toprocess.cpu.utilization,process.memory.usage,process.memory.virtualwith flow run resource attributesrun_flow()inengine.py__main__block withRunMetricsso metrics are collected for the lifetime of flow run subprocessesOTEL_EXPORTER_OTLP_METRICS_ENDPOINTenv var > auto-derived from Cloud API URL > disabledCloses: OSS-7694
🤖 Generated with Claude Code