feat(profiling): surface native-heap partition arming via dogstatsd gauge + warning

[vlad-scherbich]

Stack: A — cdylib/build (#19078) ← B — activation (#19079) ← C — live-heap (#19325) ← Phase 2 — ownership partition (#19349) ← F — arming observability (this PR).

Prev: #19349 (base of this PR).

Description

Target 1 of the native-heap observability follow-ups: make the ownership-partition arming decision confirmable at runtime.

Today the decision is emitted only as a one-shot stdlib LOG.info(...) in profiler._start_service. In real deploys the app's structured JSON logger drops ddtrace stdlib records and pods/exec is RBAC-blocked, so armed=true|false is unconfirmable at runtime — which blocks Phase 2 A/B validation.

This surfaces the decision two ways at service start, gated on native-heap profiling being enabled so the default majority of processes (which never enable it) stay completely silent — no new WARNING/metric noise:

• dogstatsd gauge profiling.native_heap.partition_armed — 1 when armed, 0 when not. Tags: domains:OBJ_MEM, size_threshold_bytes:512. Emitted via the canonical internal client (get_dogstatsd_client(agent_config.dogstatsd_url)), the same accessor runtime_metrics/writer use.
• WARNING log — the existing INFO arming line is upgraded to LOG.warning(...) with the same fields (armed=<bool> size_threshold_bytes=512 domains=OBJ|MEM) so it survives deploys whose JSON loggers filter ddtrace INFO records.

Emission is one-shot at _start_service (never on a hot path) and fail-safe: if the dogstatsd client is unavailable/raises, the error is swallowed (LOG.debug) and arming/startup is unaffected.

Out of scope (separate follow-up): the DD_PROFILING_NATIVE_HEAP_ENABLED config registration, which collides on append-only config files.

Testing

tests/profiling/test_native_heap_gotter.py (run via the run-tests skill / riot profile venv, py3.13):

• armed → gauge emitted once with value 1 + WARNING logged with armed=True
• enabled but gotter not installed → gauge value 0 + WARNING with armed=False
• dogstatsd client raising must not break arming or startup
• existing arming/partition wiring tests updated for the new WARNING on stderr (only the arming line is allowed)
• feature disabled (default) → no WARNING, no gauge (stderr unchanged)

Risks

Low. The WARNING/gauge only emit when DD_PROFILING_NATIVE_HEAP_ENABLED=true; the default path is byte-for-byte silent as before. The gauge is best-effort and cannot break profiler startup.

Additional Notes

No release note: native-heap profiling is behind unmerged/unreleased stacked PRs, so this is observability for a not-yet-released feature (changelog/no-changelog).

[cit-pr-commenter-54b7da]

Codeowners resolved as

ddtrace/profiling/profiler.py                                           @DataDog/profiling-python
tests/profiling/test_native_heap_gotter.py                              @DataDog/profiling-python

[cit-pr-commenter-54b7da]

Circular import analysis

⚠️ Existing circular imports

There are 3 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.trace -> ddtrace._trace.tracer -> ddtrace.internal.debug -> ddtrace.trace
ddtrace -> ddtrace.trace -> ddtrace._trace.tracer -> ddtrace.internal.debug -> ddtrace
ddtrace -> ddtrace.trace -> ddtrace._trace.tracer -> ddtrace.internal.debug -> ddtrace.internal.runtime.runtime_metrics -> ddtrace

[datadog-datadog-prod-us1-2]

✨ Unblock PR with BitsAI

⚠️ Warnings

🚦 1 Pipeline job failed

DataDog/apm-reliability/dd-trace-py | validate_supported_configurations_v2_local_file     

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 2f180af | Docs | Datadog PR Page | Give us feedback!}

[vlad-scherbich]

Closing: the deterministic ownership-handoff integration test (test_native_heap_ownership_handoff_end_to_end on #19349) proves Phase-2 partition arming without needing the dogstatsd gauge. Arming is also confirmable via eBPF on covered nodes. The stdlib INFO log was already invisible in JSON-logging setups; this metric was a workaround for that observability gap, superseded by the CI proof.

[vlad-scherbich]

Closing rationale (recorded for the stack): superseded by the deterministic ownership-handoff integration test on #19349 (test_native_heap_ownership_handoff_end_to_end), which now runs in CI via the dedicated profiling_native_heap_gotter_e2e job. That test proves partition arming + dedup using the gotter test-support hook-hit counter, without needing a runtime dogstatsd gauge.

The arming observability metric was draft/pending keep-or-close; closing it as redundant with the CI proof. If we still want prod dogfooding visibility later, we can revisit as optional follow-up work.