fix(llmobs): sanitize dots in metric keys

[Yun-Kim]

Description

LLM Observability ingestion interprets a dot in a metric key as a nested-path separator. Because the backend metrics field decodes as a flat map of numeric values (map[string]float64), a dotted metric key such as anomaly.query_count gets expanded into a nested map ({"anomaly": {"query_count": 8}}), fails CBOR decode in llm-obs-events-processor with cannot unmarshal map into Go struct field ...SpansIngestPayload.custom of type float64, and drops the entire span batch — so nothing gets indexed.

This was hit by a staging customer (ml_app:reveng_anomaly_tooling_api) whose workflow span set custom metrics with dotted keys via LLMObs.annotate(metrics=...).

The SDK already handles this exact downstream behavior for tags (_llmobs.py, k.replace(".", "_") in APM_AGENTLESS mode) but not for metrics. This PR extends the same treatment to metric keys.

Fix

• Add _sanitize_metric_key() in ddtrace/llmobs/_utils.py that replaces . → _ in string metric keys (non-string keys untouched; value validation already happens in annotate).
• Apply it in _annotate_llmobs_span_data — the central funnel for all metric sources — so it covers both LLMObs.annotate(metrics=...) and integration-set metrics, in both export modes.
• Log a warning when a key is rewritten, so the behavior is visible.

Chose annotation-time (mode-agnostic) over serialization-time because the backend metrics field can never represent nested metrics — flattening dots is always correct, and metrics are vulnerable regardless of export path.

Testing

• New unit test test_annotate_metrics_dotted_keys_sanitized verifies dotted keys are flattened to underscores.
• tests/llmobs/test_llmobs_service.py -k annotate_metrics: 4 passed, 0 failed.
• Lint/format clean.

Risks

Low. Metric keys containing dots were already being silently dropped (whole batch), so any prior behavior for such keys was data loss. A key collision is theoretically possible if a user submitted both a.b and a_b as separate metrics; the second wins. This is an accepted edge case since dotted keys were previously unusable entirely.

Additional Notes

Scope is metrics-only. metadata with dotted keys still nests silently on the backend (map[string]interface{} — decodes without error but loses flat-key semantics). Happy to extend the same treatment to metadata in a follow-up if desired.

Claude session: 4aa13fb9-084b-4459-bfda-cc9ee284fc15
Resume: claude --resume 4aa13fb9-084b-4459-bfda-cc9ee284fc15

🤖 Generated with Claude Code

[cit-pr-commenter-54b7da]

Codeowners resolved as

ddtrace/llmobs/_utils.py                                                @DataDog/ml-observability
releasenotes/notes/llmobs-metric-key-dot-sanitize-60221758e3555b02.yaml  @DataDog/apm-python
tests/llmobs/test_llmobs_service.py                                     @DataDog/ml-observability

[cit-pr-commenter-54b7da]

Circular import analysis

⚠️ Existing circular imports

There are 4974 circular imports that already exist on the base branch and have not been changed by this PR.

Show existing cycles (showing 5 of 4974 shortest)

ddtrace.internal.datastreams -> ddtrace.internal.datastreams.kombu -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.botocore -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.aiokafka -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.google_cloud_pubsub -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.kafka -> ddtrace.internal.datastreams

To see all cycles, download the cycles-base.json and cycles-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/cycles.py compare cycles-base.json cycles-pr.json

[datadog-datadog-prod-us1]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 6c729f6 | Docs | Datadog PR Page | Give us feedback!}