docs(otel): document runtime.telemetry.metric_prefix, properties, and DataDog OTLP setup (#1542)

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>

Author: phillipleblanc

website/docs/features/observability/index.md

@@ -162,6 +162,8 @@ export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.us3.datadoghq.com"
 export OTEL_EXPORTER_OTLP_HEADERS="DD-API-KEY=${DD_API_KEY}"
 ```
 
+For a complete Datadog setup including metric prefixing and custom tags via OTLP resource attributes, see the [Datadog monitoring guide](/docs/next/monitoring/datadog#opentelemetry-otlp-export).
+
 #### Grafana Cloud (OTLP/HTTP)
 
 Grafana Cloud's OTLP gateway expects HTTP Basic authentication. Obtain the base64-encoded `instanceID:accessPolicyToken` credential from the Grafana Cloud "OpenTelemetry" connection page and store it in a secret:
@@ -200,6 +202,25 @@ runtime:
         api-key: ${secrets:collector_api_key}
 ```
 
+## Metric Naming and Custom Tags
+
+Two runtime fields control how exported metrics are named and labeled across **all** readers (Prometheus scrape, cluster OTLP reader, and the `otel_exporter` push exporter):
+
+- [`runtime.telemetry.metric_prefix`](/docs/next/reference/spicepod/runtime#runtimetelemetrymetric_prefix) — prepends a string to every metric name (e.g. `spiceai.query_duration_ms`). Useful for namespacing in shared backends.
+- [`runtime.telemetry.properties`](/docs/next/reference/spicepod/runtime#runtimetelemetryproperties) — attaches custom key/value attributes as OpenTelemetry resource attributes, which most backends surface as dimensions or tags.
+
+```yaml
+runtime:
+  telemetry:
+    metric_prefix: 'spiceai.'
+    properties:
+      environment: prod
+      region: us-west-2
+      team: data-platform
+```
+
+Both fields apply to every exporter the runtime has enabled. See the [Datadog monitoring guide](/docs/next/monitoring/datadog#opentelemetry-otlp-export) for backend-specific notes (Datadog requires `dd-otel-metric-config` to map resource attributes to tags).
+
 ### Metric Filtering
 
 To export only specific metrics, use the `metrics` parameter:

website/docs/monitoring/datadog/index.md

@@ -43,3 +43,97 @@ instances:
 3. Dashboard is now configured to display Spice.ai OSS key performance metrics
 
 <img width="800" src="/img/datadog/spice_datadog_dashboard.png"/>
+
+## OpenTelemetry OTLP Export
+
+As an alternative to scraping the Prometheus endpoint with the Datadog Agent, Spice can push metrics directly to Datadog's [OTLP Metrics Intake Endpoint](https://docs.datadoghq.com/opentelemetry/setup/intake_endpoint/) over HTTP. This is the recommended approach for agentless deployments (e.g. serverless, ephemeral containers) and for environments where the Datadog API key is managed through Spice's [secret stores](../../components/secret-stores).
+
+### Minimal Configuration
+
+Replace `us3` with the Datadog site for the target account (`us3`, `us5`, `eu`, `ap1`, etc.) and store the Datadog API key in a secret:
+
+```yaml
+runtime:
+  telemetry:
+    otel_exporter:
+      endpoint: https://otlp.us3.datadoghq.com/v1/metrics
+      headers:
+        DD-API-KEY: ${secrets:DD_API_KEY}
+```
+
+Metrics begin appearing in the Datadog Metrics Explorer within a minute or two.
+
+### Namespace Spice Metrics with a Prefix
+
+Use [`runtime.telemetry.metric_prefix`](/docs/next/reference/spicepod/runtime#runtimetelemetrymetric_prefix) to prepend a string to every exported metric name. This avoids collisions with metrics from other services in the same Datadog account:
+
+```yaml
+runtime:
+  telemetry:
+    metric_prefix: 'spiceai.'
+```
+
+The runtime metric `query_duration_ms` is then exported as `spiceai.query_duration_ms`.
+
+### Add Custom Tags via Resource Attributes
+
+Attach custom key/value pairs to every metric using [`runtime.telemetry.properties`](/docs/next/reference/spicepod/runtime#runtimetelemetryproperties). Spice sends these as OpenTelemetry resource attributes:
+
+```yaml
+runtime:
+  telemetry:
+    properties:
+      environment: prod
+      region: us-west-2
+      team: data-platform
+```
+
+For these resource attributes to surface as **tags** in Datadog, the Datadog OTLP intake also requires the `dd-otel-metric-config` header with `resource_attributes_as_tags` enabled (see [Datadog OTLP Metrics Intake Endpoint](https://docs.datadoghq.com/opentelemetry/setup/intake_endpoint/)):
+
+```yaml
+runtime:
+  telemetry:
+    otel_exporter:
+      endpoint: https://otlp.us3.datadoghq.com/v1/metrics
+      headers:
+        DD-API-KEY: ${secrets:DD_API_KEY}
+        dd-otel-metric-config: '{"resource_attributes_as_tags": true}'
+```
+
+:::note Tags can lag behind metrics
+Datadog typically ingests OTLP metrics within seconds, but the associated tags (from resource attributes) can take noticeably longer to appear in the UI — sometimes several minutes after the first datapoints. The metrics and tags do eventually converge.
+:::
+
+:::caution Manage tag cardinality in Datadog
+Datadog [bills on custom metric cardinality](https://docs.datadoghq.com/account_management/billing/custom_metrics/), driven by the number of unique tag-value combinations per metric. The custom tags added via `runtime.telemetry.properties` are typically low-cardinality (`environment`, `region`, `team`), but Spice metrics also carry a number of automatically populated dimensions — for example `dataset`, `protocol`, `client`, `client_version`, `client_system`, `user_agent`, `runtime`, `runtime_version`, `runtime_system` (see [Available Metrics](../../features/observability#available-metrics)) — some of which can grow with the size of the deployment.
+
+Datadog's [Metrics without Limits™](https://docs.datadoghq.com/metrics/metrics-without-limits/) decouples ingestion from indexing for exactly this case. With Metrics without Limits™, every tag Spice emits is still ingested, but each metric is configured with one of:
+
+- an **allowlist** that keeps only the tags actually used in dashboards, monitors, and queries (e.g. keep `dataset` and `environment`, drop the rest), or
+- a **blocklist** that drops specific auto-populated tags that are not useful for a given metric (e.g. exclude `user_agent` or `client_version`).
+
+Only the indexed (queryable) tag combinations count toward custom metric billing. Configuration is done per metric in the Metrics Summary page or via the Metrics API, and the in-app UI surfaces an estimated indexed-metric volume before saving and can pre-populate an allowlist from tags actively queried in dashboards, monitors, and notebooks.
+:::
+
+### Full Example
+
+A complete `runtime.telemetry` block combining metric prefixing, custom tags, and Datadog OTLP export:
+
+```yaml
+runtime:
+  telemetry:
+    metric_prefix: 'spiceai.'
+    properties:
+      environment: prod
+      region: us-west-2
+      team: data-platform
+    otel_exporter:
+      endpoint: https://otlp.us3.datadoghq.com/v1/metrics
+      headers:
+        DD-API-KEY: ${secrets:DD_API_KEY}
+        dd-otel-metric-config: '{"resource_attributes_as_tags": true}'
+```
+
+With this configuration, every Spice metric (e.g. `spiceai.query_duration_ms`, `spiceai.query_executions`) arrives in Datadog tagged with `environment:prod`, `region:us-west-2`, and `team:data-platform`.
+
+For general OTLP exporter options (push interval, metric filtering, gRPC vs HTTP), see [OpenTelemetry Metrics Exporter](../../features/observability#opentelemetry-metrics-exporter).

website/docs/reference/spicepod/runtime.md

@@ -345,6 +345,37 @@ runtime:
 
 Enables or disables runtime telemetry collection. Defaults to `true`.
 
+### `runtime.telemetry.metric_prefix` {#runtimetelemetrymetric_prefix}
+
+Optional string prepended to every exported metric name. Useful for namespacing Spice metrics in shared backends (e.g. Datadog, Grafana Cloud, New Relic) so they do not collide with metrics from other services. Defaults to no prefix.
+
+The prefix applies to **all** metric readers — the Prometheus scrape endpoint (`--metrics`), the cluster on-demand OTLP reader, and the `otel_exporter` push exporter — because OpenTelemetry views are configured at the meter-provider level rather than per reader.
+
+```yaml
+runtime:
+  telemetry:
+    metric_prefix: 'spiceai.'
+```
+
+With this configuration, the runtime metric `query_duration_ms` is exported as `spiceai.query_duration_ms`.
+
+### `runtime.telemetry.properties` {#runtimetelemetryproperties}
+
+Map of custom key/value attributes attached to telemetry metrics emitted by `spiced`. Applied as OpenTelemetry resource attributes on the runtime's `MeterProvider`, so they appear as dimensions/tags on every metric exported via the Prometheus scrape endpoint, the cluster on-demand OTLP reader, and the `otel_exporter` push exporter. Defaults to empty.
+
+```yaml
+runtime:
+  telemetry:
+    properties:
+      environment: prod
+      region: us-west-2
+      team: data-platform
+```
+
+The standard OpenTelemetry environment variables (`OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES`) are still honored and act as defaults; explicit `properties` entries take precedence on key conflicts.
+
+For backends that map OTLP resource attributes to tags through additional configuration (e.g. Datadog), see the [Datadog OTLP guide](/docs/next/monitoring/datadog#opentelemetry-otlp-export).
+
 ### `runtime.telemetry.otel_exporter`
 
 Configures an [OpenTelemetry](https://opentelemetry.io/) metrics exporter to push metrics to an OpenTelemetry collector. The exporter automatically infers the protocol (gRPC or HTTP) based on the endpoint configuration.