Skip to content

Commit b6165fc

Browse files
lukekimlukekim
committed
docs(otel): Document headers/auth for OTEL metrics exporter
Document the new headers field on runtime.telemetry.otel_exporter (spiceai/spiceai#10347), plus update the 'gRPC only' note which is now outdated (HTTP has been supported for a while). features/observability/index.md: - New 'Authentication' subsection with secret-store guidance. - New examples: Datadog (OTLP/HTTP with DD-API-KEY), Grafana Cloud (OTLP/HTTP with Basic Auth), and gRPC with lowercase metadata keys. - Each cloud example includes the equivalent OTEL_EXPORTER_OTLP_* environment-variable form as a cross-reference. - Replaced the stale 'gRPC only' protocol statement with a two-line gRPC/HTTP inference summary. - Added 'headers' row to the param table. - Callout noting gRPC metadata keys must be lowercase ASCII. reference/spicepod/runtime.md: - Added 'headers' row to the otel_exporter param table with secret- store linking. - Added Datadog, Grafana Cloud, and gRPC auth-header examples.
1 parent cc3ebad commit b6165fc

2 files changed

Lines changed: 144 additions & 14 deletions

File tree

‎website/docs/features/observability/index.md‎

Lines changed: 100 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -84,20 +84,41 @@ Spice can push metrics to an [OpenTelemetry](https://opentelemetry.io/) collecto
8484

8585
Configure the OpenTelemetry exporter in `spicepod.yaml` under `runtime.telemetry.otel_exporter`:
8686

87-
| Parameter | Required | Default | Description |
88-
| --------------- | -------- | ------- | --------------------------------------------------------------------- |
89-
| `enabled` | No | `true` | Whether the OpenTelemetry exporter is enabled. |
90-
| `endpoint` | Yes | - | The OpenTelemetry collector endpoint. |
91-
| `push_interval` | No | `60s` | How frequently metrics are pushed to the collector. |
92-
| `metrics` | No | `[]` | List of metric names to export. When empty, all metrics are exported. |
87+
| Parameter | Required | Default | Description |
88+
| --------------- | -------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
89+
| `enabled` | No | `true` | Whether the OpenTelemetry exporter is enabled. |
90+
| `endpoint` | Yes | - | The OpenTelemetry collector endpoint. Protocol (gRPC or HTTP) is inferred from the format. |
91+
| `push_interval` | No | `60s` | How frequently metrics are pushed to the collector. |
92+
| `metrics` | No | `[]` | List of metric names to export. When empty, all metrics are exported. |
93+
| `headers` | No | `{}` | Map of headers to send with each export request. For HTTP: sent as HTTP headers. For gRPC: sent as metadata entries (keys must be lowercase ASCII). Values support the `${secrets:...}` replacement syntax. |
9394

9495
### Protocol
9596

96-
Spice currently supports only the gRPC protocol for OpenTelemetry metrics export. Specify the collector `endpoint` as a host and port (e.g., `localhost:4317`).
97+
Spice infers the OTLP protocol from the `endpoint` format:
98+
99+
- **gRPC** — bare host:port with no scheme (e.g. `localhost:4317`). Default port: `4317`.
100+
- **HTTP** — includes the `http://` or `https://` scheme and ends in `/v1/metrics` (e.g. `http://localhost:4318/v1/metrics`, `https://otlp.us3.datadoghq.com/v1/metrics`). Default port: `4318`.
101+
102+
### Authentication
103+
104+
For collectors that require authentication (Datadog, Grafana Cloud, New Relic, Honeycomb, etc.), set the `headers` map. Secret values should be loaded from a [supported secret store](../../components/secret-stores) using the `${secrets:...}` [replacement syntax](../../components/secret-stores#using-secrets) rather than committed to source:
105+
106+
```yaml
107+
runtime:
108+
telemetry:
109+
otel_exporter:
110+
endpoint: 'https://otlp.example.com/v1/metrics'
111+
headers:
112+
Authorization: 'Bearer ${secrets:otlp_token}'
113+
```
114+
115+
:::tip gRPC metadata keys must be lowercase
116+
When exporting over gRPC, header keys are sent as gRPC metadata and **must be lowercase ASCII** — use `authorization`, not `Authorization`. The runtime fails fast at startup if any gRPC metadata key is invalid. HTTP exports preserve the casing you provide.
117+
:::
97118

98119
### Examples
99120

100-
**gRPC (default port 4317):**
121+
#### Local gRPC collector
101122

102123
```yaml
103124
runtime:
@@ -108,6 +129,77 @@ runtime:
108129
push_interval: '30s'
109130
```
110131

132+
#### Local HTTP collector
133+
134+
```yaml
135+
runtime:
136+
telemetry:
137+
enabled: true
138+
otel_exporter:
139+
endpoint: 'http://localhost:4318/v1/metrics'
140+
push_interval: '30s'
141+
```
142+
143+
#### Datadog (OTLP/HTTP)
144+
145+
Replace `us3` with your Datadog site (`us3`, `us5`, `eu`, `ap1`, etc.) and store the API key in a secret store:
146+
147+
```yaml
148+
runtime:
149+
telemetry:
150+
enabled: true
151+
otel_exporter:
152+
endpoint: 'https://otlp.us3.datadoghq.com/v1/metrics'
153+
push_interval: '30s'
154+
headers:
155+
DD-API-KEY: ${secrets:dd_api_key}
156+
```
157+
158+
Equivalent standard OTLP environment-variable form (for cross-reference):
159+
160+
```bash
161+
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.us3.datadoghq.com"
162+
export OTEL_EXPORTER_OTLP_HEADERS="DD-API-KEY=${DD_API_KEY}"
163+
```
164+
165+
#### Grafana Cloud (OTLP/HTTP)
166+
167+
Grafana Cloud's OTLP gateway expects HTTP Basic authentication. Obtain the base64-encoded `instanceID:accessPolicyToken` credential from the Grafana Cloud "OpenTelemetry" connection page and store it in a secret:
168+
169+
```yaml
170+
runtime:
171+
telemetry:
172+
enabled: true
173+
otel_exporter:
174+
endpoint: 'https://otlp-gateway-us-central2.grafana.net/otlp/v1/metrics'
175+
push_interval: '30s'
176+
headers:
177+
Authorization: 'Basic ${secrets:grafana_cloud_auth}'
178+
```
179+
180+
Equivalent standard OTLP environment-variable form (for cross-reference):
181+
182+
```bash
183+
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-us-central2.grafana.net/otlp"
184+
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic ${GRAFANA_CLOUD_AUTH}"
185+
```
186+
187+
Match the region in the URL to your Grafana Cloud stack (`us-central2`, `eu-west-2`, `prod-ap-south-0`, etc.).
188+
189+
#### gRPC collector with auth metadata
190+
191+
```yaml
192+
runtime:
193+
telemetry:
194+
enabled: true
195+
otel_exporter:
196+
endpoint: 'otel-collector.internal:4317'
197+
push_interval: '30s'
198+
headers:
199+
# Keys MUST be lowercase for gRPC
200+
api-key: ${secrets:collector_api_key}
201+
```
202+
111203
### Metric Filtering
112204

113205
To export only specific metrics, use the `metrics` parameter:

‎website/docs/reference/spicepod/runtime.md‎

Lines changed: 44 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -349,12 +349,13 @@ Enables or disables runtime telemetry collection. Defaults to `true`.
349349

350350
Configures an [OpenTelemetry](https://opentelemetry.io/) metrics exporter to push metrics to an OpenTelemetry collector. The exporter automatically infers the protocol (gRPC or HTTP) based on the endpoint configuration.
351351

352-
| Parameter name | Optional | Default | Description |
353-
| --------------- | -------- | ------- | ------------------------------------------------------------------------------------------------ |
354-
| `enabled` | Yes | `true` | Whether the OpenTelemetry exporter is enabled. |
355-
| `endpoint` | No | - | The OpenTelemetry collector endpoint. Protocol is inferred from the format (see examples below). |
356-
| `push_interval` | Yes | `60s` | How frequently metrics are pushed to the collector. Specify as a [duration](../duration). |
357-
| `metrics` | Yes | `[]` | List of metric names to export. When empty (default), all metrics are exported. |
352+
| Parameter name | Optional | Default | Description |
353+
| --------------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
354+
| `enabled` | Yes | `true` | Whether the OpenTelemetry exporter is enabled. |
355+
| `endpoint` | No | - | The OpenTelemetry collector endpoint. Protocol is inferred from the format (see examples below). |
356+
| `push_interval` | Yes | `60s` | How frequently metrics are pushed to the collector. Specify as a [duration](../duration). |
357+
| `metrics` | Yes | `[]` | List of metric names to export. When empty (default), all metrics are exported. |
358+
| `headers` | Yes | `{}` | Map of headers to send with each export request. For HTTP these are sent as HTTP headers; for gRPC they are sent as metadata entries (keys must be lowercase ASCII). Values support the `${secrets:...}` [replacement syntax](../../components/secret-stores#using-secrets) for loading credentials from a [secret store](../../components/secret-stores). |
358359

359360
**Protocol inference:**
360361

@@ -403,6 +404,43 @@ runtime:
403404
- dataset_load_state
404405
```
405406

407+
**Authenticated exporters:**
408+
409+
For collectors that require authentication, set the `headers` map. Load credentials from a [secret store](../../components/secret-stores) via `${secrets:...}` rather than committing them to source.
410+
411+
Datadog (OTLP/HTTP) — replace `us3` with your Datadog site:
412+
413+
```yaml
414+
runtime:
415+
telemetry:
416+
otel_exporter:
417+
endpoint: 'https://otlp.us3.datadoghq.com/v1/metrics'
418+
headers:
419+
DD-API-KEY: ${secrets:dd_api_key}
420+
```
421+
422+
Grafana Cloud (OTLP/HTTP) — use the base64 `instanceID:accessPolicyToken` from the Grafana Cloud OpenTelemetry connection page:
423+
424+
```yaml
425+
runtime:
426+
telemetry:
427+
otel_exporter:
428+
endpoint: 'https://otlp-gateway-us-central2.grafana.net/otlp/v1/metrics'
429+
headers:
430+
Authorization: 'Basic ${secrets:grafana_cloud_auth}'
431+
```
432+
433+
gRPC collector with auth metadata (keys must be lowercase ASCII):
434+
435+
```yaml
436+
runtime:
437+
telemetry:
438+
otel_exporter:
439+
endpoint: 'otel-collector.internal:4317'
440+
headers:
441+
api-key: ${secrets:collector_api_key}
442+
```
443+
406444
## `runtime.metrics`
407445

408446
Specifies metrics that are disabled by default.

0 commit comments

Comments
 (0)