Skip to content

Commit 4141f6f

Browse files
eric-thomas-dagsterclaude
eric-thomas-dagster
and
claude
committed
feat(compute_log_managers): OtlpComputeLogManager + HEC-vs-OTLP guidance (v0.10.17)
Third manager in compute_log_managers/: OtlpComputeLogManager. One OTLP/HTTP wire protocol, many destinations — Splunk (via Splunk OTel Collector), Datadog OTel ingest, Honeycomb, Sumo Logic OTel, Grafana Loki via OTel Collector, AWS CloudWatch via OTel Collector, New Relic, or any OTel Collector that forwards to anywhere. Why both HEC + OTLP for Splunk: - HEC is simpler/more direct: single hop, no Collector to operate, mature. - OTLP is portable + buffered: Collector absorbs backend outages, same dagster.yaml works against Datadog or Honeycomb by changing endpoint. - They coexist cleanly — Tee can write to both for belt-and-suspenders. Shape (parallels SplunkComputeLogManager): - TruncatingCloudStorageComputeLogManager subclass with ConfigurableClass. - Builds OTLP/HTTP JSON envelope: resourceLogs → scopeLogs → logRecords. - Resource attrs: service.name, service.instance.id. - Per-record attrs: dagster.run_id, dagster.step_key, dagster.io_type, dagster.partial. - Severity: stdout→9 (INFO), stderr→13 (WARN). Conservative on stderr because Dagster doesn't distinguish app errors from operational stderr. - display_url_template with {run_id} / {step_key} / {io_type} / {location} / {service} placeholders → UI surfaces "View logs →" deep-link per step. Vendor-specific examples in the README for Honeycomb / Datadog / Splunk Web / Grafana Loki. - download_from_cloud_storage writes a stub pointing at the destination (same UX as Splunk CLM — don't round-trip every UI viewer hit to the observability backend). Config knobs: - otlp_endpoint, otlp_headers (vendor-specific auth) - service_name, location_label - display_url_template - severity_stdout, severity_stderr (override for backends that route on severity, e.g. PagerDuty rules on ERROR-level logs) - verify_ssl, batch_size, request_timeout_seconds, skip_empty_files - local_dir, upload_interval Packaging: - New optional-dependencies extras: splunk = ["requests>=2.28"] # existed otlp = ["requests>=2.28"] # new compute-log-managers = ["requests>=2.28"] # new — covers both - Top-level compute_log_managers/README.md gains a "HEC direct vs OTLP" comparison table and updated manager inventory. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent cf41494 commit 4141f6f

6 files changed

Lines changed: 709 additions & 8 deletions

File tree

‎compute_log_managers/README.md‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,13 +16,23 @@ A Dagster instance has **exactly one** compute log manager. The default `LocalCo
1616
| Manager | What it does | When to use |
1717
|---|---|---|
1818
| [`splunk.SplunkComputeLogManager`](splunk/) | Streams op stdout/stderr to Splunk HEC. UI surfaces "View in Splunk →" deep-links per step. | Customers running Dagster OSS who want compute logs in Splunk and never touching Dagster+. Or Dagster+ customers who need a Splunk copy for compliance — compose via Tee. |
19+
| [`otlp.OtlpComputeLogManager`](otlp/) | Streams op stdout/stderr via OTLP/HTTP to any OTel-compatible backend (Splunk via Splunk OTel Collector, Datadog, Honeycomb, Sumo, Loki, CloudWatch, ...). | Customers who already run an OTel Collector or want vendor portability. One config swap = different backend. |
1920
| [`tee.TeeComputeLogManager`](tee/) | Composes N inner CLMs. Fan-out writes, first-success reads. | Sending to multiple destinations from one `dagster.yaml`. Common case: Splunk + Dagster+. |
2021

21-
Coming soon:
22+
### HEC direct vs OTLP — which Splunk path?
2223

23-
| Manager | What it'll do |
24-
|---|---|
25-
| `otlp.OtlpComputeLogManager` | OTLP/HTTP logs — works against any OTel Collector. Targets Splunk (via Splunk OTel Collector), Datadog, Honeycomb, Sumo, etc. through one wire protocol. |
24+
Both target Splunk. Different ergonomics:
25+
26+
| | HEC direct ([`splunk/`](splunk/)) | OTLP ([`otlp/`](otlp/)) → Splunk OTel Collector |
27+
|---|---|---|
28+
| Hops | Dagster → Splunk | Dagster → Collector → Splunk |
29+
| Setup | HEC token + endpoint | Operate an OTel Collector |
30+
| Reliability | Splunk down = events drop | Collector usually buffers + retries |
31+
| Vendor portability | Splunk-only | Multi-vendor (Splunk, Datadog, Honeycomb, Sumo, Loki, CloudWatch, ...) |
32+
| Field mapping | Direct → Splunk fields | Collector translates OTel semantic conventions |
33+
| Splunk's current docs | Still supported, mature | "Modern recommended path" |
34+
35+
**Use HEC** if Splunk is your only observability backend and adding an OTel Collector is unwelcome ops surface. **Use OTLP** if you already run a Collector for app traces/metrics, or want to keep `dagster.yaml` portable across vendors. They coexist cleanly — Tee can write to both at once if you want belt-and-suspenders.
2636

2737
## How this differs from the `audit_logs_to_*` sinks
2838

‎compute_log_managers/otlp/README.md‎

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
# OtlpComputeLogManager
2+
3+
Send Dagster op stdout/stderr to **any OpenTelemetry-compatible logs backend** via OTLP/HTTP (JSON). One wire protocol, many destinations:
4+
5+
- **Splunk** via [Splunk Distribution of OpenTelemetry Collector](https://docs.splunk.com/observability/en/gdi/opentelemetry/opentelemetry.html)
6+
- **Datadog** OTel ingest (`https://otlp-intake.logs.datadoghq.com`)
7+
- **Honeycomb** (`https://api.honeycomb.io`)
8+
- **Sumo Logic** OTel
9+
- **Grafana Loki** via OTel Collector
10+
- **AWS CloudWatch** via OTel Collector
11+
- **New Relic** (`https://otlp.nr-data.net`)
12+
- Any OTel Collector → forwards to anything
13+
14+
Compute log manager — sits at the Dagster **instance** layer (`dagster.yaml`), not the project layer.
15+
16+
## Why OTLP vs the dedicated `SplunkComputeLogManager`?
17+
18+
| | HEC direct (`splunk/`) | OTLP (`otlp/`, this manager) |
19+
|---|---|---|
20+
| Hops | Dagster → Splunk | Dagster → Collector → Splunk |
21+
| Setup | HEC token + endpoint | Operate an OTel Collector |
22+
| Reliability | Splunk down = events drop | Collector usually buffers + retries |
23+
| Vendor portability | Splunk-only | Splunk + Datadog + Honeycomb + Sumo + Loki + CloudWatch + ... |
24+
| Splunk's docs lean | "Still supported, mature" | "Modern recommended path" |
25+
26+
Pick OTLP if you already run an OTel Collector for app traces/metrics (Dagster slots into the existing pipeline), or if you need vendor portability. Pick HEC if Splunk is your only observability backend and adding a Collector is unwelcome ops surface.
27+
28+
## What lands at the backend
29+
30+
One OTLP `LogRecord` per line of captured stdout/stderr:
31+
32+
```json
33+
{
34+
"resourceLogs": [{
35+
"resource": {
36+
"attributes": [
37+
{"key": "service.name", "value": {"stringValue": "dagster"}},
38+
{"key": "service.instance.id", "value": {"stringValue": "prod-east"}}
39+
]
40+
},
41+
"scopeLogs": [{
42+
"scope": {"name": "dagster_community_components.compute_log_managers.otlp"},
43+
"logRecords": [
44+
{
45+
"timeUnixNano": "1748534567000000000",
46+
"severityNumber": 9,
47+
"severityText": "INFO",
48+
"body": {"stringValue": "INFO - my asset processed 1234 rows"},
49+
"attributes": [
50+
{"key": "dagster.run_id", "value": {"stringValue": "abc-123-uuid"}},
51+
{"key": "dagster.step_key", "value": {"stringValue": "process_orders"}},
52+
{"key": "dagster.io_type", "value": {"stringValue": "stdout"}},
53+
{"key": "dagster.partial", "value": {"stringValue": "false"}}
54+
]
55+
}
56+
]
57+
}]
58+
}]
59+
}
60+
```
61+
62+
Severity: stdout → 9 (INFO), stderr → 13 (WARN) by default. **Not ERROR** for stderr because Dagster doesn't distinguish app errors from operational stderr (most Python loggers write everything there). Override per-backend via `severity_stdout` / `severity_stderr` if your observability strategy treats stderr differently.
63+
64+
## UI behavior
65+
66+
Per step, the Dagster UI shows a "View logs →" button that opens the URL from your `display_url_template` with `{run_id}` / `{step_key}` / `{io_type}` / `{location}` / `{service}` substituted. If `display_url_template` is unset, no deep-link is rendered (UI shows the stub message only).
67+
68+
Example templates per backend:
69+
70+
```yaml
71+
# Honeycomb
72+
display_url_template: "https://ui.honeycomb.io/acme/datasets/dagster_compute_logs?query=dagster.run_id%3D{run_id}"
73+
74+
# Datadog (logs query syntax)
75+
display_url_template: "https://app.datadoghq.com/logs?query=%40dagster.run_id%3A{run_id}"
76+
77+
# Splunk Web (via OTel Collector)
78+
display_url_template: "https://splunk.acme.com:8000/en-US/app/search/search?q=search%20dagster.run_id%3D%22{run_id}%22"
79+
80+
# Grafana Loki
81+
display_url_template: "https://grafana.acme.com/explore?left=%7B%22queries%22%3A%5B%7B%22expr%22%3A%22%7Bservice_name%3D%5C%22{service}%5C%22%7D%20%7C%3D%20%5C%22{run_id}%5C%22%22%7D%5D%7D"
82+
```
83+
84+
## Setup
85+
86+
### 1. Stand up an OTel Collector (or use a hosted one)
87+
88+
Most observability vendors offer a hosted OTLP endpoint. Examples:
89+
90+
| Vendor | Endpoint | Headers |
91+
|---|---|---|
92+
| Honeycomb | `https://api.honeycomb.io` | `x-honeycomb-team`, `x-honeycomb-dataset` |
93+
| Datadog (US1) | `https://otlp-intake.logs.datadoghq.com` | `DD-API-KEY` |
94+
| New Relic (US) | `https://otlp.nr-data.net` | `api-key` |
95+
| Splunk Observability Cloud | `https://ingest.us0.signalfx.com/v2/log` (NB: SignalFx path) | `X-SF-Token` |
96+
| Splunk Enterprise via OTC | `http://your-otc:4318` (no auth typical) | (intra-cluster) |
97+
98+
If you're running your own Collector, the receiver config looks like:
99+
100+
```yaml
101+
# otel-collector-config.yaml
102+
receivers:
103+
otlp:
104+
protocols:
105+
http:
106+
endpoint: 0.0.0.0:4318
107+
108+
exporters:
109+
splunk_hec:
110+
token: ${SPLUNK_HEC_TOKEN}
111+
endpoint: https://splunk.acme.com:8088/services/collector
112+
113+
service:
114+
pipelines:
115+
logs:
116+
receivers: [otlp]
117+
exporters: [splunk_hec]
118+
```
119+
120+
### 2. Add to `dagster.yaml`
121+
122+
```yaml
123+
compute_logs:
124+
module: dagster_community_components.compute_log_managers.otlp
125+
class: OtlpComputeLogManager
126+
config:
127+
otlp_endpoint: http://otel-collector.svc:4318
128+
service_name: dagster
129+
location_label: prod-east
130+
display_url_template: "https://splunk.acme.com:8000/en-US/app/search/search?q=search%20dagster.run_id%3D%22{run_id}%22"
131+
upload_interval: 30
132+
```
133+
134+
Full annotated config in [`example_dagster.yaml`](example_dagster.yaml).
135+
136+
### 3. Restart
137+
138+
Daemon + webserver + any user-code containers pick up `dagster.yaml` at startup.
139+
140+
## Config reference
141+
142+
| Field | Type | Default | Description |
143+
|---|---|---|---|
144+
| `otlp_endpoint` | str (env) | — | Base URL. `/v1/logs` is appended. |
145+
| `otlp_headers` | dict[str, str] | `{}` | Headers added to every OTLP POST (vendor-specific auth). |
146+
| `service_name` | str (env) | `dagster` | OTel `service.name` resource attribute. |
147+
| `location_label` | str (env) | `default` | OTel `service.instance.id` resource attribute. |
148+
| `display_url_template` | str (env) | none | URL template for the Dagster UI deep-link. Supports `{run_id}` / `{step_key}` / `{io_type}` / `{location}` / `{service}` placeholders. |
149+
| `severity_stdout` | int | `9` (INFO) | OTel severity number for stdout lines. |
150+
| `severity_stderr` | int | `13` (WARN) | OTel severity number for stderr lines. |
151+
| `verify_ssl` | bool | `true` | TLS cert verification. |
152+
| `batch_size` | int | `100` | LogRecords per OTLP POST. |
153+
| `request_timeout_seconds` | int | `30` | HTTP timeout per POST. |
154+
| `skip_empty_files` | bool | `false` | Skip uploads when local file is empty. |
155+
| `local_dir` | str (env) | system temp | Local capture directory. |
156+
| `upload_interval` | int \| null | `null` | Partial-upload interval in seconds. `null` = upload on step finish only. |
157+
158+
## Sending to OTLP AND Dagster+
159+
160+
Wrap in [`TeeComputeLogManager`](../tee/):
161+
162+
```yaml
163+
compute_logs:
164+
module: dagster_community_components.compute_log_managers.tee
165+
class: TeeComputeLogManager
166+
config:
167+
local_dir: /tmp/dagster_compute_logs
168+
display_manager_index: 0 # OTLP URL surfaces in UI
169+
managers:
170+
- module: dagster_community_components.compute_log_managers.otlp
171+
class: OtlpComputeLogManager
172+
config:
173+
otlp_endpoint: http://otel-collector.svc:4318
174+
display_url_template: "..."
175+
- module: dagster_cloud.storage.compute_logs
176+
class: CloudComputeLogManager
177+
config: {}
178+
```
179+
180+
## Cost / performance notes
181+
182+
- **Latency**: OTLP/HTTP JSON over HTTPS — single round-trip per batch, ~5–50ms typical to a local Collector, ~50–200ms to hosted vendors. Set `batch_size` higher (e.g. 500) for chatty workloads to reduce HTTP overhead.
183+
- **Reliability**: Collector failure modes vary. Most Collectors buffer locally and retry to the backend, so transient backend outages are absorbed. If your Collector is misconfigured or down, OTLP POSTs raise and Dagster's CLM machinery logs the error — the local file stays around but isn't auto-retried by this manager.
184+
- **Cardinality**: each log line has the same low-cardinality attributes (`dagster.run_id`, `dagster.step_key`, `dagster.io_type`). Won't blow up tag-based pricing systems like Datadog or Honeycomb at sensible volumes.
185+
- **Severity tagging**: stderr defaults to WARN, not ERROR. If your observability strategy treats stderr=ERROR (PagerDuty rules, etc.), bump `severity_stderr: 17` — but be aware Dagster + Python's `logging` write a lot of non-error content to stderr by default.
186+
187+
## See also
188+
189+
- [`splunk/`](../splunk/) — Splunk HEC direct (no Collector required, single hop)
190+
- [`tee/`](../tee/) — compose OTLP with Dagster+ or any other CLM
191+
- [OTLP/HTTP spec — logs](https://opentelemetry.io/docs/specs/otlp/#otlphttp-logs-binary-protocol)
192+
- [Splunk Distribution of OpenTelemetry Collector](https://docs.splunk.com/observability/en/gdi/opentelemetry/opentelemetry.html)
193+
- `dataframe_to_otlp_logs` (in `assets/sinks/`) — distinct: ships data-plane DataFrames as OTLP logs, not compute logs
194+
- `dagster_failures_to_otlp_sensor` (in `sensors/`) — distinct: ships Dagster run-failure events to OTLP, not compute logs

‎compute_log_managers/otlp/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
from .compute_log_manager import OtlpComputeLogManager
2+
3+
__all__ = ["OtlpComputeLogManager"]

0 commit comments

Comments
 (0)