Skip to content

Commit a170b53

Browse files
futurepastorifuturepastori
committed
update with latest schemas
1 parent 6397a98 commit a170b53

8 files changed

+281
-79
lines changed

‎README.md‎

Lines changed: 93 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,93 @@
1-
# Tinybird OTEL Template
2-
3-
This repository is a template for building Tinybird projects that leverage [OpenTelemetry](https://opentelemetry.io/) (OTEL) for observability and analytics. Use this project as a starting point to instrument your applications with OpenTelemetry, collect metrics, traces, and logs, and analyze or expose them efficiently via Tinybird.
4-
5-
## Overview
6-
- **Project type:** Tinybird SRC repository
7-
- **Purpose:** Enable rapid ingestion and analysis of OpenTelemetry data using Tinybird
8-
- **Contents:**
9-
- `.datasource` files defining Tinybird tables for OTEL metrics, logs, and traces
10-
- Folders for connections, pipes, endpoints, and materializations
11-
12-
## How to Use
13-
1. **Fork/clone this repository as your Tinybird project root.**
14-
2. **Instrument your services using OpenTelemetry SDKs.**
15-
3. **Forward OTEL metrics, logs, and traces to this project's datasources (using the OpenTelemetry Collector, for example).**
16-
4. **Create and publish Tinybird pipes and endpoints as needed for analytics and querying.**
17-
18-
## Typical Structure
19-
- `datasources/`: Definitions for raw OTEL data ingestion
20-
- `pipes/`: Transform/query data
21-
- `endpoints/`: Expose analytics via API
22-
- Other folders: `connections/`, `copies/`, `fixtures/`, `materializations/`, `tests/`
23-
24-
## Getting Started
25-
- See Tinybird documentation: [docs.tinybird.co](https://docs.tinybird.co/)
26-
- See OpenTelemetry docs: [opentelemetry.io/docs/](https://opentelemetry.io/docs/)
27-
28-
## Who is this for?
29-
Developers and SREs who want plug-and-play observability using OTEL and analytical capabilities via Tinybird.
30-
31-
---
32-
Want to contribute or extend? Open issues or submit PRs!
1+
# Ingest from OpenTelemetry
2+
3+
[OpenTelemetry](https://opentelemetry.io/) is an open-source observability framework for collecting, processing, and exporting telemetry data (metrics, traces, and logs) from your applications and infrastructure.
4+
5+
By integrating OpenTelemetry with Tinybird, you can analyze observability data in real time, build dashboards, and enrich it with other data sources.
6+
7+
Some common use cases for sending OpenTelemetry data to Tinybird include:
8+
9+
1. Centralizing metrics, traces, and logs for unified analytics.
10+
2. Building custom dashboards and alerts on top of observability data.
11+
3. Enriching telemetry with business or application data.
12+
13+
Read on to learn how to send data from OpenTelemetry to Tinybird.
14+
15+
## Before you start
16+
17+
Before you connect OpenTelemetry to Tinybird, ensure:
18+
19+
* You have a Tinybird workspace.
20+
* You have a Tinybird Token with **append** permissions to the target Data Sources.
21+
* You are running the Tinybird distribution of the OpenTelemetry Collector.
22+
23+
You can find the latest release of the Tinybird OpenTelemetry Collector here:
24+
25+
- [GitHub Releases](https://github.com/tinybirdco/opentelemetry-collector-contrib/releases)
26+
- [Docker Hub](https://hub.docker.com/r/tinybirdco/opentelemetry-collector-contrib/tags)
27+
28+
## Use the Tinybird OpenTelemetry project template
29+
30+
To get started quickly, you can use the [Tinybird OpenTelemetry project template](https://github.com/tinybirdco/tinybird-otel-template). This template provides ready-to-use Data Sources and Pipes for storing and analyzing your telemetry data in Tinybird.
31+
32+
## Minimal OpenTelemetry Collector configuration
33+
34+
Below is a minimal example configuration for the Tinybird OpenTelemetry Collector to export metrics, traces, and logs to Tinybird:
35+
36+
```yaml
37+
receivers:
38+
otlp:
39+
protocols:
40+
grpc:
41+
http:
42+
43+
processors:
44+
batch:
45+
timeout: 10s
46+
send_batch_size: 8192
47+
48+
exporters:
49+
tinybird:
50+
endpoint: ${OTEL_TINYBIRD_API_HOST}
51+
token: ${OTEL_TINYBIRD_TOKEN}
52+
metrics_sum:
53+
datasource: opentelemetry_metrics_sum
54+
metrics_histogram:
55+
datasource: opentelemetry_metrics_histogram
56+
metrics_exponential_histogram:
57+
datasource: opentelemetry_metrics_exponential_histogram
58+
metrics_gauge:
59+
datasource: opentelemetry_metrics_gauge
60+
traces_datasource: otel_traces
61+
logs_datasource: otel_logs
62+
63+
service:
64+
pipelines:
65+
metrics:
66+
receivers: [otlp]
67+
processors: [batch]
68+
exporters: [tinybird]
69+
70+
traces:
71+
receivers: [otlp]
72+
processors: [batch]
73+
exporters: [tinybird]
74+
75+
logs:
76+
receivers: [otlp]
77+
processors: [batch]
78+
exporters: [tinybird]
79+
```
80+
81+
### Environment variables
82+
83+
- `OTEL_TINYBIRD_API_HOST`: The API host for your Tinybird workspace (e.g., `https://api.tinybird.co`).
84+
- `OTEL_TINYBIRD_TOKEN`: A Tinybird token with **append** permissions to the target Data Sources (`otel_metrics`, `otel_traces`, `otel_logs`).
85+
86+
You can create a token in the Tinybird UI under **Tokens**. Make sure it has the required append permissions for the Data Sources you want to ingest into.
87+
88+
## Next steps
89+
90+
- Explore and customize the [Tinybird OpenTelemetry project template](https://github.com/tinybirdco/tinybird-otel-template) to fit your needs.
91+
- Use the ingested data to build real-time analytics, dashboards, and alerts in Tinybird.
92+
93+
For more details on the available configuration options, see the [Tinybird OpenTelemetry Collector documentation](https://github.com/tinybirdco/opentelemetry-collector-contrib).

‎datasources/otel_logs.datasource‎

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,17 @@ DESCRIPTION >
55

66
SCHEMA >
77
`Timestamp` DateTime64(9) `json:$.timestamp` CODEC(Delta(8), ZSTD(1)),
8-
`TimestampTime` DateTime `json:$.timestamp`,
8+
`TimestampTime` DateTime `json:$.timestamp`,
99
`TraceId` String `json:$.trace_id` CODEC(ZSTD(1)),
1010
`SpanId` String `json:$.span_id` CODEC(ZSTD(1)),
1111
`TraceFlags` UInt8 `json:$.flags` CODEC(ZSTD(1)),
1212
`SeverityText` LowCardinality(String) `json:$.severity_text` CODEC(ZSTD(1)),
1313
`SeverityNumber` UInt8 `json:$.severity_number` CODEC(ZSTD(1)),
1414
`ServiceName` LowCardinality(String) `json:$.service_name` CODEC(ZSTD(1)),
1515
`Body` String `json:$.body` CODEC(ZSTD(1)),
16-
`ResourceSchemaUrl` LowCardinality(String) `json:$.resource_schema_url` CODEC(ZSTD(1)),
16+
`ResourceSchemaUrl` String `json:$.resource_schema_url` CODEC(ZSTD(1)),
1717
`ResourceAttributes` Map(LowCardinality(String), String) `json:$.resource_attributes` CODEC(ZSTD(1)),
18-
`ScopeSchemaUrl` LowCardinality(String) `json:$.scope_schema_url` CODEC(ZSTD(1)),
18+
`ScopeSchemaUrl` String `json:$.scope_schema_url` CODEC(ZSTD(1)),
1919
`ScopeName` String `json:$.scope_name` CODEC(ZSTD(1)),
2020
`ScopeVersion` String `json:$.scope_version` CODEC(ZSTD(1)),
2121
`ScopeAttributes` Map(LowCardinality(String), String) `json:$.scope_attributes` CODEC(ZSTD(1)),
@@ -24,15 +24,14 @@ SCHEMA >
2424
ENGINE "MergeTree"
2525
ENGINE_PARTITION_KEY "toDate(TimestampTime)"
2626
ENGINE_SORTING_KEY "ServiceName, TimestampTime, Timestamp"
27-
ENGINE_TTL "toDateTime(Timestamp) + toIntervalDay(3)"
2827
ENGINE_SETTINGS index_granularity=8192
2928

3029
INDEXES >
3130
idx_trace_id TraceId TYPE bloom_filter(0.001) GRANULARITY 1,
3231
idx_res_attr_key mapKeys(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
33-
idx_res_attr_value mapValues(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
34-
idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
35-
idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
36-
idx_log_attr_key mapKeys(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
37-
idx_log_attr_value mapValues(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
38-
idx_body Body TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8
32+
idx_res_attr_value mapValues(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
33+
idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
34+
idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
35+
idx_log_attr_key mapKeys(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
36+
idx_log_attr_value mapValues(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
37+
idx_body Body TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8

‎datasources/otel_metrics.datasource‎

Lines changed: 0 additions & 34 deletions
This file was deleted.

‎datasources/otel_metrics_exponential_histogram.datasource‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
TOKEN OTEL_TINYBIRD_TOKEN APPEND
2+
3+
DESCRIPTION >
4+
This is a table that contains the metrics from the OpenTelemetry Collector.
5+
6+
SCHEMA >
7+
`ResourceAttributes` Map(LowCardinality(String), String) `json:$.resource_attributes` CODEC(ZSTD(1)),
8+
`ResourceSchemaUrl` String `json:$.resource_schema_url` CODEC(ZSTD(1)),
9+
`ScopeName` String `json:$.scope_name` CODEC(ZSTD(1)),
10+
`ScopeVersion` String `json:$.scope_version` CODEC(ZSTD(1)),
11+
`ScopeAttributes` Map(LowCardinality(String), String) `json:$.scope_attributes` CODEC(ZSTD(1)),
12+
`ScopeSchemaUrl` String `json:$.scope_schema_url` CODEC(ZSTD(1)),
13+
`ServiceName` String `json:$.service_name` DEFAULT 'unknown' CODEC(ZSTD(1)),
14+
`MetricName` LowCardinality(String) `json:$.metric_name` CODEC(ZSTD(1)),
15+
`MetricDescription` String `json:$.metric_description` CODEC(ZSTD(1)),
16+
`MetricUnit` String `json:$.metric_unit` CODEC(ZSTD(1)),
17+
`Attributes` Map(LowCardinality(String), String) `json:$.metric_attributes` CODEC(ZSTD(1)),
18+
`StartTimeUnix` DateTime64(9) `json:$.start_timestamp` CODEC(Delta(8), ZSTD(1)),
19+
`TimeUnix` DateTime64(9) `json:$.timestamp` CODEC(Delta(8), ZSTD(1)),
20+
`Count` UInt64 `json:$.count` CODEC(Delta, ZSTD(1)),
21+
`Sum` Float64 `json:$.sum` CODEC(ZSTD(1)),
22+
`Scale` Int32 `json:$.scale` CODEC(ZSTD(1)),
23+
`ZeroCount` UInt64 `json:$.zero_count` CODEC(ZSTD(1)),
24+
`PositiveOffset` Int32 `json:$.positive_offset` CODEC(ZSTD(1)),
25+
`PositiveBucketCounts` Array(UInt64) `json:$.positive_bucket_counts[:]` CODEC(ZSTD(1)),
26+
`NegativeOffset` Int32 `json:$.negative_offset` CODEC(ZSTD(1)),
27+
`NegativeBucketCounts` Array(UInt64) `json:$.negative_bucket_counts[:]` CODEC(ZSTD(1)),
28+
`ExemplarsTraceId` Array(String) `json:$.exemplars_trace_id[:]` CODEC(ZSTD(1)),
29+
`ExemplarsSpanId` Array(String) `json:$.exemplars_span_id[:]` CODEC(ZSTD(1)),
30+
`ExemplarsTimestamp` Array(DateTime64(9)) `json:$.exemplars_timestamp[:]` CODEC(ZSTD(1)),
31+
`ExemplarsValue` Array(Float64) `json:$.exemplars_value[:]` CODEC(ZSTD(1)),
32+
`ExemplarsFilteredAttributes` Array(Map(LowCardinality(String), String)) `json:$.exemplars_filtered_attributes[:]` CODEC(ZSTD(1)),
33+
`Flags` UInt8 `json:$.flags` CODEC(ZSTD(1)),
34+
`Min` Nullable(Float64) `json:$.min` CODEC(ZSTD(1)),
35+
`Max` Nullable(Float64) `json:$.max` CODEC(ZSTD(1)),
36+
`AggregationTemporality` Int32 `json:$.aggregation_temporality` CODEC(ZSTD(1))
37+
38+
ENGINE "MergeTree"
39+
ENGINE_PARTITION_KEY "toDate(TimeUnix)"
40+
ENGINE_SORTING_KEY "ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix)"
41+
ENGINE_SETTINGS index_granularity=8192
42+
43+
INDEXES >
44+
idx_res_attr_key mapKeys(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
45+
idx_res_attr_value mapValues(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
46+
idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
47+
idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
48+
idx_attr_key mapKeys(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
49+
idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1

‎datasources/otel_metrics_gauge.datasource‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
TOKEN OTEL_TINYBIRD_TOKEN APPEND
2+
3+
DESCRIPTION >
4+
This is a table that contains the metrics from the OpenTelemetry Collector.
5+
6+
SCHEMA >
7+
`ResourceAttributes` Map(LowCardinality(String), String) `json:$.resource_attributes` CODEC(ZSTD(1)),
8+
`ResourceSchemaUrl` String `json:$.resource_schema_url` CODEC(ZSTD(1)),
9+
`ScopeName` String `json:$.scope_name` CODEC(ZSTD(1)),
10+
`ScopeVersion` String `json:$.scope_version` CODEC(ZSTD(1)),
11+
`ScopeAttributes` Map(LowCardinality(String), String) `json:$.scope_attributes` CODEC(ZSTD(1)),
12+
`ScopeSchemaUrl` String `json:$.scope_schema_url` CODEC(ZSTD(1)),
13+
`ServiceName` String `json:$.service_name` DEFAULT 'unknown' CODEC(ZSTD(1)),
14+
`MetricName` LowCardinality(String) `json:$.metric_name` CODEC(ZSTD(1)),
15+
`MetricDescription` String `json:$.metric_description` CODEC(ZSTD(1)),
16+
`MetricUnit` String `json:$.metric_unit` CODEC(ZSTD(1)),
17+
`Attributes` Map(LowCardinality(String), String) `json:$.metric_attributes` CODEC(ZSTD(1)),
18+
`StartTimeUnix` DateTime64(9) `json:$.start_timestamp` CODEC(Delta(8), ZSTD(1)),
19+
`TimeUnix` DateTime64(9) `json:$.timestamp` CODEC(Delta(8), ZSTD(1)),
20+
`Value` Float64 `json:$.value` CODEC(ZSTD(1)),
21+
`Flags` UInt8 `json:$.flags` CODEC(ZSTD(1)),
22+
`ExemplarsTraceId` Array(String) `json:$.exemplars_trace_id[:]` CODEC(ZSTD(1)),
23+
`ExemplarsSpanId` Array(String) `json:$.exemplars_span_id[:]` CODEC(ZSTD(1)),
24+
`ExemplarsTimestamp` Array(DateTime64(9)) `json:$.exemplars_timestamp[:]` CODEC(ZSTD(1)),
25+
`ExemplarsValue` Array(Float64) `json:$.exemplars_value[:]` CODEC(ZSTD(1)),
26+
`ExemplarsFilteredAttributes` Array(Map(LowCardinality(String), String)) `json:$.exemplars_filtered_attributes[:]` CODEC(ZSTD(1))
27+
28+
ENGINE "MergeTree"
29+
ENGINE_PARTITION_KEY "toDate(TimeUnix)"
30+
ENGINE_SORTING_KEY "ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix)"
31+
ENGINE_SETTINGS index_granularity=8192
32+
33+
INDEXES >
34+
idx_res_attr_key mapKeys(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
35+
idx_res_attr_value mapValues(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
36+
idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
37+
idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
38+
idx_attr_key mapKeys(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
39+
idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1

‎datasources/otel_metrics_histogram.datasource‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
TOKEN OTEL_TINYBIRD_TOKEN APPEND
2+
3+
DESCRIPTION >
4+
This is a table that contains the metrics from the OpenTelemetry Collector.
5+
6+
SCHEMA >
7+
`ResourceAttributes` Map(LowCardinality(String), String) `json:$.resource_attributes` CODEC(ZSTD(1)),
8+
`ResourceSchemaUrl` String `json:$.resource_schema_url` CODEC(ZSTD(1)),
9+
`ScopeName` String `json:$.scope_name` CODEC(ZSTD(1)),
10+
`ScopeVersion` String `json:$.scope_version` CODEC(ZSTD(1)),
11+
`ScopeAttributes` Map(LowCardinality(String), String) `json:$.scope_attributes` CODEC(ZSTD(1)),
12+
`ScopeSchemaUrl` String `json:$.scope_schema_url` CODEC(ZSTD(1)),
13+
`ServiceName` String `json:$.service_name` DEFAULT 'unknown' CODEC(ZSTD(1)),
14+
`MetricName` LowCardinality(String) `json:$.metric_name` CODEC(ZSTD(1)),
15+
`MetricDescription` String `json:$.metric_description` CODEC(ZSTD(1)),
16+
`MetricUnit` String `json:$.metric_unit` CODEC(ZSTD(1)),
17+
`Attributes` Map(LowCardinality(String), String) `json:$.metric_attributes` CODEC(ZSTD(1)),
18+
`StartTimeUnix` DateTime64(9) `json:$.start_timestamp` CODEC(Delta(8), ZSTD(1)),
19+
`TimeUnix` DateTime64(9) `json:$.timestamp` CODEC(Delta(8), ZSTD(1)),
20+
`Count` UInt64 `json:$.count` CODEC(Delta, ZSTD(1)),
21+
`Sum` Float64 `json:$.sum` CODEC(ZSTD(1)),
22+
`BucketCounts` Array(UInt64) `json:$.bucket_counts[:]` CODEC(ZSTD(1)),
23+
`ExplicitBounds` Array(Float64) `json:$.explicit_bounds[:]` CODEC(ZSTD(1)),
24+
`ExemplarsTraceId` Array(String) `json:$.exemplars_trace_id[:]` CODEC(ZSTD(1)),
25+
`ExemplarsSpanId` Array(String) `json:$.exemplars_span_id[:]` CODEC(ZSTD(1)),
26+
`ExemplarsTimestamp` Array(DateTime64(9)) `json:$.exemplars_timestamp[:]` CODEC(ZSTD(1)),
27+
`ExemplarsValue` Array(Float64) `json:$.exemplars_value[:]` CODEC(ZSTD(1)),
28+
`ExemplarsFilteredAttributes` Array(Map(LowCardinality(String), String)) `json:$.exemplars_filtered_attributes[:]` CODEC(ZSTD(1)),
29+
`Flags` UInt8 `json:$.flags` CODEC(ZSTD(1)),
30+
`Min` Nullable(Float64) `json:$.min` CODEC(ZSTD(1)),
31+
`Max` Nullable(Float64) `json:$.max` CODEC(ZSTD(1)),
32+
`AggregationTemporality` Int32 `json:$.aggregation_temporality` CODEC(ZSTD(1))
33+
34+
ENGINE "MergeTree"
35+
ENGINE_PARTITION_KEY "toDate(TimeUnix)"
36+
ENGINE_SORTING_KEY "ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix)"
37+
ENGINE_SETTINGS index_granularity=8192
38+
39+
INDEXES >
40+
idx_res_attr_key mapKeys(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
41+
idx_res_attr_value mapValues(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
42+
idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
43+
idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
44+
idx_attr_key mapKeys(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
45+
idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1

0 commit comments

Comments
 (0)