|
| 1 | +# Traditional Signals: AWS VPC Flow Logs to ClickHouse via OpenTelemetry |
| 2 | + |
| 3 | +A companion example for the blog post [Before There Was OTel, There Were Signals](post.md). This directory contains a fully runnable pipeline that reads AWS VPC Flow Logs from S3, transforms them into OTLP trace spans, and ships them to ClickHouse. |
| 4 | + |
| 5 | +## What's in here |
| 6 | + |
| 7 | +``` |
| 8 | +trad-signals/ |
| 9 | +├── post.md # The blog post |
| 10 | +├── flowlogconnector/ # Custom OTel Collector connector (Go) |
| 11 | +│ ├── factory.go # Component factory registration |
| 12 | +│ ├── config.go # Config struct |
| 13 | +│ ├── connector.go # Core logic: logs → traces |
| 14 | +│ ├── fieldmap.go # AWS field name → OTel attribute mapping |
| 15 | +│ ├── go.mod |
| 16 | +│ └── go.sum |
| 17 | +├── builder-config.yaml # OCB builder manifest |
| 18 | +├── otel-collector-config.yaml # Collector pipeline config (reference) |
| 19 | +├── Dockerfile # Multi-stage build for the custom collector |
| 20 | +├── Dockerfile.seed # Seed image (mc + test data) |
| 21 | +├── helm/ # Helm chart for Kubernetes deployment |
| 22 | +│ ├── Chart.yaml |
| 23 | +│ ├── values.yaml |
| 24 | +│ └── templates/ |
| 25 | +└── testdata/ |
| 26 | + ├── vpc-flowlogs-v2.log(.gz) # Sample v2 flow log (14 fields) |
| 27 | + └── vpc-flowlogs-v5.log(.gz) # Sample v5 flow log (29 fields) |
| 28 | +``` |
| 29 | + |
| 30 | +## Prerequisites |
| 31 | + |
| 32 | +- **Kubernetes cluster** (local: [kind](https://kind.sigs.k8s.io/), [minikube](https://minikube.sigs.k8s.io/), or [Docker Desktop](https://docs.docker.com/desktop/kubernetes/); remote: any cluster with `kubectl` access) |
| 33 | +- **Helm 3** |
| 34 | +- **Docker** (for building the collector and seed images) |
| 35 | +- **Go 1.25+** (only if building the collector locally outside Docker) |
| 36 | + |
| 37 | +## Quick Start (Helm) |
| 38 | + |
| 39 | +### 1. Build the container images |
| 40 | + |
| 41 | +From the `trad-signals/` directory: |
| 42 | + |
| 43 | +```bash |
| 44 | +docker build -t flowlog-collector:latest . |
| 45 | +docker build -f Dockerfile.seed -t flowlog-seed:latest . |
| 46 | +``` |
| 47 | + |
| 48 | +If you're using **kind**, load the images into the cluster: |
| 49 | + |
| 50 | +```bash |
| 51 | +kind load docker-image flowlog-collector:latest |
| 52 | +kind load docker-image flowlog-seed:latest |
| 53 | +``` |
| 54 | + |
| 55 | +For **minikube**: |
| 56 | + |
| 57 | +```bash |
| 58 | +minikube image load flowlog-collector:latest |
| 59 | +minikube image load flowlog-seed:latest |
| 60 | +``` |
| 61 | + |
| 62 | +### 2. Install the chart |
| 63 | + |
| 64 | +```bash |
| 65 | +helm install trad-signals ./helm |
| 66 | +``` |
| 67 | + |
| 68 | +This deploys four components: |
| 69 | + |
| 70 | +| Resource | Kind | Purpose | |
| 71 | +|----------|------|---------| |
| 72 | +| **minio** | StatefulSet | S3-compatible object store | |
| 73 | +| **seed** | Job | Seeds sample flow log files into MinIO, then exits | |
| 74 | +| **clickhouse** | StatefulSet | Analytics database | |
| 75 | +| **collector** | Deployment | Custom OTel Collector | |
| 76 | + |
| 77 | +The collector reads the seeded flow logs from MinIO, transforms each line into an OTLP trace span, and writes them to ClickHouse. The `debug` exporter also prints every span to stdout so you can see them immediately. |
| 78 | + |
| 79 | +### 3. Verify the data landed |
| 80 | + |
| 81 | +Open a ClickHouse client against the running pod: |
| 82 | + |
| 83 | +```bash |
| 84 | +kubectl exec -it $(kubectl get pod -l "app.kubernetes.io/component=clickhouse,app.kubernetes.io/instance=trad-signals" -o jsonpath='{.items[0].metadata.name}') -- clickhouse-client |
| 85 | +``` |
| 86 | + |
| 87 | +Then query: |
| 88 | + |
| 89 | +```sql |
| 90 | +SELECT |
| 91 | + Timestamp, |
| 92 | + SpanAttributes['source.address'] AS src, |
| 93 | + SpanAttributes['destination.address'] AS dst, |
| 94 | + SpanAttributes['source.port'] AS src_port, |
| 95 | + SpanAttributes['destination.port'] AS dst_port, |
| 96 | + SpanAttributes['network.transport'] AS proto, |
| 97 | + SpanAttributes['security.action'] AS action |
| 98 | +FROM telemetry.flow_traces |
| 99 | +ORDER BY Timestamp |
| 100 | +LIMIT 20; |
| 101 | +``` |
| 102 | + |
| 103 | +### Tear down |
| 104 | + |
| 105 | +```bash |
| 106 | +helm uninstall trad-signals |
| 107 | +kubectl delete pvc -l app.kubernetes.io/instance=trad-signals |
| 108 | +``` |
| 109 | + |
| 110 | +## Building Locally (without Kubernetes) |
| 111 | + |
| 112 | +If you want to iterate on the connector code without rebuilding images each time. |
| 113 | + |
| 114 | +### 1. Install the OTel Collector Builder |
| 115 | + |
| 116 | +```bash |
| 117 | +go install go.opentelemetry.io/collector/cmd/builder@v0.120.0 |
| 118 | +``` |
| 119 | + |
| 120 | +### 2. Build the collector |
| 121 | + |
| 122 | +From the `trad-signals/` directory: |
| 123 | + |
| 124 | +```bash |
| 125 | +builder --config builder-config.yaml |
| 126 | +``` |
| 127 | + |
| 128 | +This generates the `./build/` directory and compiles a `flowlog-collector` binary into it. The `build/` directory is a generated artifact and is safe to delete; rerun `builder --config builder-config.yaml` anytime to recreate it. The `replaces` directive in `builder-config.yaml` resolves the `flowlogconnector` module from the local directory. |
| 129 | + |
| 130 | +### 3. Run the collector |
| 131 | + |
| 132 | +You still need MinIO and ClickHouse running. Install the Helm chart but skip the collector: |
| 133 | + |
| 134 | +```bash |
| 135 | +helm install infra ./helm --set collector.image.tag=unused --set seed.enabled=true |
| 136 | +``` |
| 137 | + |
| 138 | +Wait for the seed job to finish, then forward the required ports: |
| 139 | + |
| 140 | +```bash |
| 141 | +kubectl port-forward svc/trad-signals-minio 9000:9000 & |
| 142 | +kubectl port-forward svc/trad-signals-clickhouse 8123:8123 9000:9000 & |
| 143 | +``` |
| 144 | + |
| 145 | +Run the collector locally, pointing it at the forwarded services: |
| 146 | + |
| 147 | +```bash |
| 148 | +AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin \ |
| 149 | + ./build/flowlog-collector --config otel-collector-config.yaml |
| 150 | +``` |
| 151 | + |
| 152 | +## Helm Values |
| 153 | + |
| 154 | +Key values you can override (see `helm/values.yaml` for the full list): |
| 155 | + |
| 156 | +| Value | Default | Description | |
| 157 | +|-------|---------|-------------| |
| 158 | +| `collector.image.repository` | `flowlog-collector` | Collector container image | |
| 159 | +| `collector.starttime` | `2026-01-01` | S3 time-range start | |
| 160 | +| `collector.endtime` | `2026-01-02` | S3 time-range end | |
| 161 | +| `collector.s3.region` | `us-east-1` | S3 / MinIO region | |
| 162 | +| `collector.s3.bucket` | `my-vpc-flow-logs` | S3 bucket name | |
| 163 | +| `collector.s3.endpoint` | (empty) | Custom S3 endpoint (used when `minio.enabled=false`) | |
| 164 | +| `collector.clickhouse.ttl` | `0` (disabled) | ClickHouse trace TTL (e.g. `72h`) | |
| 165 | +| `collector.aws.accessKeyId` | `minioadmin` | AWS / MinIO access key | |
| 166 | +| `collector.aws.secretAccessKey` | `minioadmin` | AWS / MinIO secret key | |
| 167 | +| `minio.enabled` | `true` | Deploy MinIO (disable for real S3) | |
| 168 | +| `seed.enabled` | `true` | Run the seed job | |
| 169 | + |
| 170 | +### Point at real S3 |
| 171 | + |
| 172 | +Disable MinIO, provide your S3 coordinates, and supply real AWS credentials: |
| 173 | + |
| 174 | +```bash |
| 175 | +helm install trad-signals ./helm \ |
| 176 | + --set minio.enabled=false \ |
| 177 | + --set seed.enabled=false \ |
| 178 | + --set collector.s3.bucket=your-real-bucket \ |
| 179 | + --set collector.s3.prefix=AWSLogs/YOUR_ACCOUNT/vpcflowlogs/us-east-1 \ |
| 180 | + --set collector.s3.region=us-east-1 \ |
| 181 | + --set collector.aws.accessKeyId=AKIA... \ |
| 182 | + --set collector.aws.secretAccessKey=... |
| 183 | +``` |
| 184 | + |
| 185 | +For production, use [IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) (EKS) or [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) (GKE) instead of passing credentials through values. |
| 186 | + |
| 187 | +## How the Pipeline Works |
| 188 | + |
| 189 | +``` |
| 190 | +┌─────────────────────┐ |
| 191 | +│ S3 / MinIO │ VPC Flow Log files (.log.gz) |
| 192 | +│ (awss3receiver) │ Each line → one OTLP log record |
| 193 | +└────────┬────────────┘ |
| 194 | + │ logs |
| 195 | + ▼ |
| 196 | +┌─────────────────────┐ |
| 197 | +│ flowlog connector │ Parses space-delimited text |
| 198 | +│ (logs → traces) │ Maps fields to span attributes |
| 199 | +│ │ Generates TraceID from 5-tuple hash |
| 200 | +└────────┬────────────┘ |
| 201 | + │ traces |
| 202 | + ▼ |
| 203 | +┌─────────────────────┐ |
| 204 | +│ ClickHouse │ Stores OTLP trace spans |
| 205 | +│ (exporter) │ + debug exporter (stdout) |
| 206 | +└─────────────────────┘ |
| 207 | +``` |
| 208 | + |
| 209 | +**awss3receiver** reads gzipped text files from S3. The `text_encoding` extension splits each file by newline, producing one OTLP log record per line. |
| 210 | + |
| 211 | +**flowlog connector** receives those log records. The first line of each file is the header (`version account-id interface-id ...`), which the connector caches as field names. Every subsequent line is split by whitespace, zipped with the field names, and transformed into a trace span with: |
| 212 | + |
| 213 | +- **TraceID**: SHA-256 hash of the 5-tuple (srcaddr, dstaddr, srcport, dstport, protocol), truncated to 16 bytes |
| 214 | +- **Span name**: `flow_{ipVersion}_{protocol}` (e.g., `flow_ipv4_tcp`) |
| 215 | +- **Timestamps**: `start` and `end` fields converted from epoch seconds to nanoseconds |
| 216 | +- **Attributes**: mapped to OTel semantic convention keys where they exist (`source.address`, `destination.port`, `network.transport`) and namespaced keys for AWS-specific fields (`aws.vpc.id`, `cloud.region`) |
| 217 | + |
| 218 | +See `flowlogconnector/fieldmap.go` for the complete mapping of all 52 AWS VPC Flow Log fields. |
| 219 | + |
| 220 | +**clickhouseexporter** writes the spans to a `flow_traces` table in the `telemetry` database. TTL is disabled by default; set `collector.clickhouse.ttl` (e.g. `72h`) to auto-expire old data. |
| 221 | + |
| 222 | +### Add or change field mappings |
| 223 | + |
| 224 | +Edit `flowlogconnector/fieldmap.go`. The `fieldMapping` map controls which AWS field names become which span attribute keys, and whether the value is stored as a string or integer. After editing, rebuild the collector image. |
| 225 | + |
| 226 | +## What This Example Doesn't Do |
| 227 | + |
| 228 | +This is a minimal, educational pipeline. A production network observability system needs significantly more. The [ElastiFlow NetObserv collector](https://www.elastiflow.com) provides all of the following out of the box: |
| 229 | + |
| 230 | +- **Client/server resolution** -- determines which side initiated the connection, expands to `flow.client.*` / `flow.server.*` attributes |
| 231 | +- **Community ID** -- standardized bidirectional flow correlation (both directions of a conversation share a trace ID) |
| 232 | +- **IANA enrichment** -- protocol 6 becomes `TCP`, port 80 becomes `http (TCP/80)` |
| 233 | +- **ASN / GeoIP / DNS** -- resolves IPs to hostnames, autonomous systems, and geographic coordinates |
| 234 | +- **TCP flag decoding** -- bitmask 19 becomes `["FIN", "SYN", "ACK"]` with session establishment detection |
| 235 | +- **Multi-protocol** -- NetFlow v5/v9, IPFIX, sFlow, Azure flow logs, not just AWS |
| 236 | +- **100+ curated OTLP attributes** aligned to OpenTelemetry semantic conventions |
| 237 | + |
| 238 | +## Further Reading |
| 239 | + |
| 240 | +- [Flow Semantic Convention](https://docs.mermin.dev/concepts/semantic-conventions) |
| 241 | +- [ElastiFlow Learning Repo](https://github.com/elastiflow/learning) |
| 242 | +- [awss3receiver documentation](https://pkg.go.dev/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awss3receiver) |
| 243 | +- [OTel Collector Builder (ocb)](https://opentelemetry.io/docs/collector/custom-collector/) |
0 commit comments