Thyme is a Tulip-based OpenTelemetry Collector distribution optimized for high-throughput log processing. Built for performance benchmarking and validation of log collection at scale.
Thyme is a specialized distribution based on Tulip, OllyGarden's commercially supported OpenTelemetry Collector distribution. While Tulip provides a general-purpose, production-ready collector, Thyme focuses specifically on:
- High-throughput log ingestion - Optimized for processing 50-100k+ log records per second
- Performance benchmarking - Tuned configurations for measuring collector performance
- Edge collection - DaemonSet deployments reading from Kubernetes log files
Thyme was created to validate OpenTelemetry Collector performance for enterprise log collection scenarios where:
- Logs are collected from
/var/log/podson Kubernetes nodes - Kubernetes metadata enrichment is required
- Throughput requirements exceed 50k log records/second per node
- Resource consumption must remain bounded on shared nodes
Thyme includes components optimized for log collection:
Receivers:
filelog- File-based log receiver for Kubernetes pod logsotlp- OTLP Receiver for forwarding
Processors:
k8sattributes- Kubernetes metadata enrichmentbatch- Batching for efficient exportmemory_limiter- Resource consumption bounds
Exporters:
otlp- OTLP Exporter using gRPC protocolotlphttp- OTLP Exporter using HTTP protocoldebug- Debug exporter for testing
Extensions:
healthcheck- Health Check Extensionpprof- Profiling for performance analysis
| Metric | Target |
|---|---|
| Log throughput | 100k records/sec (default config) |
| Average log size | ~500 bytes (100-1000 byte range) |
| Deployment mode | DaemonSet |
| Node type | m6i.2xlarge (8 vCPU, 32GB RAM) |
The following dashboard shows collector metrics during a 100k logs/sec benchmark run:
Key metrics shown:
- otelcol_exporter_sent_log_records_total - Logs successfully exported
- otelcol_receiver_accepted_log_records_total - Logs received by collectors
- otelcol_exporter_queue_batch_send_size - Batch sizes being sent
- otelcol_process_runtime_heap_alloc_bytes - Memory usage
- otelcol_process_cpu_seconds_total - CPU consumption
- Go 1.22 or later
- OpenTelemetry Collector Builder (ocb) - automatically downloaded by the Makefile
- (Optional) Docker & Docker Compose for containerized testing
Build the Thyme distribution:
make buildThis generates the binary at distributions/thyme/build/thyme.
Run locally with the test configuration:
make runClean build artifacts:
make cleanBuild the Docker image:
make docker-buildBuild and push to GitHub Container Registry:
make docker-pushBuild and load into k3d cluster:
make k3d-load
# Or specify a cluster name
K3D_CLUSTER=my-cluster make k3d-loadThyme includes two configuration files:
config.yaml: Production configuration with filelog receiver and k8sattributes processor for Kubernetes DaemonSet deploymentsconfig-local.yaml: Local testing configuration with OTLP receiver only
Thyme can be deployed in three ways:
See deployment/compose/ for local testing with Docker Compose.
cd deployment/compose
docker-compose upSee deployment/kubernetes/ for Kubernetes manifests.
kubectl apply -k deployment/kubernetes/See infrastructure/aws/ for OpenTofu/Terraform infrastructure-as-code and deployment/aws/ for AWS-specific Kubernetes overlays.
# Provision infrastructure
cd infrastructure/aws
tofu init && tofu apply
# Configure kubectl
aws eks update-kubeconfig --region eu-central-1 --name thyme-benchmark
# Deploy stack
kubectl apply -k deployment/aws/Cost: $2.50/hour ($60/day) for 3× m6i.2xlarge nodes in eu-central-1. See infrastructure/aws/README.md for details.
The setup creates a realistic two-stage collection pipeline for performance testing:
loggen (10,000 lines/sec × 10 pods = 100k/sec)
↓ (writes to stdout → containerd)
/var/log/pods/*
↓ (filelog receiver)
thyme collector (DaemonSet)
↓ (k8sattributes → batch → OTLP exporter)
nop-collector
↓ (nop exporter - discards data)
∅
Both collectors send internal telemetry → LGTM
Components:
- loggen: Generates 10,000 log lines/sec per pod (100-1000 byte lines) to stdout, captured by containerd
- thyme: Reads logs via filelog receiver, processes (batch, memory_limiter, resource), exports via OTLP to nop-collector
- nop-collector: Receives OTLP, processes, exports to nop (discard)
- LGTM: Collects internal telemetry (metrics, traces) from both collectors on port 3000
Exposed Ports:
3000- Grafana (LGTM)8080- loggen metrics55679- nop-collector zpages55680- thyme zpages1777- nop-collector pprof1778- thyme pprof
Access Grafana via port-forward (kubectl port-forward -n lgtm service/grafana 3000:3000) at http://localhost:3000 to observe collector internal telemetry (throughput, latency, resource usage). For k3d local development, NodePort is also available at http://localhost:30000.
Run a complete automated benchmark with report generation:
# Run 60-minute benchmark (default)
./scripts/run-benchmark.sh
# Run custom duration (in minutes)
./scripts/run-benchmark.sh 10
# Or use Claude Code skill
# In Claude Code: "Run a benchmark"This will:
- Create k3d cluster with 2 agents
- Build and deploy Thyme + LGTM stack
- Monitor performance for specified duration
- Collect metrics and logs
- Generate report in
./local/reports/YYYY-MM-DD-NN/
See scripts/README.md for details.
Run production-scale benchmarks on AWS EKS with automated infrastructure provisioning:
# Run benchmark with default 30-minute active phase + ramp-up/cool-down
./scripts/run-benchmark-aws.sh
# Custom active duration (60 minutes)
./scripts/run-benchmark-aws.sh 60
# Custom cluster name
./scripts/run-benchmark-aws.sh 30 my-test-cluster
# Disable auto-cleanup (keep cluster after benchmark)
AUTO_CLEANUP=false ./scripts/run-benchmark-aws.shBenchmark phases (default 45 minutes total):
- Ramp-up: 5 minutes (system stabilization)
- Active: 30 minutes (performance measurement)
- Cool-down: 10 minutes (observe tail behavior)
This will:
- Provision EKS cluster with 3× m6i.2xlarge nodes (~15 minutes)
- Deploy Thyme + LGTM stack with LoadBalancer for Grafana
- Run phased benchmark with health monitoring
- Collect metrics via Prometheus API
- Generate report in
./local/reports/YYYY-MM-DD-NN-aws/ - Automatically destroy infrastructure (unless
AUTO_CLEANUP=false)
Cost: ~$2.50/hour. A 45-minute benchmark costs ~$1.90.
See infrastructure/aws/README.md and deployment/aws/README.md for details.
For production-like performance testing with k3d, see the Kubernetes deployment guide.
Quick test (10 minutes):
# 1. Deploy to k3d
k3d cluster create thyme && make k3d-load K3D_CLUSTER=thyme
kubectl apply -k deployment/kubernetes/
# 2. Access Grafana
kubectl port-forward -n lgtm service/grafana 3000:3000 &
open http://localhost:3000 # admin/admin
# (or use http://localhost:30000 for k3d NodePort)
# 3. Monitor throughput (Explore → Prometheus)
rate(otelcol_receiver_accepted_log_records_total{service_name="nop-collector"}[1m])
# Expected: ~100,000 logs/secthyme/
├── docs/
│ └── images/ # Documentation images
│ └── grafana-benchmark-metrics.png
├── deployment/
│ ├── compose/ # Docker Compose for local development
│ │ ├── docker-compose.yaml
│ │ └── nop-collector-config.yaml
│ ├── kubernetes/ # Kubernetes base manifests
│ │ ├── namespace.yaml
│ │ ├── serviceaccount.yaml
│ │ ├── *-configmap.yaml
│ │ ├── *-deployment.yaml
│ │ ├── thyme-daemonset.yaml
│ │ └── kustomization.yaml
│ └── aws/ # AWS EKS Kustomize overlay
│ ├── kustomization.yaml
│ ├── grafana-loadbalancer.yaml
│ ├── loggen-pod-affinity.yaml
│ └── README.md
├── distributions/
│ └── thyme/
│ ├── manifest.yaml # Component manifest
│ ├── config.yaml # Production configuration
│ ├── config-local.yaml # Local testing configuration
│ ├── Makefile # Distribution build automation
│ ├── bin/ # Downloaded ocb binary
│ └── build/ # Generated sources and binary
├── infrastructure/
│ └── aws/ # OpenTofu/Terraform for EKS
│ ├── main.tf
│ ├── variables.tf
│ ├── vpc.tf
│ ├── eks.tf
│ ├── eks-node-group.tf
│ ├── iam.tf
│ ├── security-groups.tf
│ ├── outputs.tf
│ └── README.md
├── scripts/
│ ├── run-benchmark.sh # Automated k3d benchmark
│ └── run-benchmark-aws.sh # Automated AWS EKS benchmark
├── Dockerfile # Container image definition
├── Makefile # Root build automation
└── README.md # This file
- Tulip - Parent distribution
- OpenTelemetry Collector Documentation
Q: What's up with this name? A: ⏱️ Thyme sounds like "time" - fitting for a distribution focused on time-sensitive, high-throughput log processing where every millisecond counts. In the garden, thyme is a hardy, fast-growing herb that thrives under pressure, just like this collector under heavy log load.
Apache License, Version 2.0. See LICENSE for the full license text.
Thyme is a specialized distribution for benchmarking and validation. For production deployments, see Tulip.