Skip to content

Commit 698c799

Browse files
miconeilawsmiconeilaws
committed
feat(otel): Add custom OTLP metrics endpoint with mTLS auth
Expose a node-local OTLP endpoint (container-insights-otlp:4319/4320) from the Container Insights agent that accepts customer application metrics and enriches them with Kubernetes and cloud attributes automatically. - OTLP gRPC/HTTP receiver with configurable auth (mTLS default, none optional) - k8sattributes processor resolves pod identity from connection IP - Full enrichment pipeline: cluster, node, workload, cloud/EC2 metadata - Service with internalTrafficPolicy: Local for node-local routing - Server cert SANs include the new service name - Client cert reuses existing agent-client-cert (same CA) - cert-manager path supported for bring-your-own-CA - Go demo app proving zero-code-change mTLS via OTel spec env vars - Feature docs and architecture brief
1 parent ade4944 commit 698c799

19 files changed

Lines changed: 1431 additions & 0 deletions

‎charts/amazon-cloudwatch-observability/docs/custom-otlp-endpoint-feature-brief.md‎

Lines changed: 271 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,271 @@
1+
# Container Insights: Custom OTLP Metrics Endpoint
2+
3+
## What is it
4+
5+
A built-in, node-local OTLP endpoint that ships with Container Insights. Any workload in the cluster can send custom application metrics to CloudWatch — with automatic Kubernetes and cloud enrichment — by setting a single environment variable.
6+
7+
**It's on by default.** When a customer installs the CloudWatch EKS Add-On or the `amazon-cloudwatch-observability` Helm chart with OTel Container Insights enabled, the endpoint is already live and accepting metrics. No additional configuration, no extra installations, no collector sidecars.
8+
9+
```
10+
Endpoint: container-insights-otlp.amazon-cloudwatch:4319 (gRPC)
11+
container-insights-otlp.amazon-cloudwatch:4320 (HTTP)
12+
```
13+
14+
---
15+
16+
## Benefits
17+
18+
- **Zero infrastructure to manage** — no collector deployments, no sidecars, no per-team configuration. The collector is already running as part of Container Insights.
19+
- **Automatic enrichment** — every metric gets ~40 Kubernetes and cloud attributes (pod, namespace, workload, node, cluster, region, account) without application-side effort. This standardizes the dimensional model across all services in the cluster.
20+
- **Push-based, not pull-based** — applications push metrics when they're ready, not when a scraper decides to poll. No `/metrics` endpoint to expose, no scrape intervals to tune, no missed scrapes during pod restarts.
21+
- **Works with any OTel SDK** — standard OTel protocol, no AWS-specific libraries. Teams already using OTel get CloudWatch integration for free.
22+
- **Node-local, low latency** — traffic never leaves the node. The Service routes exclusively to the agent on the same node as the sending pod.
23+
- **Secure by default** — mTLS out of the box. Access is controlled by which namespaces have the client cert.
24+
25+
## Where does the collector run
26+
27+
The OTLP endpoint is served by the **existing CloudWatch Agent DaemonSet** — the same agent that already collects Container Insights infrastructure metrics (cAdvisor, kubelet, node-exporter, etc.). There is no additional collector deployment.
28+
29+
```
30+
Every node in the cluster:
31+
┌─────────────────────────────────────────────────────┐
32+
│ CloudWatch Agent (DaemonSet pod) │
33+
│ │
34+
│ • Container Insights pipelines (existing) │
35+
│ • Custom OTLP receiver on port 4319/4320 (new) │
36+
│ │
37+
│ All pipelines share the same agent process, │
38+
│ same SigV4 credentials, same CloudWatch exporter. │
39+
└─────────────────────────────────────────────────────┘
40+
```
41+
42+
The custom OTLP receiver is an additional pipeline within the same agent — no new pods, no new resource consumption beyond the metrics being processed.
43+
44+
## How is this better than Prometheus scraping
45+
46+
| | Push (OTLP endpoint) | Pull (Prometheus scrape) |
47+
|--|---|---|
48+
| **Setup for app teams** | 1 env var | Expose `/metrics` endpoint, annotate pods or create ServiceMonitor/PodMonitor |
49+
| **Pod restarts** | No data loss — SDK buffers and retries | Missed scrape windows = data gaps |
50+
| **High-cardinality metrics** | App controls what it sends | Scraper pulls everything exposed — cardinality explosions are silent until the bill arrives |
51+
| **Firewall/network policy** | Outbound only from app pod | Inbound required — agent must reach app pod's metrics port |
52+
| **Ephemeral/batch jobs** | Job pushes metrics before exit | Job may complete between scrape intervals — metrics lost |
53+
| **SDK semantics** | Histograms, exemplars, exponential histograms (OTLP-native) | Limited to Prometheus exposition format (no exemplars in scrape, no exponential histograms) |
54+
| **Discovery** | None needed — app decides to send | Requires service discovery config (labels, annotations, CRDs) |
55+
| **Enrichment** | Automatic (connection IP → full pod/workload/cloud context) | Requires relabel_configs or additional processors |
56+
57+
Prometheus scraping remains valuable for infrastructure components that already expose `/metrics` (node-exporter, kube-state-metrics, etc.) — and Container Insights continues to use it for those. But for **application-owned custom metrics**, the push-based OTLP path is simpler, more reliable, and requires no infrastructure awareness from application teams.
58+
59+
---
60+
61+
## How it works
62+
63+
Customers configure their OTel SDK using standard environment variables defined in the [OpenTelemetry Protocol Exporter specification](https://opentelemetry.io/docs/specs/otel/protocol/exporter/). That's it — the SDK handles everything from there.
64+
65+
### Without auth (dev/test clusters)
66+
67+
```yaml
68+
env:
69+
- name: OTEL_EXPORTER_OTLP_ENDPOINT
70+
value: "http://container-insights-otlp.amazon-cloudwatch:4319"
71+
- name: OTEL_EXPORTER_OTLP_INSECURE
72+
value: "true"
73+
```
74+
75+
### With mTLS auth (default, production)
76+
77+
```yaml
78+
env:
79+
- name: OTEL_EXPORTER_OTLP_ENDPOINT
80+
value: "https://container-insights-otlp.amazon-cloudwatch:4319"
81+
- name: OTEL_EXPORTER_OTLP_CERTIFICATE
82+
value: "/var/run/secrets/otlp/ca.crt"
83+
- name: OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE
84+
value: "/var/run/secrets/otlp/tls.crt"
85+
- name: OTEL_EXPORTER_OTLP_CLIENT_KEY
86+
value: "/var/run/secrets/otlp/tls.key"
87+
```
88+
89+
All four env vars are defined in the OTel spec. Every spec-compliant SDK reads them automatically — **zero code changes** in the application.
90+
91+
### Application code (any language)
92+
93+
```go
94+
// Go — one line, no endpoint or TLS config in code
95+
exporter, _ := otlpmetricgrpc.New(ctx)
96+
```
97+
98+
```java
99+
// Java — one line, no endpoint or TLS config in code
100+
OtlpGrpcMetricExporter exporter = OtlpGrpcMetricExporter.builder().build();
101+
```
102+
103+
The SDK reads the endpoint and TLS credentials from the environment variables. The application never imports any AWS-specific libraries.
104+
105+
---
106+
107+
## Authentication
108+
109+
### mTLS (default)
110+
111+
mTLS is enabled by default. The agent's OTLP receiver requires clients to present a certificate signed by the Container Insights CA. This is configured using the OTel spec environment variables — no code changes required.
112+
113+
**SDK support for mTLS env vars:**
114+
115+
| Language | `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE` / `CLIENT_KEY` support | Code changes needed |
116+
|----------|--------------------------------------------------------------|-------------------|
117+
| Go | Fully supported | None |
118+
| Java | Fully supported | None |
119+
| .NET | Fully supported | None |
120+
| Python | Not yet implemented in SDK | ~10 lines to read cert files and build gRPC credentials |
121+
| Node.js | Not yet implemented in SDK | ~8 lines for metadata generator |
122+
123+
The Python/Node.js limitation is an **OTel SDK implementation gap** — the spec defines these env vars, but those SDKs haven't implemented them yet. This is not a CloudWatch limitation. For those languages, customers can either:
124+
- Add a small credential setup block (~10 lines)
125+
- Or disable auth (`auth.type: none`) if their cluster doesn't require it
126+
127+
### Disabling auth
128+
129+
For dev/test clusters or languages without mTLS env var support, auth can be disabled via helm:
130+
131+
```bash
132+
helm upgrade cw-otel amazon-cloudwatch-observability \
133+
--set otelContainerInsights.customTelemetry.auth.type=none
134+
```
135+
136+
Or in values.yaml:
137+
138+
```yaml
139+
otelContainerInsights:
140+
customTelemetry:
141+
auth:
142+
type: none
143+
```
144+
145+
This switches the endpoint to plaintext — any pod can send without a certificate. The customer pod spec simplifies to just one env var:
146+
147+
```yaml
148+
env:
149+
- name: OTEL_EXPORTER_OTLP_ENDPOINT
150+
value: "http://container-insights-otlp.amazon-cloudwatch:4319"
151+
- name: OTEL_EXPORTER_OTLP_INSECURE
152+
value: "true"
153+
```
154+
155+
Not recommended for production.
156+
157+
### Configuration
158+
159+
```yaml
160+
otelContainerInsights:
161+
customTelemetry:
162+
enabled: true # endpoint is on (default)
163+
grpcPort: 4319
164+
httpPort: 4320
165+
auth:
166+
type: mtls # mtls (default) | none
167+
```
168+
169+
---
170+
171+
## Automatic Enrichment
172+
173+
This is the core value. The customer publishes a simple metric with a few business attributes. Container Insights enriches it with ~40 Kubernetes, cloud, and infrastructure attributes before it reaches CloudWatch.
174+
175+
### Customer publishes
176+
177+
```
178+
orders.processed_total{order.region="us-east", order.tier="premium"} = 1
179+
```
180+
181+
With resource attributes: `service.name=order-service`, `service.version=1.2.0`
182+
183+
### What arrives in CloudWatch
184+
185+
| Category | Attribute | Example value | Source |
186+
|----------|-----------|---------------|--------|
187+
| **Customer-provided** | `order.region` | `us-east` | Application code |
188+
| | `order.tier` | `premium` | Application code |
189+
| **Service identity** | `@resource.service.name` | `order-service` | OTel SDK resource |
190+
| | `@resource.service.version` | `1.2.0` | OTel SDK resource |
191+
| | `@resource.telemetry.sdk.language` | `go` | OTel SDK auto |
192+
| | `@resource.telemetry.sdk.name` | `opentelemetry` | OTel SDK auto |
193+
| | `@resource.telemetry.sdk.version` | `1.25.0` | OTel SDK auto |
194+
| **Pod identity** | `@resource.k8s.pod.name` | `order-service-7c54698bc8-kdpjp` | k8sattributes (connection IP) |
195+
| | `@resource.k8s.pod.uid` | `a1b2c3d4-e5f6-...` | k8sattributes |
196+
| | `@resource.k8s.pod.ip` | `10.0.145.86` | k8sattributes |
197+
| | `@resource.k8s.namespace.name` | `orders-team` | k8sattributes |
198+
| | `@resource.k8s.pod.label.app` | `order-service` | k8sattributes |
199+
| **Workload identity** | `@resource.k8s.deployment.name` | `order-service` | k8sattributes |
200+
| | `@resource.k8s.replicaset.name` | `order-service-7c54698bc8` | k8sattributes |
201+
| | `@resource.k8s.workload.name` | `order-service` | Workload derivation |
202+
| | `@resource.k8s.workload.type` | `Deployment` | Workload derivation |
203+
| **Node** | `@resource.k8s.node.name` | `ip-10-0-161-222.ec2.internal` | Node env var |
204+
| | `@resource.k8s.node.uid` | `f1e2d3c4-...` | k8sattributes |
205+
| | `@resource.k8s.node.label.eks.amazonaws.com/capacityType` | `ON_DEMAND` | k8sattributes |
206+
| | `@resource.k8s.node.label.eks.amazonaws.com/nodegroup` | `standard-workers` | k8sattributes |
207+
| | `@resource.k8s.node.label.kubernetes.io/arch` | `amd64` | k8sattributes |
208+
| | `@resource.k8s.node.label.kubernetes.io/os` | `linux` | k8sattributes |
209+
| | `@resource.k8s.node.label.topology.k8s.aws/zone-id` | `use1-az1` | k8sattributes |
210+
| **Cluster** | `@resource.k8s.cluster.name` | `production-cluster` | Helm config |
211+
| | `@resource.cloud.resource_id` | `arn:aws:eks:us-east-1:123456789012:cluster/production-cluster` | Derived |
212+
| **Cloud / Host** | `@resource.cloud.provider` | `aws` | EC2 metadata |
213+
| | `@resource.cloud.platform` | `aws_eks` | EKS detection |
214+
| | `@resource.cloud.region` | `us-east-1` | EC2 metadata |
215+
| | `@resource.cloud.availability_zone` | `us-east-1a` | EC2 metadata |
216+
| | `@resource.cloud.account.id` | `123456789012` | EC2 metadata |
217+
| | `@resource.host.id` | `i-0abc123def456789` | EC2 metadata |
218+
| | `@resource.host.name` | `ip-10-0-161-222.ec2.internal` | EC2 metadata |
219+
| | `@resource.host.type` | `m5.xlarge` | EC2 metadata |
220+
| | `@resource.host.image.id` | `ami-0abcdef1234567890` | EC2 metadata |
221+
| **Pipeline attribution** | `@instrumentation.cloudwatch.source` | `cloudwatch-agent` | Scope transform |
222+
| | `@instrumentation.cloudwatch.solution` | `k8s-otel-container-insights` | Scope transform |
223+
| | `@instrumentation.cloudwatch.pipeline` | `custom-otlp` | Scope transform |
224+
| | `@instrumentation.@name` | `order-service` | OTel meter name |
225+
| | `@instrumentation.@version` | `1.2.0` | OTel meter version |
226+
| **Derived** | `@aws.account` | `123456789012` | CloudWatch derived |
227+
| | `@aws.region` | `us-east-1` | CloudWatch derived |
228+
229+
**The customer wrote 4 attributes. CloudWatch received 40+.** Every application in the cluster gets this same standardized dimensional model automatically — no per-team configuration, no enforcement overhead, no drift between services.
230+
231+
---
232+
233+
## Bring Your Own Certificate
234+
235+
Customers who want to use their own CA (corporate PKI, Vault, etc.) instead of the auto-generated self-signed cert can do so via cert-manager integration:
236+
237+
```yaml
238+
# Helm values
239+
agent:
240+
autoGenerateCert:
241+
enabled: false
242+
certManager:
243+
enabled: true
244+
issuerRef:
245+
kind: ClusterIssuer
246+
name: my-corporate-ca # customer's own CA issuer
247+
```
248+
249+
cert-manager issues all certificates (including the OTLP client cert) from the customer's CA. Applications that already trust that CA don't need any additional CA cert distribution.
250+
251+
For customers without cert-manager, the default self-signed CA works out of the box with zero external dependencies.
252+
253+
---
254+
255+
## Summary
256+
257+
| Aspect | Detail |
258+
|--------|--------|
259+
| **Setup required** | None — endpoint is live by default |
260+
| **Application code changes** | None (Go, Java, .NET) |
261+
| **Configuration** | Standard OTel env vars ([spec](https://opentelemetry.io/docs/specs/otel/protocol/exporter/)) |
262+
| **Auth default** | mTLS (client cert required) |
263+
| **Auth override** | `auth.type: none` for plaintext |
264+
| **Cert management** | Auto-generated self-signed, or bring-your-own via cert-manager |
265+
| **Enrichment** | ~40 attributes (pod, workload, node, cluster, cloud) added automatically |
266+
| **Pod identity** | Resolved from connection IP — no client-side config needed |
267+
| **Routing** | Node-local only (`internalTrafficPolicy: Local`) — no cross-node hops |
268+
| **CloudWatch namespace** | Same OTLP endpoint as Container Insights metrics |
269+
| **AWS dependencies** | None in application code — standard OTel SDK only |
270+
271+
**Zero setup. Point your telemetry at the endpoint. CloudWatch does the rest.**

0 commit comments

Comments
 (0)