Add OTEL-native Container Insights log pipelines (app, host) by louisall · Pull Request #310 · aws-observability/helm-charts

louisall · 2026-05-11T12:14:11Z

Add OTEL log pipelines for Container Insights

Adds application and host log collection via OTEL, running inside the existing CloudWatch Agent DaemonSet. Opt-in via otelContainerInsights.logs.enabled: true (requires otelContainerInsights.enabled: true).

What it does

When enabled, the agent collects:

Application logs — all container stdout/stderr from /var/log/containers/*.log (excluding agent and system pods)
Host logs — /var/log/messages, /var/log/dmesg, /var/log/secure

Logs are sent to CloudWatch Logs via OTLP with per-node streams:

/aws/otel/containerinsights/<cluster>/application
/aws/otel/containerinsights/<cluster>/host

Metrics–logs attribute parity

Every log record carries the same resource attributes as the metrics for the same pod/node. This means you can correlate logs and metrics by k8s.pod.name, k8s.namespace.name, k8s.workload.name, k8s.node.name, cloud.resource_id, pod labels, node labels, etc. — all populated by the same shared k8sattributes processor instances (one K8s API watch serves both signals).

How it works

filelog receiver reads container/host log files
container_log_parser operator strips CRI/Docker envelope (depends on otel-contrib#533)
awscloudwatchlogsprovisioner extension pre-creates log groups/streams lazily (depends on agent#2113)
Shared k8sattributes processors enrich logs with the same pod/node metadata as metrics
Transform processors add cluster name, cloud resource ID, workload name — identical to the metrics pipeline

Config

otelContainerInsights:
  enabled: true
  logs:
    enabled: true   # opt-in, default true (no-op when enabled=false)

If enabled is false, everything under otelContainerInsights is ignored (including logs.enabled).

Testing

flag_matrix.sh validates all 8 flag combinations (template-level, wired into make all)
Integration test scenarios: logs-disabled, dual-publish, hybrid-metrics-fluentbit, config-merge-isolation

Changes since initial review

otelContainerInsights.logs restructured from flat boolean to otelContainerInsights.logs.enabled (future-proofing for additional log knobs)
Removed phantom dataplane receivers/pipelines from tests (deferred to future PR)
Removed runlogjournal/varlogjournal volume checks (FluentBit volumes, not CW Agent)
Cleaned up stale cross-flag rule enforcement code and comments — all 8 flag combinations are valid
Fixed default_test.go comment accuracy
Wired flag_matrix.sh into CI via make helm-flag-matrix (runs in Build job)
filestorage for checkpoint persistence deferred to follow-up PR (requires writable hostPath — security posture change)
queue_size: 500 retained per benchmark data (Appendix C.8 — eliminates drops at X-Large with zero CPU/memory cost)

…ension Register container_log_parser stanza operator (plugins.go) and cwlogsprovision extension (extension/cwlogsprovision/) in the agent's default components for OTEL Container Insights log pipelines. Also includes SafeChannel.Close() nil-receiver safety fix. Depends on: - amazon-contributing/opentelemetry-collector-contrib#533 (container_log_parser) - aws-observability/helm-charts#310 (pipeline config)

…ovisioner extension Register container_log_parser stanza operator (plugins.go) and awscloudwatchlogsprovisioner extension in the agent's default components for OTEL Container Insights log pipelines. The provisioner extension (from otel-contrib PR #511) lazily creates CloudWatch log groups/streams on first request via the HTTP auth chain. Replaces the earlier cwlogsprovision approach. Depends on: - amazon-contributing/opentelemetry-collector-contrib#533 (container_log_parser) - amazon-contributing/opentelemetry-collector-contrib#511 (awscloudwatchlogsprovisioner) - aws-observability/helm-charts#310 (pipeline config)

sky333999 · 2026-05-20T00:02:43Z

+#       as expected (minikube terraform harness cannot test "expected failure")
+#
+# Run from the repo root:
+#     bash charts/amazon-cloudwatch-observability/tests/flag_matrix.sh


Is this meant to be checked in? What going to actually run this in the Github CI? Dont see any changes to our workflows.

If there is value on top of the minikube tests, can we wire it into helm-lint or a flag-matrix-test make target?

Good catch, wired it into CI via a helm-flag-matrix Make target. It now runs as part of make all. No minikube needed — it's pure helm template so it runs in ~10 seconds

this feels redundant with minikube tests getting added. nice to have an extra layer but it is weird we test a feature (flags) at build time. My point is we should be consistent that this is going to be a new pattern we follow by enforcing various rules at build time OR we just stay with the existing pattern of simple minikube tests.

Lemme take a look if minikube tests cover everything for now and remove this.

Agreed — kept it simple by going with consistent minikube-only coverage.

Resolution in 1d84963:

Removed flag_matrix.sh and the helm-flag-matrix Make target

Added new minikube scenario otlp-logs-otel-only for state Fixing helm template function call for instrumentation images in oper… #7 (enabled=true, logs=true, containerLogs=false) — this was the only state Fixing helm template function call for instrumentation images in oper… #7 was uniquely covered by the matrix script

Minikube coverage of the 8 flag combinations:

State enabled / logs / containerLogs Minikube scenario

#1 f / f / f covered by other "off" scenarios

#2 f / f / t default (default values)

#3 f / t / f degenerate to #1 (logs ignored when enabled=f)

#4 f / t / t degenerate to #2

#5 t / f / f otlp-logs-disabled

#6 t / f / t otlp-hybrid-metrics-fluentbit

#7 t / t / f otlp-logs-otel-only ✨ new

#8 t / t / t otlp-logs-dual-publish

States #3/#4 are template-pure (silent ignore of logs.enabled) and degenerate to existing minikube cases, so no new scenario is needed.

Can we update PR desc to reflect the same?

louisall · 2026-05-20T12:58:15Z

Failures unrelated. This PR will fix them: #312

movence · 2026-05-22T13:16:58Z


+{{- if .Values.otelContainerInsights.logs.enabled }}
+  otlphttp/cw_k8s_ci_v0_app_logs_dest:
+    endpoint: {{ if .Values.otelContainerInsights.cloudwatchLogsEndpoint }}{{ .Values.otelContainerInsights.cloudwatchLogsEndpoint | quote }}{{ else }}"https://logs.{{ .Values.region }}.amazonaws.com:443"{{ end }}


This should handle ADC regions similar how FB does currently

{{- if hasKey .Values.adcEndpointOverrides .Values.region }} endpoint logs.${AWS_REGION}.{{ index .Values.adcEndpointOverrides .Values.region }} {{- end }}

movence · 2026-05-22T13:27:04Z

+#       as expected (minikube terraform harness cannot test "expected failure")
+#
+# Run from the repo root:
+#     bash charts/amazon-cloudwatch-observability/tests/flag_matrix.sh


this feels redundant with minikube tests getting added. nice to have an extra layer but it is weird we test a feature (flags) at build time. My point is we should be consistent that this is going to be a new pattern we follow by enforcing various rules at build time OR we just stay with the existing pattern of simple minikube tests.

movence · 2026-05-22T13:52:43Z

+
+	// AmazonCloudWatchAgent CR should carry OTEL metrics config but no log pipelines.
+	dynamicClient, err := k8sClient.GetDynamicClient()
+	if !assert.NoError(t, err) {


nit: seeing this pattern in other places too. can be simplied with

Suggested change

if !assert.NoError(t, err) {

require.NoError(t, err, "failed to get dynamic client")

movence · 2026-05-22T14:00:55Z

+    log_statements:
+      - context: resource
+        statements:
+          - set(attributes["cloud.resource_id"], Concat(["arn:aws:eks:", attributes["cloud.region"], ":", attributes["cloud.account.id"], ":cluster/", attributes["k8s.cluster.name"]], ""))


non-eks support matters?

Don't think we have added support for these yet, we can revisit once we are here. Will add as followup.

movence · 2026-05-26T15:51:50Z

+
 exporters:
  otlphttp/cw_k8s_ci_v0_metrics_dest:
    endpoint: {{ if .Values.otelContainerInsights.cloudwatchMetricsEndpoint }}{{ .Values.otelContainerInsights.cloudwatchMetricsEndpoint | quote }}{{ else }}"https://monitoring.{{ .Values.region }}.amazonaws.com:443"{{ end }}


we should support ADC as well here like logs

…host) Adds opt-in OTEL log collection to the existing CloudWatch Agent DaemonSet. No new K8s resources created — logs config is injected into the agent's otelConfig when otelContainerInsights.logs=true. Pipelines: - logs/cw_k8s_ci_v0_app: container logs via filelog + container_log_parser - logs/cw_k8s_ci_v0_node: host OS logs (/var/log/messages, dmesg, secure) Key design decisions: - Shares k8sattributes processor instances with metrics (one API watch) - cwlogsprovision extension pre-creates log groups/streams at startup - Log groups: /aws/otel/containerinsights/<cluster>/{application,host} - Simple config: enabled=false ignores everything under otelContainerInsights - No cross-flag validation with FluentBit (deferred to v7.0.0) - Dataplane logs deferred (no journald/aws-node collection) Depends on: - amazon-contributing/opentelemetry-collector-contrib#533 - aws/amazon-cloudwatch-agent#2113

- Add ADC region endpoint support to OTEL log exporters (movence) Render logs.${region}.${adcEndpointOverrides} for isolated regions, matching FluentBit's existing behavior. - Replace assert.NoError + t.Fatal pattern with require.NoError (movence) Simplifies 7 occurrences across the OTLP minikube test scenarios. - Add otlp-logs-otel-only minikube scenario for state #7 (otelContainerInsights.enabled=true, logs.enabled=true, containerLogs=false). This was the only flag_matrix.sh state without minikube coverage. - Remove flag_matrix.sh and helm-flag-matrix Make target (movence) Coverage is now consolidated in minikube scenarios for consistency.

- Drop redundant v6.x comment and version references from values/scenarios - Reword logs comment to reflect it's enabled by default - Fix nit: logs.enabled=false in supported combinations - Gate varlog mount by targetAgent (cluster-scraper doesn't need /var/log) - Exhaustive fragment list in otlp_logs_disabled_test - Reuse shared resourcedetection/cw_k8s_ci_v0 instead of duplicate _logs instance - Extract flag validation into chart.validate-flags helper - Add agent-config-merge-isolation to CI scenario list

mitali-salvi · 2026-05-27T14:16:32Z

+
+	// Namespace + operator sanity.
+	ns, err := k8sClient.GetNamespace(minikube.Namespace)
+	assert.NoError(t, err)


Might wanna reconsider updating this to require.NoError(t, err) coz line 38 will panic if ns == nil

Same applies for other files including validations/minikube/scenarios/otlp_logs_disabled_test.go, otlp_logs_dual_publish_test.go, otlp_logs_otel_only_test.go, otlp_hybrid_metrics_fluentbit_test.go

mitali-salvi · 2026-05-27T14:23:24Z

+      # FluentBit's existing Exclude_Path behavior.
+      - /var/log/containers/aws-node*
+      - /var/log/containers/kube-proxy*
+    start_at: end


nit - I know its referenced in the PR description but might be worth adding a comment here as well
No checkpoint persistence — logs during agent downtime are lost. Filestorage support is tracked in follow-up PR.

mitali-salvi · 2026-05-27T14:24:15Z

+          # Logs need service.name; metrics use k8s.workload.name directly.
+          - set(attributes["service.name"], attributes["k8s.workload.name"]) where attributes["service.name"] == nil and attributes["k8s.workload.name"] != nil
+
+  transform/cw_k8s_ci_v0_logs_set_cluster_name:


nit - this seems like its setting the node name attribute as well, might wanna rename this

- Rename transform/logs_set_cluster_name to logs_set_cluster_and_node - Add filestorage caveat comment at start_at: end - Change assert.NoError to require.NoError for namespace checks (prevents nil panic) - Restore flag_matrix.sh and Makefile target (lost during stash conflict)

* Add OTEL Container Insights infrastructure (#282) * Add v6.0.0 release notes * Cluster-scraper deployment now uses cloudwatch-agent SA (cluster-scraper SA retained as dead resource) * Bump chart version to 6.0.1 and add release notes * Scope config-map permission for cloudwatch-agent to namespace only (#287) * scope configmap permission to namesapce * integ tests * updates * Retain configmap get permission at cluster scope (#291) * Fix template whitespace control and hardcoded test names - Use {{- if instead of {{ if in cloudwatch-agent-role.yaml and cloudwatch-agent-rolebinding.yaml to prevent leading blank lines - Extract agentName constant in configmap_permission_scoping_test.go to replace hardcoded strings * Keep configmaps get at cluster scope, scope create/update to namespace - Add configmaps get-only rule to ClusterRole - Narrow namespace-scoped Role to create and update only - Update test assertions to match * Run integration tests on PRs targeting release-* branches * Set hostNetwork: true for default $.agent * Feature/ksm enrichment and label preservation (#296) * Fix agent.config bleed-through into cluster-scraper deployment (#298) * Fix scrapeTimeout, add missing scrape_timeout, remove dead RBAC, cleanup docs and tests (#297) * Add ClusterIP Service for node-exporter and use service DNS for scraping (#304) * Refactor fluent-bit region-specific (isolated) config using tpl conditionals (#271) * Pin GitHub Actions to commit SHAs (#306) * minor fixes (#307) * release v6.1.0 * Add LIS CSI metrics support to OTEL container insights config (#308) Add prometheus receiver, scope transform, attribute promote transform, and pipeline for local instance store CSI driver metrics collection, mirroring the existing EBS CSI pattern. Scrapes pods with label app=ec2-instance-store-plugin in kube-system namespace and promotes instance_id and volume_id to resource attributes. * feat(otel): Add k8sattributes processor node filter to daemonset (#311) * fix(integration-test): include node-exporter-service in EKS resource counts (#312) * Add OTEL-native Container Insights log pipelines (app, host) (#310) * release v6.2.0 * Address PR review: use v0.18 CRD schema, add version bumps to release notes --------- Co-authored-by: Louise Allen <147041218+louisall@users.noreply.github.com> Co-authored-by: Kaushik Surya <kausyas@amazon.com> Co-authored-by: Kaushik Surya <108111936+sky333999@users.noreply.github.com> Co-authored-by: Petru Anica-Popa <66786772+petruanica@users.noreply.github.com> Co-authored-by: Trey Richbourg <63218430+treyrichbourg@users.noreply.github.com> Co-authored-by: POOJA REDDY NATHALA <poojardy@amazon.com> Co-authored-by: Michael O'Neill <miconeil@amazon.com> Co-authored-by: Mitali Salvi <mitsalvi@amazon.com>

* Add OTEL Container Insights infrastructure (aws-observability#282) * Add v6.0.0 release notes * Cluster-scraper deployment now uses cloudwatch-agent SA (cluster-scraper SA retained as dead resource) * Bump chart version to 6.0.1 and add release notes * Scope config-map permission for cloudwatch-agent to namespace only (aws-observability#287) * scope configmap permission to namesapce * integ tests * updates * Retain configmap get permission at cluster scope (aws-observability#291) * Fix template whitespace control and hardcoded test names - Use {{- if instead of {{ if in cloudwatch-agent-role.yaml and cloudwatch-agent-rolebinding.yaml to prevent leading blank lines - Extract agentName constant in configmap_permission_scoping_test.go to replace hardcoded strings * Keep configmaps get at cluster scope, scope create/update to namespace - Add configmaps get-only rule to ClusterRole - Narrow namespace-scoped Role to create and update only - Update test assertions to match * Run integration tests on PRs targeting release-* branches * Set hostNetwork: true for default $.agent * Feature/ksm enrichment and label preservation (aws-observability#296) * Fix agent.config bleed-through into cluster-scraper deployment (aws-observability#298) * Fix scrapeTimeout, add missing scrape_timeout, remove dead RBAC, cleanup docs and tests (aws-observability#297) * Add ClusterIP Service for node-exporter and use service DNS for scraping (aws-observability#304) * Refactor fluent-bit region-specific (isolated) config using tpl conditionals (aws-observability#271) * Pin GitHub Actions to commit SHAs (aws-observability#306) * minor fixes (aws-observability#307) * release v6.1.0 * Add LIS CSI metrics support to OTEL container insights config (aws-observability#308) Add prometheus receiver, scope transform, attribute promote transform, and pipeline for local instance store CSI driver metrics collection, mirroring the existing EBS CSI pattern. Scrapes pods with label app=ec2-instance-store-plugin in kube-system namespace and promotes instance_id and volume_id to resource attributes. * feat(otel): Add k8sattributes processor node filter to daemonset (aws-observability#311) * fix(integration-test): include node-exporter-service in EKS resource counts (aws-observability#312) * Add OTEL-native Container Insights log pipelines (app, host) (aws-observability#310) * release v6.2.0 * Address PR review: use v0.18 CRD schema, add version bumps to release notes --------- Co-authored-by: Louise Allen <147041218+louisall@users.noreply.github.com> Co-authored-by: Kaushik Surya <kausyas@amazon.com> Co-authored-by: Kaushik Surya <108111936+sky333999@users.noreply.github.com> Co-authored-by: Petru Anica-Popa <66786772+petruanica@users.noreply.github.com> Co-authored-by: Trey Richbourg <63218430+treyrichbourg@users.noreply.github.com> Co-authored-by: POOJA REDDY NATHALA <poojardy@amazon.com> Co-authored-by: Michael O'Neill <miconeil@amazon.com> Co-authored-by: Mitali Salvi <mitsalvi@amazon.com>

louisall mentioned this pull request May 11, 2026

Add OTEL CI logs agent components aws/amazon-cloudwatch-agent#2113

Merged

louisall requested a review from sky333999 May 11, 2026 13:40

miconeilaws force-pushed the feature/otelify-ci-logs branch from f7edde7 to ae355c3 Compare May 14, 2026 10:09

miconeilaws changed the title ~~Add OTEL-native Container Insights log pipelines (app, host, dataplane)~~ Add OTEL-native Container Insights log pipelines (app, host,) May 14, 2026

miconeilaws force-pushed the feature/otelify-ci-logs branch from ae355c3 to 4e20a09 Compare May 15, 2026 08:30

miconeilaws force-pushed the feature/otelify-ci-logs branch from 4e20a09 to a12e3dd Compare May 15, 2026 13:26

sky333999 reviewed May 20, 2026

View reviewed changes

louisall force-pushed the feature/otelify-ci-logs branch from a12e3dd to 6955339 Compare May 20, 2026 09:11

louisall changed the base branch from release-6.1.0 to release-6.2.0 May 20, 2026 09:11

louisall force-pushed the feature/otelify-ci-logs branch 3 times, most recently from 5d46bd9 to 59e8b5d Compare May 20, 2026 12:27

louisall force-pushed the feature/otelify-ci-logs branch from 59e8b5d to ec80478 Compare May 20, 2026 13:04

movence reviewed May 22, 2026

View reviewed changes

spanaik requested review from movence and sky333999 May 25, 2026 15:07

louisall mentioned this pull request May 26, 2026

feat: Add file_storage extension for filelog checkpoint persistence #313

Open

movence reviewed May 26, 2026

View reviewed changes

movence approved these changes May 26, 2026

View reviewed changes

sky333999 changed the title ~~Add OTEL-native Container Insights log pipelines (app, host,)~~ Add OTEL-native Container Insights log pipelines (app, host) May 27, 2026

sky333999 reviewed May 27, 2026

View reviewed changes

miconeilaws and others added 3 commits May 27, 2026 12:10

louisall force-pushed the feature/otelify-ci-logs branch from 4f62b89 to 79c48f7 Compare May 27, 2026 12:12

mitali-salvi reviewed May 27, 2026

View reviewed changes

sky333999 approved these changes May 28, 2026

View reviewed changes

sky333999 merged commit e2a56c2 into release-6.2.0 May 28, 2026
35 of 37 checks passed

sky333999 deleted the feature/otelify-ci-logs branch May 28, 2026 03:48

movence mentioned this pull request May 29, 2026

fix(otel): explicitly enable otelContainerInsights in all EKS daemon otel tests aws/amazon-cloudwatch-agent-test#707

Merged

2 tasks

scardena mentioned this pull request Jun 25, 2026

otelContainerInsights.enabled default changed true -> false in v6.2.0 without deprecation notice #323

Open

State	enabled / logs / containerLogs	Minikube scenario
#1	f / f / f	covered by other "off" scenarios
#2	f / f / t	`default` (default values)
#3	f / t / f	degenerate to #1 (logs ignored when enabled=f)
#4	f / t / t	degenerate to #2
#5	t / f / f	`otlp-logs-disabled`
#6	t / f / t	`otlp-hybrid-metrics-fluentbit`
#7	t / t / f	`otlp-logs-otel-only` ✨ new
#8	t / t / t	`otlp-logs-dual-publish`

	if !assert.NoError(t, err) {
	require.NoError(t, err, "failed to get dynamic client")

Uh oh!

Conversation

louisall commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add OTEL log pipelines for Container Insights

What it does

Metrics–logs attribute parity

How it works

Config

Testing

Changes since initial review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

louisall commented May 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

louisall commented May 11, 2026 •

edited

Loading