feat(new transform): add AWS ECS metadata transform by hrko · Pull Request #25579 · vectordotdev/vector

hrko · 2026-06-05T13:27:29Z

Summary

Adds a new aws_ecs_metadata transform that enriches log and metric events with Amazon ECS task metadata from the ECS Task Metadata v4 endpoint.

The transform supports the ECS metadata endpoint exposed through ECS_CONTAINER_METADATA_URI_V4, with an explicit endpoint override for tests and non-standard deployments. It fetches task metadata at startup, refreshes it periodically, and can optionally require metadata availability before Vector starts processing events.

This PR also adds component reference documentation, generated docs, and a changelog fragment.

Generated substantially with Codex (gpt-5.5 default).

Vector configuration

transforms:
  add_ecs_metadata:
    type: aws_ecs_metadata
    inputs:
      - app_logs
    required: true

For local testing with a mock endpoint:

transforms:
  add_ecs_metadata:
    type: aws_ecs_metadata
    inputs:
      - app_logs
    endpoint: http://127.0.0.1:8080
    refresh_interval_secs: 60
    refresh_timeout_secs: 5

How did you test this PR?

I validated the implementation with unit tests, documentation checks, and local equivalents of the relevant CI checks.

Ran successfully:

make check-fmt
make check-clippy
make test
make check-events
./scripts/check_changelog_fragments.sh
make check-markdown
make check-licenses
make check-deny
make check-deny-licenses
mise exec node@22 cue@0.16.1 -- make check-docs
cd website && mise exec node@22 cue@0.16.1 -- make cue-build
mise exec node@22 cue@0.16.1 -- make check-generated-docs
cd rust-doc && make ci-docs-build
make test-vrl
cd lib/vector-vrl/web-playground && wasm-pack build --target web --out-dir public/pkg

Additional real AWS validation:

Provisioned temporary ECS test infrastructure with OpenTofu and validated the transform against the ECS Task Metadata v4 endpoint in all targeted ECS execution environments.

Environment	Observed launch type	Logs enriched	Metrics enriched	`container_name` override
Fargate	`FARGATE`	pass	pass	pass
EC2 capacity provider	`EC2`	pass	pass	pass
ECS Managed Instances	`MANAGED_INSTANCES`	pass	pass	pass

The ECS validation task ran a Vector container from this PR branch plus an app sidecar container. The Vector config exercised both default current-container enrichment and container_name: app enrichment, and verified enriched log events and internal metrics in CloudWatch Logs.

I can provide the detailed validation report, CloudWatch Logs evidence samples, and the OpenTofu reproduction code if reviewers would like to inspect the setup in more detail.

Change Type

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

Changelog fragment added:

changelog.d/2894_aws_ecs_metadata_transform.feature.md

References

Closes: Support for AWS ECS Task Metadata #2894

Notes

The implementation intentionally mirrors the existing aws_ec2_metadata transform shape where practical, while keeping ECS-specific behavior isolated:

metadata is fetched from the ECS Task Metadata v4 endpoint;
required controls whether startup must fail when metadata cannot be fetched;
refresh failures retain the last successful metadata snapshot;
request timeout covers send, status handling, and response body collection;
metadata fields are scoped to commonly available ECS task/container fields, with room for future expansion.

When container_name is configured, select the container from /task directly instead of requiring the current-container endpoint to succeed. Also add support for the documented container Snapshotter scalar field and cover both changes with regression tests.

github-actions · 2026-06-05T13:27:45Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

hrko · 2026-06-05T13:29:19Z

I have read the CLA Document and I hereby sign the CLA

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fec5314883

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

rtrieu

left some editorial suggestions

rtrieu · 2026-06-09T16:37:23Z

+		]
+		warnings: [
+			"""
+				Do not enable this transform if you are running Vector as an Aggregator, metadata


Suggested change

Do not enable this transform if you are running Vector as an Aggregator, metadata

Do not enable this transform if you are running Vector as an Aggregator. Metadata

rtrieu · 2026-06-09T16:37:46Z

+		warnings: [
+			"""
+				Do not enable this transform if you are running Vector as an Aggregator, metadata
+				will be sourced from the Aggregator task's metadata endpoint and not the client's.


Suggested change

will be sourced from the Aggregator task's metadata endpoint and not the client's.

will be sourced from the Aggregator task's metadata endpoint and not the originating task's endpoint.

rtrieu · 2026-06-09T16:38:16Z

+		}
+	}
+	required: {
+		description: "Requires the transform to be able to successfully query the ECS metadata before starting to process the data."


Suggested change

description: "Requires the transform to be able to successfully query the ECS metadata before starting to process the data."

description: "If true, Vector will not start until ECS metadata is successfully retrieved."

rtrieu · 2026-06-09T16:38:55Z

+		}
+	}
+	initial_retry_attempts: {
+		description: "The number of initial metadata refresh attempts before the transform starts."


Suggested change

description: "The number of initial metadata refresh attempts before the transform starts."

description: "The number of times to attempt fetching metadata at startup before Vector begins processing events."

rtrieu · 2026-06-09T16:39:19Z

+		}
+	}
+	refresh_interval_secs: {
+		description: "The interval between querying for updated metadata, in seconds."


Suggested change

description: "The interval between querying for updated metadata, in seconds."

description: "Interval between metadata refresh requests, in seconds."

rtrieu · 2026-06-09T16:46:29Z

+		]
+		warnings: [
+			"""
+				Do not enable this transform if you are running Vector as an Aggregator, metadata


Suggested change

Do not enable this transform if you are running Vector as an Aggregator, metadata

Do not enable this transform if you are running Vector as an Aggregator. Metadata

rtrieu · 2026-06-09T16:46:56Z

+		}
+	}
+	required: {
+		description: "Requires the transform to be able to successfully query the ECS metadata before starting to process the data."


Suggested change

description: "Requires the transform to be able to successfully query the ECS metadata before starting to process the data."

description: "Requires the transform to successfully query the ECS metadata endpoint before processing events."

hrko · 2026-06-09T17:16:44Z

Thanks for the suggestions. I applied them in 7df82ed.

hrko added 6 commits June 5, 2026 20:04

feat(aws ecs metadata transform): add transform

2e29867

test(aws_ecs_metadata transform): cover documented launch types

80052d9

docs(aws ecs metadata transform): document design intent

4abf6df

docs(aws ecs metadata transform): clarify enrichment scope

94154b4

docs(aws ecs metadata transform): type metadata endpoint env var

98eebb2

github-actions Bot added domain: transforms Anything related to Vector's transform components domain: external docs Anything related to Vector's external, public documentation docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. labels Jun 5, 2026

hrko marked this pull request as ready for review June 5, 2026 13:37

hrko requested review from a team as code owners June 5, 2026 13:37

hrko added 2 commits June 8, 2026 13:41

docs(aws ecs metadata transform): document output metadata

9ad08c6

Merge branch 'master' into feat/aws-ecs-metadata-transform

fec5314

chatgpt-codex-connector Bot reviewed Jun 8, 2026

View reviewed changes

Comment thread src/transforms/aws_ecs_metadata.rs Outdated

Comment thread src/transforms/aws_ecs_metadata.rs

Comment thread src/transforms/aws_ecs_metadata.rs Outdated

hrko added 2 commits June 8, 2026 14:53

fix(aws ecs metadata transform): refine container lookup

34d1295

Merge branch 'master' into feat/aws-ecs-metadata-transform

8f7759b

rtrieu approved these changes Jun 9, 2026

View reviewed changes

docs(aws ecs metadata transform): address review suggestions

7df82ed

	Do not enable this transform if you are running Vector as an Aggregator, metadata
	Do not enable this transform if you are running Vector as an Aggregator. Metadata

	will be sourced from the Aggregator task's metadata endpoint and not the client's.
	will be sourced from the Aggregator task's metadata endpoint and not the originating task's endpoint.

	description: "Requires the transform to be able to successfully query the ECS metadata before starting to process the data."
	description: "If true, Vector will not start until ECS metadata is successfully retrieved."

	description: "The number of initial metadata refresh attempts before the transform starts."
	description: "The number of times to attempt fetching metadata at startup before Vector begins processing events."

	description: "The interval between querying for updated metadata, in seconds."
	description: "Interval between metadata refresh requests, in seconds."

Conversation

hrko commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

github-actions Bot commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hrko commented Jun 5, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rtrieu left a comment

Choose a reason for hiding this comment

Uh oh!

rtrieu Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

rtrieu Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

rtrieu Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

rtrieu Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

rtrieu Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

rtrieu Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

rtrieu Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

hrko commented Jun 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hrko commented Jun 5, 2026 •

edited

Loading

github-actions Bot commented Jun 5, 2026 •

edited

Loading