Skip to content

Add StreamConsumer resource support - #93

Open
bjorn-stange-expel wants to merge 3 commits into
aws-controllers-k8s:mainfrom
bjorn-stange-expel:add-stream-consumer
Open

bjorn-stange-expel wants to merge 3 commits into
aws-controllers-k8s:mainfrom
bjorn-stange-expel:add-stream-consumer

Conversation

@bjorn-stange-expel

@bjorn-stange-expel bjorn-stange-expel commented Jul 9, 2026

Copy link
Copy Markdown

Resolves aws-controllers-k8s/community#2844

Why?

The kinesis-controller currently only manages Stream resources. Kinesis enhanced fan-out requires registering stream consumers (RegisterStreamConsumer), which today has to be done outside Kubernetes. This PR adds a StreamConsumer custom resource so consumers can be managed declaratively alongside their streams.

What changed?

All controller code is generated by ACK code-generator (v0.61.0) — the source change is in generator.yaml, which maps a new StreamConsumer resource to the consumer APIs:

  • RegisterStreamConsumer (create), DescribeStreamConsumer (read one), DeregisterStreamConsumer (delete). There is no update API — consumers are immutable, so sdkUpdate returns a terminal NotImplemented.
  • consumerName is the primary key. streamARN supports ACK resource references, so a consumer can point at its stream with spec.streamRef.from.name instead of a hard-coded ARN.
  • The ResourceSynced condition is set once ConsumerStatus is ACTIVE.
  • Tag ensure is ignored for this resource (tags.ignore: true). The Kinesis API recently added consumer tagging (Tags on RegisterStreamConsumer, plus TagResource/ListTagsForResource), but those operations don't exist yet in the aws-sdk-go-v2/service/kinesis version this controller is generated against, so tag support isn't generatable today. A natural follow-up once the SDK pin is bumped.

Everything else (apis/, pkg/resource/stream_consumer/, CRDs, RBAC, helm) is generated output. Release version was kept at v1.3.2 so no release artifacts churn.

Also adds an e2e test (test_stream_consumer.py) that creates a Stream, registers a consumer against it via streamRef, verifies the consumer goes ACTIVE with its ARN in status, and tears both down.

Verification

  • go build ./... passes; gofmt clean
  • go test ./... passes (no unit test files in this repo; e2e lives in test/e2e)
  • Generated with the pinned toolchain: code-generator v0.61.0, runtime v0.61.0, matching upstream main
  • e2e test passed against real AWS (2026-07-08): built the controller image locally, installed it into a kind cluster via the helm chart, and ran pytest e2e/tests/test_stream_consumer.py against us-east-1. Full CRUD verified in 2m02s — Stream created, consumer registered via streamRef reference resolution, went ACTIVE with ResourceSynced=True and status.consumerARN matching the AWS-side ARN, then deregistered and cleaned up (no leaked resources).
e2e/tests/test_stream_consumer.py::TestStreamConsumer::test_crud PASSED
================== 1 passed, 12 warnings in 122.81s (0:02:02) ==================

🤖 Generated with Claude Code

bjorn-stange-expel and others added 2 commits July 8, 2026 10:58
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds a new StreamConsumer custom resource mapping to the Kinesis
enhanced fan-out consumer APIs (RegisterStreamConsumer,
DescribeStreamConsumer, DeregisterStreamConsumer), generated with
code-generator v0.61.0.

- consumerName is the primary key; streamARN supports resource
  references to Stream via streamRef
- ResourceSynced condition is set once ConsumerStatus is ACTIVE
- Stream consumers do not support tags, so tag ensure is ignored
- Adds e2e test registering a consumer against a stream via streamRef

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ack-prow ack-prow Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 9, 2026
@ack-prow
ack-prow Bot requested a review from a-hilaly July 9, 2026 18:29
@ack-prow

ack-prow Bot commented Jul 9, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bjorn-stange-expel
Once this PR has been reviewed and has the lgtm label, please assign jlbutler for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ack-prow
ack-prow Bot requested a review from gustavodiaz7722 July 9, 2026 18:29
@ack-prow ack-prow Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 9, 2026
@ack-prow

ack-prow Bot commented Jul 9, 2026

Copy link
Copy Markdown

Hi @bjorn-stange-expel. Thanks for your PR.

I'm waiting for a aws-controllers-k8s member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

The Kinesis API does support tagging consumers as of a recent API
update, but the SDK version this controller is generated against
predates it, so tag support remains ignored rather than unsupported.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bjorn-stange-expel
bjorn-stange-expel marked this pull request as ready for review July 9, 2026 18:43
@ack-prow ack-prow Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 9, 2026
@ack-prow
ack-prow Bot requested a review from jlbutler July 9, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kinesis Support Enhanced Fan-out

1 participant