Skip to content

Conversation

@lexfrei
Copy link
Contributor

@lexfrei lexfrei commented Oct 23, 2025

Summary

Addresses #5901
Related: #4056

Proposes comprehensive solutions for Gateway API annotation placement confusion, including both short-term documentation improvements and long-term annotation inheritance strategy.

Background

Users frequently misconfigure annotations when using Gateway API sources:

  1. Issue Documentation gap: Provider-specific annotations placement for Gateway API sources #5901: Platform engineer placed cloudflare-proxied annotation on Gateway, expecting it to apply to all Routes, but it was silently ignored (annotations must be on Route resources)
  2. Issue Question: Gatway-api target on HTTPRoute #4056: User tried to override target annotation on HTTPRoute to specify different targets per Route, but target only works on Gateway

This proposal addresses both use cases.

Proposed Solutions

Solution 1: Documentation Improvements (Quick Win)

Status: Already proposed in PR #5918 for quick merge.

Adds clear documentation to annotations.md and gateway-api.md explaining:

  • Which annotations go on Gateway (only target)
  • Which annotations go on Routes (everything else: hostname, ttl, provider-specific)
  • Examples for Cloudflare and AWS providers
  • Common mistakes to avoid

Note: If Solution 2 is implemented, the documentation from PR #5918 will need updates to reflect the new inheritance behavior.

Solution 2: Annotation Inheritance and Merging (Long-term)

Implement annotation merging where:

  • Gateway annotations serve as defaults for all Routes
  • Route annotations override Gateway defaults
  • Enables centralized configuration without losing flexibility

Example:

kind: Gateway
metadata:
  annotations:
    # Default for all Routes using this Gateway
    external-dns.alpha.kubernetes.io/cloudflare-proxied: "true"
    external-dns.alpha.kubernetes.io/ttl: "300"
---
kind: HTTPRoute
metadata:
  annotations:
    # Override: disable proxying for this specific Route
    external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
    # Inherits: ttl=300 from Gateway

Alternatives Considered

  1. Do nothing - Users continue experiencing confusion ❌
  2. Move all annotations to Gateway only - Breaks Gateway API architecture ❌
  3. Support both with strict validation - Complex without solving core UX issue ⚠️
  4. Create dedicated GatewayDNSConfig CRD - Significant effort, scope too large ⚠️
  5. Wait for PR docs(proposal): annotations support graduation to beta #5080 - Timeline uncertain, doc improvements valuable now ⚠️

Recommendation

Phased approach:

  1. Immediate: Merge PR docs(gateway-api): clarify annotation placement for sources #5918 (documentation improvements)
  2. Future: Re-evaluate annotation merging after gathering user feedback and assessing compatibility with annotation standardization efforts

Request for Feedback

This proposal seeks maintainer input on:

  1. Is the current behavior (annotations split between Gateway/Route) intentional design?
  2. Would annotation inheritance/merging be acceptable, or does it conflict with future plans?
  3. Should we prioritize documentation alone, or is feature enhancement warranted?

Related PRs:

Addresses kubernetes-sigs#5901
Related: kubernetes-sigs#4056

Propose comprehensive solutions for Gateway API annotation placement confusion,
including both short-term documentation improvements and long-term annotation
inheritance strategy.

Documentation improvements are proposed separately in PR kubernetes-sigs#5918 for quick merge.

Co-Authored-By: Claude <[email protected]>
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 23, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @lexfrei. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added docs size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 23, 2025
@szuecs
Copy link
Contributor

szuecs commented Oct 24, 2025

@lexfrei do we need this proposal if we have #5918 ?

@lexfrei
Copy link
Contributor Author

lexfrei commented Oct 24, 2025

@szuecs Good question!

Short answer: #5918 (documentation) is a quick fix for the immediate problem. This proposal discusses whether we want to go further.

The difference:

PR #5918 (documentation hotfix):

  • Documents current behavior: target on Gateway, everything else on Routes
  • Helps users avoid confusion right now
  • Doesn't change any code behavior

This proposal (long-term discussion):

  • Solution 1 is just docs(gateway-api): clarify annotation placement for sources #5918 (documentation) - already covered ✅
  • Solution 2 (annotation merging/inheritance) is the interesting part:
    • Allows setting defaults on Gateway that apply to all Routes
    • Routes can override Gateway defaults when needed
    • Makes external-dns more flexible for users managing many Routes

Example use case for Solution 2:

# Today: must repeat annotations on EVERY Route
kind: HTTPRoute
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/cloudflare-proxied: "true"
    external-dns.alpha.kubernetes.io/ttl: "300"
---
kind: HTTPRoute  
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/cloudflare-proxied: "true"  # repeated
    external-dns.alpha.kubernetes.io/ttl: "300"  # repeated

# With Solution 2: set once on Gateway, apply to all Routes
kind: Gateway
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/cloudflare-proxied: "true"  # default for all
    external-dns.alpha.kubernetes.io/ttl: "300"  # default for all
---
kind: HTTPRoute
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"  # override for this one

What I'm asking:

  1. Is docs(gateway-api): clarify annotation placement for sources #5918 (documentation) enough? → If yes, close this proposal
  2. Would annotation inheritance (Solution 2) be useful? → If yes, I can implement it
  3. Are there concerns with Solution 2? → Let's discuss alternatives

I'm fine with just merging #5918 and closing this proposal if you think documentation alone solves the problem. This proposal is mainly to gauge interest in the more flexible approach.

@mloiseleur
Copy link
Collaborator

Is the current behavior (annotations split between Gateway/Route) intentional design?

Yes, we try to stick and be aligned as much as possible with Gateway API design.
See #5319 (comment)
One can check PRs history; some PRs have been refused because of this.

Would annotation inheritance/merging be acceptable, or does it conflict with future plans?

At first glance, it looks good and makes sense to me. You'll need to include ListenerSet in your design.
But maybe I missed something, anything to add, @abursavich ?

Should we prioritize documentation alone, or is feature enhancement warranted?

  • We should always prioritize clear and concise documentation
  • It's Open Source, it comes with no warranty

#### Story 2: User Attempting Route-Specific Targets (#4056)

*As a user*, I want to specify different target DNS records for specific hosts while sharing a common Gateway. I
added `external-dns.alpha.kubernetes.io/target` annotation on HTTPRoute to override the Gateway's target for one
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 Would this PR solve this use case ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR is a complex way to do so.
We'll have yet another annotation to configure the behavior. This is unclear, and not obvious.
The Solution 2 (merge) is the best way, IMO

@mloiseleur
Copy link
Collaborator

/ok-to-test
/lgtm

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 7, 2025
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 7, 2025
- Change status from draft to provisional
- Add reference to PR kubernetes-sigs#5998
- Update pseudocode: target annotation is now inheritable and overridable
- Update example to show intranet/public target override use case
- Add benefit: solves User Story 2 (per-Route target overrides)
- Update recommendation to include near-term merging of Solution 2

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 7, 2025
@k8s-ci-robot
Copy link
Contributor

New changes are detected. LGTM label has been removed.

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from mloiseleur. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@lexfrei
Copy link
Contributor Author

lexfrei commented Dec 7, 2025

The proposal has been reviewed and updated to reflect the actual implementation needs.

Key changes:

  • Status changed from draft to provisional (is this appropriate given the proposal hasn't been formally accepted yet? It seems to better reflect the current state with a reference implementation available)
  • Solution 2 (Annotation Inheritance) now has a reference implementation: feat(gateway): implement annotation inheritance from Gateway to Route #5998
  • Updated the pseudocode to allow target annotation inheritance and per-Route overrides
  • Added a practical example demonstrating intranet/public IP override use case

Recommendation update:
I suggest we no longer consider Solution 1 (Documentation-Only) as a standalone approach. User feedback from #5901 and #4056 demonstrates a clear need for annotation inheritance functionality, not just better documentation. The reference implementation in #5998 provides backward-compatible annotation merging with comprehensive test coverage.

@coveralls
Copy link

Pull Request Test Coverage Report for Build 20006369865

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 180 unchanged lines in 11 files lost coverage.
  • Overall coverage increased (+0.01%) to 78.679%

Files with Coverage Reduction New Missed Lines %
openshift_route.go 1 79.49%
apis/externaldns/types.go 1 99.71%
plan.go 3 95.85%
source.go 3 83.33%
zz_generated.deepcopy.go 5 0.0%
endpoint.go 6 90.6%
istio_gateway.go 8 90.1%
service.go 20 93.33%
coredns/coredns.go 30 90.36%
gloo_proxy.go 47 74.18%
Totals Coverage Status
Change from base Build 18674696388: 0.01%
Covered Lines: 16008
Relevant Lines: 20346

💛 - Coveralls

@lexfrei lexfrei requested a review from mloiseleur December 7, 2025 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. docs ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants