-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for TCP_UDP to NLB TargetGroups and Listeners (rebase) #3807
base: main
Are you sure you want to change the base?
Conversation
Welcome @lyda! |
Hi @lyda. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Thanks for taking this up, I get to subscribe to another pull request now. Hope you have much better luck than the last one. Just in time for HTTP3 going more mainstream! |
b025fee
to
23e2f43
Compare
OK, I think I've addressed the issues with the rebase. But the verify step is now failing. Any ideas why? |
@M00nF1sh / @oliviassss : Hey there, I think this is done and reviewable. I've run it in a test EKS system and it does what I needed it to do. However I'm not an expert in this code base or with load balancers so would love feedback from yourselves. Thank you! |
1fb8e33
to
2ab64b4
Compare
I've done manual testing and it worked fine. There really aren't docs to update. The information that the current aws-load-balancer-controller doesn't support listening to UDP and TCP on the same port number is information you'll find in the issues. Is there a slack/discord/irc channel I can chat about this to address any remaining issues? |
I believe developers (I'm not one of them) are somewhat reachable via https://slack.k8s.io/, channel |
@@ -87,6 +87,9 @@ const ( | |||
|
|||
// NetworkingProtocolUDP is the UDP protocol. | |||
NetworkingProtocolUDP NetworkingProtocol = "UDP" | |||
|
|||
// NetworkingProtocolTCP_UDP is the TCP_UDP protocol. | |||
NetworkingProtocolTCP_UDP NetworkingProtocol = "TCP_UDP" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we don't need this.
the networking rules here is meant for security groups(which don't have a tcp_udp rule protocol)
instead of TCP_UDP, we should just generate two rules in TGB's networking, one for TCP and another one for UDP.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm sorry, but we do. This is for AWS load balancer target groups, not security groups, and as much I find it annoying, TCP_UDP
is a protocol for AWS load balancers. It's used in the single case where the TCP
and UDP
port numbers are the same. So, for example, if you're implementing a domain name server and need to listen on port 53 for both TCP
and UDP
, the load balancer must be configured to have a target group protocol of TCP_UDP
. I have no idea why they did this, but they did.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm, this type is used for a lot of things. For some TCP
, UDP
and TCP_UDP
are allowed. For others, only TCP
and UDP
.
Should I split out the types across the code base? It makes the PR bigger, but it would work better with kubebuilder.
} | ||
} | ||
|
||
// execute build listener | ||
for _, port := range t.service.Spec.Ports { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not just loop over portMap
instead of t.service.Spec.Ports
. (we don't need this portMap[key]
check this way)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because it's trying to find the case of the same port number being listened to on TCP
and UDP
. portMap
is empty and is being used to find duplicate port numbers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Aren't you populating portMap
in the "group by listener port number" immediately above?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes.
The complete map needs to be populated before building the listener for the AWS load balancer. This might be easier to understand with an example.
Say the service exports out the following ports: 53/UDP, 53/TCP and 443/TCP.
The loop above would create a port map like this with that data (pseudo code):
53: [53/UDP, 53/TCP],
443: [443/TCP]
}
Now when we go through this loop we can see if any port has been set for TCP and UDP. If so, it creates a single TCP_UDP port which is what the AWS Load Balancer requires.
This is the most efficient way to do this - the portMap
map is an optimisation. Otherwise for each t.service.Spec.Ports
we'd need to check all remaining ports to see if we need to allocate a TCP_UDP port. We'd also need to track if we already allocated a port as a TCP_UDP port.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At this point, I don't see the difference in the current loop and this:
// execute build listener
for _, vals := range portMap {
var port corev1.ServicePort
if len(vals) > 1 {
port, err = mergeServicePortsForListener(vals)
if err != nil {
return err
}
} else {
port = vals[0]
}
_, err := t.buildListener(ctx, port, cfg, scheme)
if err != nil {
return err
}
}
except that this doesn't modify t.service.Spec.Ports
in-place. Is that in-place modification an expected side-effect here? If so, then the current loop will not remove the second of the two merged ports after replacing the first one with the result of mergeServicePortsForListener
. (That in-place modification is also an example of the leakage of the TCP_UDP
protocol name outside this code, but I haven't checked to see if this actually has any effect).
portMap
in this version of the loop could probably be map[int32][]*corev1.ServicePort
if we're not modifying t.service.Spec.Ports
in-place, saving a bunch of string-copying. That var port corev1.ServicePort
could probably also be a pointer, if I remember my Go lifetime rules correctly, and we could even pass a pointer to the ServicePort
down into buildListener
, as I see that was already done with cfg
in this PR. (That said, I'm speculating on GitHub; I haven't asked the compiler to validate this particular idea.)
pkg/service/model_build_listener.go
Outdated
@@ -234,3 +256,24 @@ func (t *defaultModelBuildTask) buildListenerAttributes(ctx context.Context, svc | |||
} | |||
return attributes, nil | |||
} | |||
|
|||
func mergeServicePortsForListener(ports []corev1.ServicePort) corev1.ServicePort { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this works but have two issues:
- we are hacking the k8s
corev1.ServicePort
concept by introduce a newcorev1.Protocol("TCP_UDP")
, which don't exists in k8s, and in theory is an malformed `corev1.ServicePort. (TCP_UDP is not a validate Protocol enum) - this don't generate a warning/error if unsupported protocol/protocol conbination is found.
- this exists as a "glue" code to make the existing
buildListeners
function happy, but in the long run would make the code harder to understand and maintain.
Instead, i think a better option would be:
- modify buildListeners to accept a list of corev1.ServicePort. inside buildListeners, it first validate whether this list of corev1.ServicePort can form a single listener(and error out when cannot), and generate listener of correct protocol.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm confused, that's what buildListeners
now does. It loops through the corev1.ServicePort
s and determines if there's a case where the same port number exists for TCP
and UDP
. It then synthesises a single corev1.ServicePort
from the original two that uses the TCP_UDP
protocol that AWS load balancers support.
Either way it then calls buildListener
.
I could change everything from buildListener
on down to use an internal data structure that includes a slice of corev1.ServicePort
s as well as the protocol being used for the load balancer. That seems like a rather complex change. And it will still have the rats nest of conditionals the current code has. Plus all corev1.ServicePort
stuff would operate off the first element - so might as well just include a single corev1.ServicePort
. And at that point all I've done is avoided abusing an enum.
That said, point number two is definitely valid and I'm now returning errors for a few scenarios up the chain.
Does it make sense to mark this PR as closing #1608? That should cause GitHub to link to this PR from that ticket, rather than relying on the comments. (It might also keep k8s-ci-robot from closing that issue as stale while this PR is in-progress... maybe?) I note that this PR isn't solving the actual use-case for which that ticket was opened, but the bulk of the discussion in that ticket was for this use-case, so tying them together makes sense to me. A solution for the original use-case would revolve around some kind of new |
Awesome, thanks for the review. I'll work through these this week. |
Good point. I'll turn that off because this is completely different to the original issue and just addresses a comment within. Sorry I missed that. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I think I've gone through them all. I see your point on buildListener
but I don't think the change will make it any better. The code will still have to pick through the three "protocols" (I really do dislike AWS's TCP_UDP
"feature" here so I have to put quotes around protocols at least once) and that's going to make the config translation code more complex than just the binary choices before.
The real decision is: do you want to support this feature for the few of us who need it and pay the complexity price or do you not. I completely understand either choice. I would prefer the former obviously because I have zero interest in maintaining a fork, but that's a me problem, not a you problem.
@@ -87,6 +87,9 @@ const ( | |||
|
|||
// NetworkingProtocolUDP is the UDP protocol. | |||
NetworkingProtocolUDP NetworkingProtocol = "UDP" | |||
|
|||
// NetworkingProtocolTCP_UDP is the TCP_UDP protocol. | |||
NetworkingProtocolTCP_UDP NetworkingProtocol = "TCP_UDP" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm sorry, but we do. This is for AWS load balancer target groups, not security groups, and as much I find it annoying, TCP_UDP
is a protocol for AWS load balancers. It's used in the single case where the TCP
and UDP
port numbers are the same. So, for example, if you're implementing a domain name server and need to listen on port 53 for both TCP
and UDP
, the load balancer must be configured to have a target group protocol of TCP_UDP
. I have no idea why they did this, but they did.
} | ||
} | ||
|
||
// execute build listener | ||
for _, port := range t.service.Spec.Ports { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because it's trying to find the case of the same port number being listened to on TCP
and UDP
. portMap
is empty and is being used to find duplicate port numbers.
pkg/service/model_build_listener.go
Outdated
@@ -234,3 +256,24 @@ func (t *defaultModelBuildTask) buildListenerAttributes(ctx context.Context, svc | |||
} | |||
return attributes, nil | |||
} | |||
|
|||
func mergeServicePortsForListener(ports []corev1.ServicePort) corev1.ServicePort { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm confused, that's what buildListeners
now does. It loops through the corev1.ServicePort
s and determines if there's a case where the same port number exists for TCP
and UDP
. It then synthesises a single corev1.ServicePort
from the original two that uses the TCP_UDP
protocol that AWS load balancers support.
Either way it then calls buildListener
.
I could change everything from buildListener
on down to use an internal data structure that includes a slice of corev1.ServicePort
s as well as the protocol being used for the load balancer. That seems like a rather complex change. And it will still have the rats nest of conditionals the current code has. Plus all corev1.ServicePort
stuff would operate off the first element - so might as well just include a single corev1.ServicePort
. And at that point all I've done is avoided abusing an enum.
That said, point number two is definitely valid and I'm now returning errors for a few scenarios up the chain.
Previously, aws-load-balancer-controller ignored extra overlapping ServicePorts defined in the Kubernetes Service spec if the external port numbers were the same even if the protocols were different (e.g. TCP:53, UDP:53). This behavior prevented users from exposing services that support TCP and UDP on the same external load balancer port number. This patch solves the problem by detecting when a user defines multiple ServicePorts for the same external load balancer port number but using TCP and UDP protocols separately. In such situations, a TCP_UDP TargetGroup and Listener are created and SecurityGroup rules are updated accordingly. If more than two ServicePorts are defined, only the first two mergeable ServicePorts are used. Otherwise, the first ServicePort is used. Note: rebasing errors would be my fault -- Kevin Lyda Signed-off-by: Kevin Lyda <[email protected]>
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: lyda The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
} | ||
} | ||
|
||
// execute build listener | ||
for _, port := range t.service.Spec.Ports { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Aren't you populating portMap
in the "group by listener port number" immediately above?
@@ -234,3 +260,20 @@ func (t *defaultModelBuildTask) buildListenerAttributes(ctx context.Context, svc | |||
} | |||
return attributes, nil | |||
} | |||
|
|||
func mergeServicePortsForListener(ports []corev1.ServicePort) (corev1.ServicePort, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I haven't traced this all-the-way through, but perhaps this should return a new type, rather than a corev1.ServicePort
? It seems like an intrusive change, so I'm not sure if there's value there compared to the risk of somehow leaking a TCP_UDP
ServicePort
outside the code that understands it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would need to completely recreate the corev1.ServicePort
type. The only change would be to the Protocol
member.
The port
parameter gets used by (not an exhaustive list) buildListener
, buildListenerSpec
, buildTargetGroup
. In places it uses k8s
modules so I would need a way to convert this custom type to a temporary variable of the corev1.ServicePort
. How would the conversion work - what is TCP_UDP changed into? Right now we don't call anything that references the Protocol
part of the corev1.ServicePort
, but we'd need to populate it with something.
I'm happy to do this but it will involve changing an awful lot of code, adding routines to convert between this custom type and corev1.ServicePort
and I'm dubious it will make things more understandable.
port := ports[0] | ||
port.Protocol = corev1.Protocol("TCP_UDP") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given that we're taking just the first listed port's details, should we also validate that targetPort and/or nodePort match? (Which one matters I guess depends on whether it's an Instance or IP target group, which we can't see in this function)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We know they match because that's what the portMap above did - it grouped by port.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It only grouped by port, but that doesn't ensure the targetPort and/or nodePort are identical. If I recall correctly, there was a test-case which showed this, and is currently a success:true
case, but once that merged port is given to the rest of the code, it'll produce a firewall entry that is sending TCP traffic to the right nodePort, but then the UDP traffic would be sent to the same nodePort even though the service's nodePort for UDP was different.
We can't resolve that case with TCP_UDP, so it probably should fail in this function, as we can't catch that case in code that receives the merged ServicePort
.
It's complicated a little because if we're using IP target group mode, then the nodePorts won't be used, and so it won't matter if they don't match, and similarly for the targetPort values in instance target group mode. So maybe this function needs to know if we're working with an instance target group or IP target group, so it knows which of nodePort
and targetPort
to validate.
@@ -39,7 +65,7 @@ func (t *defaultModelBuildTask) buildListener(ctx context.Context, port corev1.S | |||
return ls, nil | |||
} | |||
|
|||
func (t *defaultModelBuildTask) buildListenerSpec(ctx context.Context, port corev1.ServicePort, cfg listenerConfig, | |||
func (t *defaultModelBuildTask) buildListenerSpec(ctx context.Context, port corev1.ServicePort, cfg *listenerConfig, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can't comment directly on it, but I think the logic for switching to ProtocolTLS
immediately below probably needs to be specific to ProtocolTCP
rather than !ProtocolUDP
as I don't think turning a TCP_UDP
port into a TLS port is a valid conversion. Maybe it should fail if we get a TCP_UDP
port where the TCP side would match the TLS configuration? (This might be the only place we use port.Name
, from a brief check, which would also resolve the concern that the merged TCP_UDP
port gains the name of the first-processed ServicePort
, and loses the second's name; as they cannot be identical)
@@ -484,6 +484,8 @@ func (t *defaultModelBuildTask) buildTargetGroupBindingNetworking(_ context.Cont | |||
Protocol: &protocolTCP, | |||
Port: &tgPort, | |||
}) | |||
case corev1.Protocol("TCP_UDP"): | |||
fallthrough |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we not need to build a port-list with both TCP and UDP here? buildTargetGroupBindingNetworkingLegacy
does it that way, but I admit I'm not familiar with the difference in these two functions.
I also note that the test for this function has not had a TCP_UDP
test added, while buildTargetGroupBindingNetworkingLegacy
's test has been updated.
The original PR had a test for mergeServicePortsForListener tacked on. That test has been moved into the test for mergeServicePortsForListener. This also fixes the test to deal with the error checking that was added to mergeServicePortsForListener.
OK, some of these have been addressed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ugh, I hate Github's UI. OK, comments for some of what's been addressed.
} | ||
} | ||
|
||
// execute build listener | ||
for _, port := range t.service.Spec.Ports { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes.
The complete map needs to be populated before building the listener for the AWS load balancer. This might be easier to understand with an example.
Say the service exports out the following ports: 53/UDP, 53/TCP and 443/TCP.
The loop above would create a port map like this with that data (pseudo code):
53: [53/UDP, 53/TCP],
443: [443/TCP]
}
Now when we go through this loop we can see if any port has been set for TCP and UDP. If so, it creates a single TCP_UDP port which is what the AWS Load Balancer requires.
This is the most efficient way to do this - the portMap
map is an optimisation. Otherwise for each t.service.Spec.Ports
we'd need to check all remaining ports to see if we need to allocate a TCP_UDP port. We'd also need to track if we already allocated a port as a TCP_UDP port.
@@ -234,3 +260,20 @@ func (t *defaultModelBuildTask) buildListenerAttributes(ctx context.Context, svc | |||
} | |||
return attributes, nil | |||
} | |||
|
|||
func mergeServicePortsForListener(ports []corev1.ServicePort) (corev1.ServicePort, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would need to completely recreate the corev1.ServicePort
type. The only change would be to the Protocol
member.
The port
parameter gets used by (not an exhaustive list) buildListener
, buildListenerSpec
, buildTargetGroup
. In places it uses k8s
modules so I would need a way to convert this custom type to a temporary variable of the corev1.ServicePort
. How would the conversion work - what is TCP_UDP changed into? Right now we don't call anything that references the Protocol
part of the corev1.ServicePort
, but we'd need to populate it with something.
I'm happy to do this but it will involve changing an awful lot of code, adding routines to convert between this custom type and corev1.ServicePort
and I'm dubious it will make things more understandable.
port := ports[0] | ||
port.Protocol = corev1.Protocol("TCP_UDP") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We know they match because that's what the portMap above did - it grouped by port.
@@ -87,6 +87,9 @@ const ( | |||
|
|||
// NetworkingProtocolUDP is the UDP protocol. | |||
NetworkingProtocolUDP NetworkingProtocol = "UDP" | |||
|
|||
// NetworkingProtocolTCP_UDP is the TCP_UDP protocol. | |||
NetworkingProtocolTCP_UDP NetworkingProtocol = "TCP_UDP" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm, this type is used for a lot of things. For some TCP
, UDP
and TCP_UDP
are allowed. For others, only TCP
and UDP
.
Should I split out the types across the code base? It makes the PR bigger, but it would work better with kubebuilder.
This is a work in progress - I need to test it.
Issue
#1608 (comment)
And based on this PR: #2275
Description
Previously, aws-load-balancer-controller ignored extra overlapping
ServicePorts defined in the Kubernetes Service spec if the external port
numbers were the same even if the protocols were different (e.g. TCP:53,
UDP:53).
This behavior prevented users from exposing services that support TCP
and UDP on the same external load balancer port number.
This patch solves the problem by detecting when a user defines multiple
ServicePorts for the same external load balancer port number but using
TCP and UDP protocols separately. In such situations, a TCP_UDP
TargetGroup and Listener are created and SecurityGroup rules are
updated accordingly. If more than two ServicePorts are defined, only the
first two mergeable ServicePorts are used. Otherwise, the first
ServicePort is used.
Checklist
README.md
, or thedocs
directory)BONUS POINTS checklist: complete for good vibes and maybe prizes?! 🤯