Skip to content

Adopt Envoy in place of HAProxy for load balancing #4064

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shwetha-s-poojary wants to merge 1 commit into
base: main
Choose a base branch
from
+92 −40
Open

Adopt Envoy in place of HAProxy for load balancing #4064

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions pkg/cluster/internal/create/actions/loadbalancer/loadbalancer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import (

"sigs.k8s.io/kind/pkg/cluster/constants"
"sigs.k8s.io/kind/pkg/errors"
"sigs.k8s.io/kind/pkg/exec"
"sigs.k8s.io/kind/pkg/internal/apis/config"

"sigs.k8s.io/kind/pkg/cluster/internal/create/actions"
Expand Down Expand Up @@ -90,9 +91,9 @@ func (a *Action) Execute(ctx *actions.ActionContext) error {
return errors.Wrap(err, "failed to copy loadbalancer config to node")
}

// reload the config. haproxy will reload on SIGHUP
if err := loadBalancerNode.Command("kill", "-s", "HUP", "1").Run(); err != nil {
return errors.Wrap(err, "failed to reload loadbalancer")
// restart loadbalancer to apply static configuration changes
if err := exec.Command("docker", "restart", "kind-external-load-balancer").Run(); err != nil {
Copy link
Contributor

@stmcginnis stmcginnis Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Others, please correct me if I am wrong, but I don't believe we want to explicitly call docker commands here. User may be using podman or nerdctl and not have docker present on their system. Is there an equivalent to sending HUP to envoy?

Copy link
Contributor

@aojea aojea Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you do not need to restart , see how is done in cloud provider kind https://github.com/kubernetes-sigs/cloud-provider-kind/blob/4820e03979d5ff7fbcd8645794db2ff39a608b3f/pkg/loadbalancer/proxy.go#L276-L313

Copy link
Member

@BenTheElder BenTheElder Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we should not exec directly to docker in particular here, but we also don't need to.

Copy link
Author

@shwetha-s-poojary shwetha-s-poojary Dec 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stmcginnis envoy does support hot restarts, but for this scenario a simple container restart seems to be the most reliable and straightforward option. I’ll look into detecting the container runtime so we can use the appropriate tool instead of assuming docker.
@aojea @BenTheElder from my understanding, only dynamic (xDS) configuration avoids restarts; static configuration still requires one. For the kind use case, static config seems sufficient and avoids the complexity of a full dynamic config setup. The cloud-provider-kind example you shared above uses dynamic config, but I believe the earlier version relied on static config and required restarts (ref: PR link).
Please correct me if I’m mistaken anywhere — I appreciate the guidance.

Copy link
Contributor

@aojea aojea Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shwetha-s-poojary if we know all the configuration before hand you just need to pass the final config during the entrypoint and you can just use existing libraries to create the container, see

https://github.com/kubernetes-sigs/cloud-provider-kind/blob/dc1a6e4c3b21716be9f87476d5cdb78389f05537/pkg/loadbalancer/server.go#L247-L263

the example I provided in my previous comment is to update the configuration dynamically without having to restart the container, you can write files on the filesystem of the container and envoy will pick the config and autoconfigure itself, no need to restarts

Copy link
Member

@BenTheElder BenTheElder Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’ll look into detecting the container runtime so we can use the appropriate tool instead of assuming docker.

I'd really rather not expand the node container runtime abstraction just for this purpose, haproxy has been working ~fine.

But see antonio's comment above.

Copy link
Contributor

@aojea aojea Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shwetha-s-poojary so you need to follow the cloud-provider-kind implementation, a straw man approach will be

  1. bootstrap the container with an initial config that allow to use dynamic configuration based on files

unfortunately I think this is per provider https://github.com/search?q=repo%3Akubernetes-sigs%2Fkind%20runArgsForLoadBalancer&type=code

  1. at runtime, add the expected configuration

create the config as in cloudprovider kind

https://github.com/kubernetes-sigs/cloud-provider-kind/blob/4820e03979d5ff7fbcd8645794db2ff39a608b3f/pkg/loadbalancer/proxy.go#L82-L273

and write it to the corresponding node in the expected path (configured in step 1)

kind/pkg/cluster/internal/create/actions/loadbalancer/loadbalancer.go

Lines 77 to 91 in 02382a0

// create loadbalancer config data
loadbalancerConfig, err := loadbalancer.Config(&loadbalancer.ConfigData{
ControlPlanePort: common.APIServerInternalPort,
BackendServers: backendServers,
IPv6: ctx.Config.Networking.IPFamily == config.IPv6Family,
})
if err != nil {
return errors.Wrap(err, "failed to generate loadbalancer config data")
}
// create loadbalancer config on the node
if err := nodeutils.WriteFile(loadBalancerNode, loadbalancer.ConfigPath, loadbalancerConfig); err != nil {
// TODO: logging here
return errors.Wrap(err, "failed to copy loadbalancer config to node")
}

Copy link
Author

@shwetha-s-poojary shwetha-s-poojary Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aojea @BenTheElder @stmcginnis Thanks for the guidance! Looks like switching to dynamic config is the way to go.I'll get on it.

return errors.Wrap(err, "failed to restart loadbalancer container")
}

ctx.Status.End(true)
Expand Down
121 changes: 86 additions & 35 deletions pkg/cluster/internal/loadbalancer/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ package loadbalancer

import (
"bytes"
"net"
"text/template"

"sigs.k8s.io/kind/pkg/errors"
Expand All @@ -32,53 +33,103 @@ type ConfigData struct {

// DefaultConfigTemplate is the loadbalancer config template
const DefaultConfigTemplate = `# generated by kind
global
log /dev/log local0
log /dev/log local1 notice
daemon
# limit memory usage to approximately 18 MB
maxconn 100000

resolvers docker
parse-resolv-conf

defaults
log global
mode tcp
option dontlognull
# TODO: tune these
timeout connect 5000
timeout client 50000
timeout server 50000
# allow to boot despite dns don't resolve backends
default-server init-addr none

frontend control-plane
bind *:{{ .ControlPlanePort }}
{{ if .IPv6 -}}
bind :::{{ .ControlPlanePort }};
static_resources:
listeners:
- name: control-plane-listener-ipv4
address:
socket_address:
address: "0.0.0.0"
port_value: {{ .ControlPlanePort }}
filter_chains:
- filters:
- name: envoy.filters.network.tcp_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
stat_prefix: kube_apiserver
cluster: kube_apiservers
{{- if .IPv6 }}
- name: control-plane-listener-ipv6
address:
socket_address:
address: "::"
port_value: {{ .ControlPlanePort }}
filter_chains:
- filters:
- name: envoy.filters.network.tcp_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
stat_prefix: kube_apiserver
cluster: kube_apiservers
{{- end }}
default_backend kube-apiservers
clusters:
- name: kube_apiservers
type: STRICT_DNS
connect_timeout: 5s
lb_policy: ROUND_ROBIN
dns_lookup_family: AUTO
circuit_breakers:
thresholds:
- priority: DEFAULT
max_connections: 100000
max_pending_requests: 20000
max_requests: 100000
load_assignment:
cluster_name: kube_apiservers
endpoints:
- lb_endpoints:
{{- range $server, $address := .BackendServers }}
{{- $hp := hostPort $address }}
- endpoint:
address:
socket_address:
address: {{ $hp.host }}
port_value: {{ $hp.port }}
{{- end }}

backend kube-apiservers
option httpchk GET /healthz
# TODO: we should be verifying (!)
{{range $server, $address := .BackendServers}}
server {{ $server }} {{ $address }} check check-ssl verify none resolvers docker resolve-prefer {{ if $.IPv6 -}} ipv6 {{- else -}} ipv4 {{- end }}
{{- end}}
health_checks:
- timeout: 5s
interval: 10s
unhealthy_threshold: 3
healthy_threshold: 1
http_health_check:
path: /livez
expected_statuses:
- start: 200
end: 399
codec_client_type: HTTP1
admin:
address:
socket_address:
address: 127.0.0.1
port_value: 9901
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/stdout
`

func hostPort(addr string) (map[string]string, error) {
host, port, err := net.SplitHostPort(addr)
if err != nil {
return nil, err
}
return map[string]string{"host": host, "port": port}, nil
}

// Config returns a kubeadm config generated from config data, in particular
// the kubernetes version
func Config(data *ConfigData) (config string, err error) {
t, err := template.New("loadbalancer-config").Parse(DefaultConfigTemplate)
funcs := template.FuncMap{
"hostPort": hostPort,
}
t, err := template.New("envoy-config").Funcs(funcs).Parse(DefaultConfigTemplate)
if err != nil {
return "", errors.Wrap(err, "failed to parse config template")
}
// execute the template
var buff bytes.Buffer
err = t.Execute(&buff, data)
if err != nil {
if err := t.Execute(&buff, data); err != nil {
return "", errors.Wrap(err, "error executing config template")
}
return buff.String(), nil
Expand Down
4 changes: 2 additions & 2 deletions pkg/cluster/internal/loadbalancer/const.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ limitations under the License.
package loadbalancer

// Image defines the loadbalancer image:tag
const Image = "docker.io/kindest/haproxy:v20230606-42a2262b"
const Image = "docker.io/envoyproxy/envoy:v1.36.2"

// ConfigPath defines the path to the config file in the image
const ConfigPath = "/usr/local/etc/haproxy/haproxy.cfg"
const ConfigPath = "/etc/envoy/envoy.yaml"