Consider alternatives to insecure entrypoints

[jspdown]

Proposal

The indirect whitelist middlewares, used by routers in the context of a TrafficSplit backend, relies on the X-Forwarded-For header. In order to get this header, the proxy needs to be trusted by either explicitly listing trusted sources or by declaring the entrypoint as insecure.
Due to the dynamic nature of the number of proxies, as being instantiated by a DaemonSet, we can't build easily a list of trusted sources. While rewriting the Provider, it has been chosen to rely on insecured entrypoint.

This preliminary solution is far from perfect and we should consider having a stricter way of establishing a list of trusted sources.

Workarounds

The current workaround is to use entrypoint.insecure=true https://docs.traefik.io/routing/entrypoints/#forwarded-headers

[jspdown]

The forwardedHeader option of an endpoint controls how X-Forwarded-* headers will be trusted. If not provided, a recursive request (used for implementing TrafficSplits) won't be able to rely on the X-Forwarded-For for whitelisting.

To avoid using the entrypoint.forwardedHeader.insecure we have to provide a list of trusted source using the entrypoint.forwardedHeaders.trustedIPs. This list must contain the IP addresses of all the traefik pods. As this list is provided in the static configuration via a flag, if a new node comes in, a new IP will have to be added. Meaning, all proxies will have to be restarted.

One additional challenge for moving to trustedIPs is a strange behavior in Kubernetes: When making a pod-to-self request via a ClusterIP service, the source IP is not what you expect. Typically, in a pod-to-service request the source IP is the client pod IP. But in this context it gets SNATed and the source IP won't be the client pod IP. So, adding traefik pods IP, right now, won't solve the issue.

[jspdown]

I see 2 options on how we could go further on this:

• Remove the need to rely on X-Forwarded-* headers by preventing users to define middlewares on TrafficSplit backends. If we do this, TrafficSplits could be implemented without a recursive call to traefik.
• Always make pod-to-self request when reaching TrafficSplits backends. This way, each proxy would have just one trustedIP which is itself. This solution can be implemented only there's actually in bug in how kubernetes preserves the source IP in the context of a pod-to-self request and if it get fixed.