NGINX: Correctly determine client IP.

[dmeremyanin]

What this PR does / why we need it:

This update ensures that the correct client IP address is determined in Ingress NGINX when the PROXY protocol is enabled. The logic now properly handles proxy chains by checking trusted proxy addresses and using the appropriate forwarded-for headers.

Here's an example when Cloudflare is used:

flowchart LR
    user(User) --> proxy
    proxy(Cloudflare) -- http --> lb
    subgraph VPC
    lb(TCP LB) -- proxy protocol --> nginx(ingress-nginx)
    end

Loading

In this scenario, Cloudflare's IP address appears as the client IP. This PR ensures that the actual client IP is correctly identified, even when Cloudflare (or other proxies) are involved.

The algorithm

To ensure the correct client IP is identified, we need to enable use-forwarded-headers and properly configure proxy-real-ip-cidr to trust all intermediate proxies (both within the private network and any external proxies). Caution: By default, the proxy-real-ip-cidr is set to 0.0.0.0/0, which means all IP addresses are trusted.

The process works as follows:

1. Check if the $proxy_protocol_addr is trusted (i.e., whether it's in the proxy-real-ip-cidr list).
2. If trusted, use the configured forwarded-for header (X-Forwarded-For by default).
3. If not trusted, fall back to using the $proxy_protocol_addr.

Once the appropriate header is determined, its value is used with the real_ip_header directive, allowing NGINX to correctly identify the client IP.

---

Issues addressed by this PR:

• Allow customizing the real_ip_header directive when proxy_protocol is enabled #11623
• Add a way to customize X-Forwarded-For #10749
• client IP is incorrect when using CloudFront -> ELB -> Nginx and proxy protocol #5233
• Cloudflare & Proxy Protocol #4731
• and others

In these issues, a workaround was suggested, such as adding the following annotation:

nginx.ingress.kubernetes.io/server-snippet: |
  real_ip_header X-Forwarded-For;

However, this approach may not work correctly in scenarios where clients can connect directly to the TCP load balancer (bypassing the trusted HTTP proxy):

1. Security risk: Any client may be able to forge the X-Forwarded-For header, potentially spoofing the client IP.
2. Incorrect client IP detection: If the X-Forwarded-For header is not passed, the TCP load balancer's IP address will be used as the client IP. In such cases, the proxy_protocol should be used to correctly identify the client IP.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• CVE Report (Scanner found CVE and adding report)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation only

It looks more like a new feature because it's now possible to use use-proxy-protocol with use-forwarded-headers configuration option to change the behavior of client IP address identifying.

How Has This Been Tested?

I tested the functionality in a test environment simulating a typical production setup. The environment includes a Layer 7 proxy (Cloudflare) in front of a cloud-based TCP load balancer with the PROXY protocol enabled, and Ingress NGINX Controller within the K8S cluster. To verify the results, I exposed the whoami service using the Ingress Controller and inspected the incoming X-Real-IP and X-Forwarded-For headers reaching the service.

Here's the setup (IPs are fictional):

1. Client IP address is 100.64.0.1
2. HTTP proxy IP address is 198.18.0.1 (passing the real client IP in the X-Forwarded-For header)
3. TCP LB private IP address is 10.1.1.1 (with PROXY protocol enabled)
4. Kubernetes worker node IP address is 10.2.1.1
5. Config option use-proxy-protocol is enabled in Ingress NGINX Controller

Testing Scenarios

With HTTP Proxy (client → HTTP proxy → TCP LB → Ingress Controller)

1. Neither use-forwarded-headers nor proxy-real-ip-cidr are configured.

   Result: 🔴 HTTP proxy IP 198.18.0.1 is identified as the client IP.

2. use-forwarded-headers disabled, proxy-real-ip-cidr set to 198.18.0.0/24 to trust the HTTP proxy subnet.

   Result: 🔴 Kubernetes worker node IP 10.2.1.1 is identified as the client IP.

3. use-forwarded-headers disabled, proxy-real-ip-cidr set to 10.0.0.0/8,198.18.0.0/24 to the trust private network and the HTTP proxy subnet.

   Result: 🔴 HTTP proxy IP 198.18.0.1 is identified as the client IP.

4. use-forwarded-headers enabled, proxy-real-ip-cidr not configured (0.0.0.0/0 by default).

   Result: 🟢 Correct client IP 100.64.0.1 is identified.

5. use-forwarded-headers enabled, proxy-real-ip-cidr not configured, passed custom "X-Forwarded-For: 1.1.1.1" header.

   Result: 🔴 Spoofed IP 1.1.1.1 is used as the client IP.

6. use-forwarded-headers enabled, proxy-real-ip-cidr set to 198.18.0.0/24 to trust the HTTP proxy subnet.

   Result: 🔴 Kubernetes worker node IP 10.2.1.1 is identified as the client IP.

7. use-forwarded-headers enabled, proxy-real-ip-cidr set to 10.0.0.0/8,198.18.0.0/24 to trust the private network and the HTTP proxy subnet.

   Result: 🟢 Correct client IP 100.64.0.1 is identified.

8. use-forwarded-headers enabled, proxy-real-ip-cidr set to 10.0.0.0/8,198.18.0.0/24 to trust the private network and the HTTP proxy subnet, passed custom "X-Forwarded-For: 1.1.1.1" header.

   Result: 🟢 Correct client IP 100.64.0.1 is identified.

---

Without intermediate HTTP Proxy (client → TCP LB → Ingress Controller)

1. Neither use-forwarded-headers nor proxy-real-ip-cidr are configured.

   Result: 🟢 Correct client IP 100.64.0.1 is identified.

2. use-forwarded-headers disabled, proxy-real-ip-cidr set to 198.18.0.0/24 to trust the HTTP proxy subnet.

   Result: 🔴 Kubernetes worker node IP 10.2.1.1 is identified as the client IP.

3. use-forwarded-headers disabled, proxy-real-ip-cidr set to 10.0.0.0/8,198.18.0.0/24 to trust the private network and the HTTP proxy subnet.

   Result: 🟢 Correct client IP 100.64.0.1 is identified.

4. use-forwarded-headers enabled, proxy-real-ip-cidr not configured (0.0.0.0/0 by default).

   Result: 🔴 Kubernetes worker node IP 10.2.1.1 is identified as the client IP.

5. use-forwarded-headers enabled, proxy-real-ip-cidr not configured, passed "X-Forwarded-For: 1.1.1.1" header.

   Result: 🔴 Spoofed IP 1.1.1.1 is used as the client IP.

6. use-forwarded-headers enabled, proxy-real-ip-cidr set to 198.18.0.0/24 to trust the HTTP proxy subnet.

   Result: 🔴 Kubernetes worker node IP 10.2.1.1 is identified as the client IP.

7. use-forwarded-headers enabled, proxy-real-ip-cidr set to 10.0.0.0/8,198.18.0.0/24 to trust the private network and the HTTP proxy subnet.

   Result: 🟢 Correct client IP 100.64.0.1 is identified.

8. use-forwarded-headers enabled, proxy-real-ip-cidr set to 10.0.0.0/8,198.18.0.0/24 to trust the private network and the HTTP proxy subnet, passed custom "X-Forwarded-For: 1.1.1.1" header.

   Result: 🟢 Correct client IP 100.64.0.1 is identified.

Conclusion and Questions

1. The default client IP identification behavior remains unchanged. The identification process only behaves differently when both use-proxy-protocol and use-forwarded-headers are enabled.
2. I've added only two end-to-end tests to cover the main functionality because they are very slow to run, so it's a bit of a compromise.
3. Wouldn't it be better to set proxy-real-ip-cidr to something like 10.0.0.0/8 by default? Setting this default would help prevent misconfigurations and ensure that only trusted internal IP ranges are accepted, reducing the risk of spoofed IPs.

Checklist:

• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I've read the CONTRIBUTION guide
• I have added unit and/or e2e tests to cover my changes.
• All new and existing tests passed.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: dmeremyanin / name: DM (83b2394)

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-ci-robot]

Welcome @dmeremyanin!

It looks like this is your first PR to kubernetes/ingress-nginx 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/ingress-nginx has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @dmeremyanin. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[netlify]

✅ Deploy Preview for kubernetes-ingress-nginx ready!

Name	Link
🔨 Latest commit	83b2394
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-ingress-nginx/deploys/67a2a030252a9100086ba005
😎 Deploy Preview	https://deploy-preview-12768--kubernetes-ingress-nginx.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[dmeremyanin]

I've updated the docs to reflect the changes.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dmeremyanin, lubervn
Once this PR has been reviewed and has the lgtm label, please assign strongjz for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dmeremyanin]

Hey @Gacko @tao12345666333, is there anything I need to do to get the PR reviewed, or should I just be patient and wait?

[Gacko]

Hello @dmeremyanin,

time. Just time. 😬 Sorry to say, but we currently are something between 2 and 3 maintainers also having a day to day job and still a lot of other maintenance work to do and PRs to merge before we can proceed to the latest ones.

Kind regards
Marco

[claneys]

Hello,

This PR solves one of our problem here and avoid us to maintain a piece of lua code to achieve the same :) Our setup use Cloudfront in front of an NLB then ingress-nginx. We tested it in one of our cluster and works as expected.

Thanks you very much.

[lubervn]

@dmeremyanin Thank you, we are using exact the same setup and your PR works well for us.
+1 to merge it, please