Feature request: retain last-known-good Policy (or make failure mode configurable) instead of returning 500 when a referenced Policy becomes invalid

[wd]

Is your feature request related to a problem? Please describe.

Yes. When a Policy resource that is referenced by VirtualServers becomes invalid, NGINX Ingress Controller replaces every affected route with a return 500. This is the documented behavior (VirtualServer.Policy docs):

"If the policy doesn't exist or invalid, NGINX will respond with an error response with the 500 status code."

We recently had a production-impacting incident caused by this. An accessControl Policy that guards internal routes across the whole cluster received an update in which one entry in the allow list was not a valid CIDR (a templating artifact from our config pipeline). What happened:

1. The apply itself succeeded — nothing rejected the invalid CIDR at submission time. There is no admission-time validation that would have stopped it.
2. NIC then detected the Policy as invalid and rejected it internally (status Rejected).
3. Instead of continuing to serve with the previous version of the Policy — which had been valid and serving traffic for months — NIC switched every route referencing the Policy to 500.

So one malformed list entry in one widely-referenced Policy took down every VirtualServer route guarded by it, cluster-wide, instantly. The config mistake was ours, but the blast radius was decided by NIC: there was no point in the pipeline where the bad change could be stopped, and no fail-safe once it landed. For a data-plane component, "one bad line in a shared policy = whole-site outage" is very unexpected behavior.

Describe the solution you'd like

Either (or both) of the following would have prevented the disaster — we'd be happy with either:

1. Reject the invalid change at submission time. Validate Policy contents (e.g. that accessControl.allow/deny entries are valid IPs/CIDRs) in a validating admission webhook, so a kubectl apply / controller apply of an invalid Policy fails loudly at the API server instead of being accepted and then breaking traffic. Structural/CRD validation clearly doesn't catch this today.

2. Keep the last-known-good configuration. When a previously-valid Policy is updated into an invalid state, keep enforcing the last valid version (and surface the rejection via Status and an Event, as today) instead of replacing the affected routes with return 500. NIC already embraces this principle elsewhere: the GlobalConfiguration docs promise that an invalid update is ignored and the previous version stays in effect, and in NIC removes all listeners when rejecting a new one on a reserved port #4775 maintainers acknowledged teardown-on-rejection as "not the experience we want customers to have".

For the case of a Policy that is invalid at creation (no last-known-good exists) or a reference to a non-existent Policy, the current 500 behavior is defensible — failing open on an access-control policy would be a security hole. We are not asking to change that. Note that retaining the last-known-good version is also fail-closed: the previously-intended restrictions stay enforced. A hard 500 is stricter than fail-closed — it converts every partial config mistake into downtime.

Describe alternatives you've considered

• Client-side prevention only (manifest validation in CI): we do this now, but it cannot fully close the window — any automated deploy pipeline can produce a transiently invalid manifest, and the data plane is the last line of defense.
• A configurable failure mode on the policy reference or via ConfigMap, e.g. policyFailureMode: retain-last-good | reject-with-500, defaulting to today's behavior for backward compatibility, if changing the default is not acceptable.
• errorPages to mask the 500: only changes the response cosmetics; traffic is still broken.

Additional context

• NIC version: 3.x (OSS), deployed via the official nginx-ingress Helm chart.
• Related precedent in this repo: NIC removes all listeners when rejecting a new one on a reserved port #4775 (GlobalConfiguration teardown on rejection, acknowledged as undesirable UX).
• Happy to provide a full reproduction (Policy + VirtualServer manifests) if useful.

[github-actions]

Hi @wd thanks for reporting!

Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this 🙂

If your issue concerns any of the enterprise features please open a ticket with F5 directly (For NGINX Plus customers NGINX Ingress Controller (when used with NGINX Plus) is covered by the support contract.)

Cheers!

[AlexFenlon]

Hi @wd, thank you so much for the suggestion, the team will discuss this internally and let you know!

[AlexFenlon]

Hi @.wd,

We are looking into making our validation on CRDS even more robust so changes don't make it passed the API level.

Really apprecate the feedback on this. Noted down.

We will need more internal discussion about this.