[BUG] Security config updates cause valid credentials to be rejected, tripping ip_rate_limiting and locking out all clients on a shared source IP

[lyuboto]

What is the bug?

Writing to _plugins/_security/api/* (creating an internal user, role, or role mapping) triggers a cluster-wide security config reload and invalidates the user cache. During that window, clients presenting valid credentials are rejected with 401 Authentication finally failed. It started happening right after the cluster was upgraded from 3.5 to 3.7.

Those spurious rejections are counted as authentication failures by auth_failure_listeners. When multiple clients share a source IP — a NAT gateway, egress proxy, or Kubernetes node — the default ip_rate_limiting threshold of 10 failures in 60 seconds is crossed within seconds, and the entire source IP is blocked.

The net effect is that routine provisioning locks out every client behind one address, and the clients being blocked are the ones using correct credentials. The rate limiter is working as designed; it is being fed failures that should never have occurred.

This appears to be the same underlying fault as #5544, where valid credentials sporadically fail with Authentication finally failed immediately after the user cache is "cleared due to EXPIRED" — but reproduced deterministically by forcing the cache invalidation with a config write, and with the added consequence of tripping the IP limiter.

Two aggravating factors:

• The outage outlasts a single block. block_expiry_seconds does expire, but on release the accumulated clients re-authenticate simultaneously against a still-cold cache, produce a fresh batch of spurious rejections, and re-trip the limiter. We have seen a source IP effectively unavailable for over an hour from a single provisioning operation.
• The rejections are invisible. No FAILED_LOGIN audit events are emitted, even with the category enabled. The only signal is the INFO-level Blocking /<ip> line, which managed offerings do not surface (WARN/ERROR only). There is also no API to list currently blocked clients, so an operator sees a total outage with no diagnostic trail.

How can one reproduce the bug?

1. Cluster with FGAC + internal user database and the default auth failure listener:

   {"ip_rate_limiting": {"type": "ip", "allowed_tries": 10, "time_window_seconds": 60,
    "block_expiry_seconds": 180, "max_blocked_clients": 10000, "max_tracked_clients": 10000}}

2. Have a population of clients behind a single source IP continuously authenticating with valid basic-auth credentials. In our case ~97 distinct internal users, ~500–600 authenticated requests per 5 minutes.
3. From any source, perform one security config write:

   PUT /_plugins/_security/api/internalusers/canary   {"password": "..."}
   

   The source of the write does not matter — writes issued from an unrelated IP produce the same result.
4. Immediately probe from the shared IP with known-good credentials:

   GET /_plugins/_security/api/account
   → 401 {"status":"error","reason":"Authentication finally failed"}
   

5. The same credentials from a different source IP return 200 concurrently, confirming the credentials are valid and the rejection is scoped to the blocked address.
6. With audit logging enabled and FAILED_LOGIN not in disabled_rest_categories, zero FAILED_LOGIN events are recorded for the affected IP.

What is the expected behavior?

1. A security config reload must not cause clients with valid credentials to be rejected. Cache invalidation should degrade authentication to a slower path (re-evaluating the backend), never to a failure.
2. Failing that, rejections originating from internal reload/cache state must not be counted by auth_failure_listeners. Only genuine credential failures should arm the limiter.
3. Any rejection that does count toward the limiter should be auditable, so operators can identify what armed it.

What is your host/environment?

• OS: Amazon Linux (Amazon OpenSearch Service, managed)
• Version: OpenSearch 3.7
• Plugins: opensearch-security — FGAC enabled, internal user database enabled, SAML enabled, anonymous auth disabled
• Topology: single-node t3.medium.search; ~100 clients reaching the domain through one NAT gateway address

Do you have any screenshots?

No screenshots. Audit-log evidence instead — authenticated activity from the shared IP, 5-minute buckets, around a provisioning operation:

bucket(5m)   events   distinct_users
09:40:00        428       97
09:35:00        605       97
09:30:00        572       97
09:25:00        594       97
(traffic stops at ~09:45, immediately after a security config write)

Authenticated traffic from 97 distinct valid users stops dead, and no FAILED_LOGIN events appear in that window.

Do you have any additional context?

What we ruled out, in case it saves triage time:

• Not the credentials. The same credential returned 200 from one source IP and 401 from another at the same time.
• Not the client. Reproduced with urllib.request, urllib3, opensearch-py, and plain curl.
• Not the resource access policy. A policy denial returns 403 with {"Message": ...}; we receive 401 with Authentication finally failed, which is the security plugin's own response.
• Not cluster health. CPU 27–57%, JVM memory pressure 62–83%, cluster status green throughout.

Impact. Any deployment that provisions internal users or roles at runtime — multi-tenant systems in particular — will intermittently lock out its entire client fleet whenever those clients share an egress address. On managed
services securityconfig is read-only:

{"Message":"Your request: '/_plugins/_security/api/securityconfig' is not allowed."}

so operators cannot disable or tune the listener as a workaround.

[github-actions]

Possible Duplicate

Found 1 possible duplicate issue(s):

• [BUG] - Rare and random unexpected 401 with correct credentials #5544

If this is not a duplicate:

• Add a comment on this issue, or
• 👎 this comment to prevent auto-closure

Otherwise, this issue will be automatically closed in 7 days.

[dmtar]

@cwperks I hope you can shed some light on that, it will be highly appreciated!

[cwperks]

@dmtar is this self-managed OpenSearch?

[lyuboto]

@cwperks this is AWS-managed Opensearch service

[cwperks]

@lyuboto thank you for confirming. We will get back with updates ASAP, can you please also open a support case for tracking and communication?

[dmtar]

@cwperks Any updates on this?

[lyuboto]

@cwperks do you have any update on this? Were you able to reproduce the issue? I'm not able to open an AWS support case.