[BUG] auth_failure_listeners does not block users after tries with wrong password

[dmantas]

What is the bug?
In the documentation API Rate Limiting based on username is described.
However, it doesn't seem to work. After trying to authenticate with a wrong password allowed_tries times, the user doesn't get blocked and, giving the correct password, still has access to the API.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Have this config in place:

_meta:
  type: "config"
  config_version: "2"
config:
    dynamic:
      kibana:
        multitenancy_enabled: true
        private_tenant_enabled: true
        default_tenant: global tenant
        server_username: kibanaserver
        index: '.kibana'
      do_not_fail_on_forbidden: false
      http:
        anonymous_auth_enabled: false
      authc:
        basic_internal_auth_domain:
          http_enabled: true
          transport_enabled: true
          order: 1
          http_authenticator:
            type: basic
            challenge: true
          authentication_backend:
            type: intern
        openid_auth_domain:
          http_enabled: true
          transport_enabled: true
          order: 0
          http_authenticator:
            type: openid
            challenge: false
            config:
              subject_key: preferred_username
              roles_key: roles
              openid_connect_url: https://my.openid.provider.com/auth/realms/my-realm/.well-known/openid-configuration
              openid_connect_idp.enable_ssl: true
              openid_connect_idp.pemtrustedcas_filepath: /usr/share/opensearch/config/my-ca/rootcax1.crt
          authentication_backend:
            type: noop
        ldap:
          http_enabled: true
          transport_enabled: true
          order: 3
          http_authenticator:
            type: basic
            challenge: true
          authentication_backend:
            type: ldap
            config:
              pemtrustedcas_filepath: /usr/share/opensearch/config/my-ca/rootcax1.crt
              enable_ssl: true
              enable_start_tls: false
              enable_ssl_client_auth: false
              verify_hostnames: false
              hosts:
                - my.ldap.provider.com:636
              bind_dn: cn=my-cn,ou=my-ou,ou=my-ou-2,dc=my-dc,dc=com
              password: ldap_pass
              userbase: 'ou=users,ou=my-ou,dc=my-dc,dc=com'
              usersearch: '(uid={0})'
              username_attribute: uid           
      authz:
        ldap:
          http_enabled: true
          transport_enabled: true
          authorization_backend:
            type: ldap
            config:
              pemtrustedcas_filepath: /usr/share/opensearch/config/my-ca/rootcax1.crt
              enable_ssl: true
              enable_start_tls: false
              enable_ssl_client_auth: false
              verify_hostnames: false
              hosts:
                - my.ldap.provider.com:636
              bind_dn: cn=my-cn,ou=my-ou,ou=my-ou-2,dc=my-dc,dc=com
              password: ldap_pass
              userbase: 'ou=users,ou=my-ou,dc=my-dc,dc=com'
              usersearch: '(uid={0})'
              username_attribute: uid
              rolebase: 'ou=groups,dc=my-dc,dc=com'
              rolesearch: '(uniqueMember={0})'
              userroleattribute: null
              userrolename: none
              rolename: cn
              resolve_nested_roles: false
              skip_users:
                - admin
                - kibanaserver
                - kibanaro
                - logstash
                - dashboarduser
                - readall
                - anomalyadmin
      auth_failure_listeners:
        internal_authentication_backend_limiting:
          type: username
          authentication_backend: internal
          allowed_tries: 3
          time_window_seconds: 60
          block_expiry_seconds: 60
          max_blocked_clients: 100000
          max_tracked_clients: 100000

2. Try to authenticate with a user giving wrong password, a number of times:

curl --silent  -u username:wrong_password 'https://my-opensearch-api.com/_cluster/health'    

Authentication is failing with Unauthorized.

3. Try to authenticate again giving the correct password. Authentication works and a response is returned.

What is the expected behavior?
We expected that after a number of failed authentications, the correct password wouldn't work, but it did.

What is your host/environment?

• OS: Kubernetes (using Opensearch Operator)
• Version 2.18
• Plugins

Do you have any additional context?
I have the following questions:

• Is there any known issue with this functionality?
• We are running a multi-node cluster with 3 cluster managers and 3 data nodes. Is allowed_tries counted in each node by any chance?
• Does the blocking work only for users in internal_users.yml or does it also work for users created using REST API / Dashboards as well?

[dmantas]

OK, after some testing I understand that the blocking is per node. So, I tried to send e.g. 50 requests with wrong password and the account was blocked on all nodes, I even saw some logs:

Blocking username

Is this how it's supposed to work?

Moreove, I see that users created via Dashboards and REST API are also blocked, which is good.

What about users that authenticate via LDAP?

I see this for an LDAP user:

[2025-10-10T10:54:29,066][INFO ][o.o.s.a.b.HeapBasedClientBlockRegistry] [cca-opensearch-dev-cluster-managers-2] Blocking my_ldap_user
[2025-10-10T10:54:29,066][INFO ][o.o.s.a.b.HeapBasedClientBlockRegistry] [cca-opensearch-dev-cluster-managers-2] Unblocking my_ldap_user

So blocking/unblocking simultaneously.

Any ideas?

[cwperks]

[Triage] that's odd that it shows both blocking and unblocking in the same call. looks like a bug.