Backend Throttling and Burst Control Malfunctioning in Hybrid Mode After Losing Connection to Redis Cache Server Following New Update

[sajith-madhusanka]

Description

When the Hybrid throttle mode is enabled and connectivity with the Redis cache server is lost, requests to the backend server are allowed in the initial second until the allocated local quota is reached. However, afterwards, requests fail with a 500 HTTP status code, and the TPS drastically decreases.

{"description":"Error during validating the request","message":"Unclassified Validation Failure","code":"102500"}

Steps to Reproduce

Deploy a setup similar to the one described below:

TEST SETUP

• Choreo Connect Router 1 node (image tag: 1.0.0.26)
• Choreo Connect Enforcer 1 node (image tag: 1.0.0.26)
• Choreo Connect Adapter 1 node (image tag: 1.0.0.26)
• WSO2 AM 4.0.0 1 node
• Redis Cache server 1 node

TEST CASE (throttleSyncAsyncHybridModeEnabled = true)

• Max backend throughput : 100 TPS
• Test Duration: 15 mins
• Single thread group Invoke single API.
• Redis cache server connectivity breaks after 8 minutes.
• Enforcer Throttling configuration

[enforcer.throttling]
      enableGlobalEventPublishing = true
      jmsConnectionProviderURL = "amqp://admin:admin@carbon/carbon?brokerlist='tcp://wso2apim:5672'"
      [enforcer.throttling.publisher]
        username = "admin"
        password = "$env{tm_admin_pwd}"
        [[enforcer.throttling.publisher.URLGroup]]
          receiverURLs = ["tcp://wso2apim:9611"]
          authURLs = ["ssl://wso2apim:9711"]
      [enforcer.throttling.localThrottler]
        throttleFrequency = 1000
        throttlingPoolSize = 1000
        throttleSyncAsyncHybridModeEnabled = true
        [enforcer.throttling.localThrottler.distributedCounter]
         enabled = true
         redisHost = "redis-service"
         redisPort = 6379
         redisUser = "default"
         redisPass = "userpass"
         gatewayID = "$env{GATEWAY_NODE_ID}"
         minGatewayCount = 1

Version

1.0.0

Environment Details (with versions)

No response

Relevant Log Output

No response

Related Issues

No response

Suggested Labels

No response