Skip to content

Intermittent 500 Errors During Basic Auth API Invocations Due to APIKeyMgtRemoteUserStoreMgtService Failures #3806

Open
wso2/carbon-apimgt
#13064
@sajith-madhusanka

Description

@sajith-madhusanka

Description

API invocations intermittently fail with a 500 HTTP status due to failures in admin service invocations. The root cause is the auto-cleanup mechanism in the Axis2 service client. Specifically, the cleanupTransport() method, triggered during the creation of the service client, attempts to release resources from previous invocations without verifying if the previous response was fully consumed. This auto-cleanup process is noted to be unsuitable for performance-critical scenarios and may introduce threading issues.

Steps to Reproduce

  • Deployed a two-node APIM setup, where one node is configured as the Traffic Manager, and the other contains the Gateway, Key Manager, and Portals components.
  • Deployed a sample API and enabled Basic Auth as the authorization mechanism.
  • Disabled the Gateway token cache to simulate a load on the Traffic Manager node.
  • Invoked the API by executing the following script:
#!/bin/bash

# Configuration
REQUESTS_PER_SECOND=350
DURATION_SECONDS=1800  # Change this to your desired duration
URL='https://localhost:8243/pizzashack/1.0.0/menu'
AUTH_HEADER='Authorization: Basic YWRtaW46YWRtaW4='

# Run the requests
for ((i=1; i<=DURATION_SECONDS; i++)); do
  echo "Sending batch $i..."
  seq $REQUESTS_PER_SECOND | xargs -n1 -P $REQUESTS_PER_SECOND -I{} \
    curl -s -o /dev/null -w "%{http_code}\n" -k -X 'GET' "$URL" \
    -H 'accept: application/json' \
    -H "$AUTH_HEADER"

done

echo "Completed sending requests."

Version

3.2.0

Environment Details (with versions)

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions