Research: user-service high cpu and high latency

[kenahrens]

Research why the user-service requires so much CPU and takes so long to respond even under medium load scenarios. Baseline the performance on the current version, then suggest code changes to improve performance.

[kenahrens]

TL;DR

• Staging replay shows severe degradation vs local for availability checks.
• /api/users/check-username measured ~10.01s and 100% failed in staging sample replay vs ~42ms local baseline.
• I implemented and opened a fix PR: Improve user-service availability-check latency under replay load #70.

Issue Background

Issue: #69 (Research: user-service high cpu and high latency)

Observed from replay analysis:

• local and staging diverge sharply on user availability path
• highest-risk endpoint: /api/users/check-username

Evidence Summary

Replay source in this analysis: backend/user-service/proxymock/recorded-complete.

Environment comparison (before fix)

Endpoint	Local Avg	Local P95	Staging Avg	Staging P95	Staging Failed
/api/users/check-username	41.97 ms	41.97 ms	10.01 s	10.01 s	100%
/actuator/health/readiness	1 ms	1 ms	94.92 ms	94.92 ms	0%
/actuator/health	10.99 ms	10.99 ms	281.18 ms	281.18 ms	0%
/actuator/health/liveness	47 ms	1 ms	226 ms	103.03 ms	0%

Fix Implemented

PR: #70

What changed

• Added short-lived (5s TTL) in-memory cache for:
  • usernameExists
  • emailExists
• Invalidated relevant cache entries after successful user registration.
• Reduced log overhead on hot availability endpoints (info -> debug in controller checks).
• Changed default logging posture to safer production defaults via env-configurable levels:
  • com.banking.userservice: ${APP_LOG_LEVEL:INFO}
  • org.springframework.security: ${SECURITY_LOG_LEVEL:WARN}

Validation Run Results

Local load replay before vs after fix

Command used:

• proxymock replay --test-against localhost:<port> --in proxymock/recorded-complete --times 30 --vus 5

Metric	Before (master)	After (PR #70)
/api/users/check-username avg	4.33 ms	3.69 ms
/api/users/check-username p95	6.99 ms	6.00 ms
total requests	750	750
replay failures	0%	0%

Additional check:

• make test passed in backend/user-service.

This comment supersedes earlier interim notes; this is the consolidated issue/findings/fix/PR report.