[Architecture Pattern] Bypassing Service Bus Session limits for Sync-over-Async HTTP Gateways

[ShivamSaluja]

Hi everyone!

I recently tackled a major scaling bottleneck when building an edge API Gateway for slow-running AI workloads. We needed to bridge legacy synchronous HTTP clients (expecting a fast 200 OK) to long-running asynchronous AI workers over Azure Service Bus.

The standard approach is the Sync-over-Async pattern using Service Bus Sessions to correlate the request and the reply. However, at scale, holding Session locks forces the Gateway pods to become stateful, ruining horizontal elasticity and eventually hitting the namespace session limits.

The Solution: Stateless Filtered Topics
Instead of Sessions, I built an architecture that uses a shared Reply Topic with dynamic SQL Subscriptions. The gateway injects a unique CorrelationId property, and the broker evaluates the SQL filter to push the reply down the exact pipe to the correct pod. This makes the Gateway layer 100% stateless.

To make this easy for the Spring Boot community, I abstracted the dynamic Administration Client and Receiver logic into a new open-source starter:

📦 GitHub Repo: https://github.com/ShivamSaluja/sentinel-servicebus-starter

📖 Technical Deep Dive: https://dev.to/shivamsaluja/sync-over-async-bypassing-azure-service-bus-session-limits-for-ai-workloads-269d

sequenceDiagram
    participant API as API Client
    participant GW as Gateway (Instance A)
    participant SB_Q as Request Queue
    participant W_X as Worker Node
    participant SB_T as Reply Topic

    API->>GW: POST /api/process
    Note over GW: Generates correlation_id & InstanceId
    GW->>SB_Q: Send Message (ReplyToInstance=InstanceA, CorrelationId=123)
    Note right of GW: Returns CompletableFuture
    
    SB_Q->>W_X: Consume Message
    Note over W_X: Worker logic...
    W_X->>SB_T: Send Reply (ReplyToInstance=InstanceA, CorrelationId=123)
    
    note over SB_T: Filter: ReplyToInstance='InstanceA'
    SB_T-->>GW: Push to specific Topic Subscription
    Note over GW: Completes CompletableFuture
    GW->>API: Async HTTP Response

Loading

I would love to get feedback from the Azure Java SDK maintainers and the community on this pattern. Have you run into similar session exhaustion issues at the edge?