RFC: Server Architecture Refactor for Horizontal Scalability and Bottleneck Elimination

[zoedsoupe]

RFC: Server Architecture Refactor for Horizontal Scalability and Bottleneck Elimination

TL;DR

This RFC proposes eliminating the single Base server bottleneck by introducing a session-centric architecture with pluggable registry support for both local and distributed deployments.

Problem Statement

Current architecture has several critical bottlenecks identified in production:

• Single Process Bottleneck (Why are all server requests sent to a single process for processing? cloudwalk/hermes-mcp#179): All MCP requests serialize through Base GenServer
• Session Management Overhead ([Server] MCP Sessions in clustered Distributed Erlang cloudwalk/hermes-mcp#141): Centralized session tracking with O(n) complexity
• Clustering Challenges: Sessions don't distribute properly across Erlang clusters
• Blocking Requests: Long-running tools block other requests

Proposed Solution

Session-Centric Architecture

Replace the heavy Base server with:

1. Lightweight Router: Stateless request routing
2. Session-Based Processing: Sessions handle their own requests directly
3. Pluggable Registry: Support ETS for default/local implementation and extensibility for distribution (like horde) or even external storages for sessions (like redis)
4. Async Request Processing: Eliminate blocking operations

Key Benefits

• Eliminate bottlenecks: No more single-process serialization
• Horizontal scaling: Sessions distribute across cluster nodes
• Backward compatibility: Zero or minimal number of breaking changes to public API

Architecture Diagrams

Current vs Proposed Request Flow

Current (Bottleneck):

Client → Transport (bottleneck) → Base Server (bottleneck) → Session → Handler
           ↑ All requests routed here    ↑ All requests serialized here

Proposed (Distributed):

Client → Transport → Router → Session → Handler (async)
                        ↑ Lightweight routing only
                                ↑ Direct processing per session

Community Input Needed

1. Architecture feedback: Does the session-centric approach make sense?
2. Registry design: ETS vs Horde trade-offs and alternatives?
3. Migration concerns: Any deployment scenarios we should consider?
4. Performance priorities: What metrics matter most to your use case?
5. Distribution needs: How do you currently handle clustering/scaling?

Let's discuss! 🚀 so for that im asking for the original issues authors: @byu @feng19

[zoedsoupe]

RFC: Anubis MCP Server Architecture Refactor

Status: Draft
Author: @zoedsoupe
Created: 2025-08-21
Type: Technical Architecture

Abstract

This RFC proposes a comprehensive refactor of the Anubis MCP server architecture to eliminate single-process bottlenecks while maintaining backward compatibility and adhering to Elixir/OTP best practices. Drawing inspiration from official Go and TypeScript SDK patterns, the proposed architecture introduces dual-mode operation (stateful/stateless), separates transport sessions (internal) from application state (user-facing Frame), and implements async request processing suitable for both simple deployments and distributed Erlang clusters.

Background

Community Feedback Analysis

Based on Elixir Forum discussion and GitHub issues:

• Users report timeouts in clustered environments (3+ nodes)
• Performance degradation with concurrent long-running tools
• Need for better session affinity and distribution patterns
• Request for LiveView-like developer experience
• Desire for simpler deployment options alongside cluster support

Official SDK Architecture Analysis

Analysis of Go and TypeScript official MCP SDKs reveals cleaner patterns:

1. Three Distinct Session Concepts:

   • Transport Sessions: Internal HTTP/SSE connection management
   • Protocol State: MCP initialization and lifecycle tracking
   • Application State: User-facing state container (equivalent to our Frame)

2. Dual-Mode Operation:

   • Stateless Mode: Each request processed as pure function with explicit context
   • Stateful Mode: Optional persistent state between requests when needed

3. Clear Separation of Concerns:

   • Transport layer handles connections transparently
   • Protocol layer manages MCP specification compliance
   • Application layer provides user-facing abstractions

4. Performance Characteristics:

   • Zero overhead for simple stateless tools
   • Optional complexity for advanced stateful workflows
   • No single-process bottlenecks in request processing

Goals

Primary Objectives

1. Eliminate Single-Process Bottlenecks: Remove Base server as serialization point
2. Improve Horizontal Scalability: Support both single-node and cluster deployments
3. Maintain Backward Compatibility: Preserve existing public API
4. Enhance Performance: Reduce latency and improve concurrent request handling
5. Simplify Architecture: Reduce complexity while adding distribution capabilities

Non-Goals

• Breaking changes to component registration API
• Fundamental changes to MCP protocol implementation
• Performance optimization for non-bottleneck scenarios

Detailed Design

Architecture Overview

The proposed architecture introduces a dual-mode request processing model inspired by official SDK patterns where:

1. Transport Sessions (internal) - Handle HTTP/SSE connection management transparently
2. Protocol State (framework) - Track MCP initialization and lifecycle
3. Application State (Frame) - Remain user-facing like LiveView socket assigns
4. Dual-Mode Operation - Support both stateful (session-aware) and stateless (functional) request processing
5. Async Request Processing - Eliminate single-process bottlenecks

Core Components

1. Dual-Mode Request Processing

Plain Function Processing (Stateless Mode):

defmodule Anubis.Server.RequestProcessor do
  @moduledoc """
  Stateless request processing inspired by official SDKs.
  Each request is processed as a pure function with explicit context.
  """
  
  def process_request(method, params, %Frame{} = frame, server_module) do
    # Create request context with all necessary state
    request_context = %{
      id: generate_request_id(),
      method: method,
      params: params || %{}
    }
    
    frame = Frame.put_request(frame, request_context)
    
    # Call server handler as pure function
    server_module.handle_request(
      %{"method" => method, "params" => params, "id" => request_context.id},
      frame
    )
  end
end

GenServer Session Processing (Stateful Mode):

defmodule Anubis.Server.Session do
  use GenServer
  
  # Optional: For servers that need persistent state between requests
  def handle_call({:mcp_request, method, params, context}, from, state) do
    Task.Supervisor.async_nolink(state.task_supervisor, fn ->
      result = RequestProcessor.process_request(method, params, state.frame, state.server_module)
      GenServer.reply(from, result)
    end)
    {:noreply, state}
  end
end

Benefits:

• Stateless Mode: Zero overhead for simple tools, maximum performance
• Stateful Mode: Session persistence for complex workflows
• Gradual Migration: Existing servers work unchanged
• Performance Flexibility: Choose the right pattern for each use case

2. Transport Layer Simplification

Transport Session Management (Internal):

defmodule Anubis.Server.Transport.StreamableHTTP do
  @moduledoc """
  Transport sessions are completely internal to the MCP protocol layer.
  They handle HTTP/SSE connection lifecycle transparently.
  """
  
  def handle_message(transport, session_id, message, context, opts \\ []) do
    # Transport sessions are internal - just track HTTP/SSE connections
    # Protocol state and Frame management happen at higher layers
    
    Task.Supervisor.async(transport.task_supervisor, fn ->
      # Route directly to request processor without session bottleneck
      RequestProcessor.process_request_with_context(
        message["method"],
        message["params"],
        context,
        transport.server_module
      )
    end)
  end
  
  # Transport manages HTTP connections, not application state
  defp get_or_create_transport_session(session_id) do
    # This only tracks HTTP/SSE connection state, not MCP application state
    case :ets.lookup(@transport_sessions, session_id) do
      [{^session_id, connection}] -> {:ok, connection}
      [] -> create_http_connection(session_id)
    end
  end
end

3. Frame-Centric State Management

Frame as Application State Container:

defmodule Anubis.Server.Frame do
  @moduledoc """
  Frame remains the primary user-facing abstraction like LiveView socket.
  It contains application state, assigns, and component registrations.
  """
  
  defstruct [
    assigns: %{},           # User data (like LiveView socket assigns)
    initialized: false,     # MCP protocol state
    private: %{},          # Framework internal state
    request: nil,          # Current request context
    transport: %{}         # Transport context (read-only)
  ]
  
  # Frame loading is separate from transport sessions
  def load_for_context(context, server_module) do
    # Load application state based on context
    # This can be stateless (new frame each time) or stateful (cached)
    base_frame = new()
    
    case context[:session_id] do
      nil -> 
        # Stateless mode - fresh frame
        populate_frame(base_frame, context, server_module)
      
      session_id ->
        # Stateful mode - load persistent frame state
        case get_persistent_frame(session_id) do
          {:ok, frame} -> frame
          {:error, :not_found} -> 
            populate_frame(base_frame, context, server_module)
        end
    end
  end
end

Component Registration Strategy:

defmodule Anubis.Server.ComponentRegistry do
  @moduledoc """
  Components are registered per-server module, not per-session.
  This eliminates the need for session-specific registries.
  """
  
  def get_components(server_module, frame) do
    # Static components from module
    static = server_module.__components__(:all)
    
    # Dynamic components from frame (if any)
    dynamic = Frame.get_components(frame)
    
    merge_components(static, dynamic)
  end
end

Updated Supervision Tree

Anubis.Server.Supervisor (one_for_all)
├── Task.Supervisor (RequestTaskSupervisor - for async request processing)
├── Optional: DynamicSupervisor (SessionSupervisor - only for stateful mode)
│   └── Session (optional, for persistent state between requests)
├── Frame.Cache (optional ETS table for stateful frame persistence)
└── Transport (StreamableHTTP/STDIO - handles connection lifecycle)

Key Changes:

• Eliminated Base Server: No single bottleneck process
• Optional Sessions: Only created when stateful behavior is needed
• Transport Isolation: HTTP/SSE management separated from application logic
• Simplified Hierarchy: Fewer moving parts, clearer responsibilities

Request Processing Flow

Stateless Mode (Default):

sequenceDiagram
    participant C as Client
    participant T as Transport
    participant RP as RequestProcessor
    participant F as Frame
    participant H as Handler
    participant TS as TaskSupervisor
    
    C->>T: MCP Request
    T->>TS: Task.Supervisor.async(process_request)
    TS->>RP: RequestProcessor.process_request()
    RP->>F: Frame.load_for_context(context)
    F-->>RP: fresh_frame
    RP->>H: server.handle_request(request, frame)
    H-->>RP: {:reply, result, updated_frame}
    RP-->>TS: encoded_response
    TS-->>T: response
    T-->>C: MCP Response

Loading

Stateful Mode (Optional):

sequenceDiagram
    participant C as Client
    participant T as Transport
    participant S as Session
    participant RP as RequestProcessor
    participant F as Frame
    participant H as Handler
    participant TS as TaskSupervisor
    
    C->>T: MCP Request
    T->>S: GenServer.call({:request, ...})
    S->>TS: Task.Supervisor.async(process_request)
    TS->>RP: RequestProcessor.process_request()
    RP->>F: Frame.load_persistent(session_id)
    F-->>RP: persistent_frame
    RP->>H: server.handle_request(request, frame)
    H-->>RP: {:reply, result, updated_frame}
    RP->>F: Frame.save_persistent(session_id, frame)
    RP-->>TS: encoded_response
    TS-->>S: response
    S-->>T: response
    T-->>C: MCP Response

Loading

Backward Compatibility

Public API Preservation

All existing public APIs remain unchanged:

# This continues to work exactly the same
defmodule MyServer do
  use Anubis.Server,
    name: "my-server",
    version: "1.0.0"
    
  def handle_tool("calculate", params, frame) do
    # Existing handler logic unchanged
    {:reply, %{result: params["x"] + params["y"]}, frame}
  end
end

Performance Implications

Expected Improvements

1. Reduced Latency: Eliminate Base server serialization bottleneck
2. Increased Throughput: Parallel request processing per session
3. Better Resource Utilization: Async processing prevents blocking
4. Horizontal Scalability: Sessions distribute across cluster nodes

Benchmarking Plan

1. Single-Node Performance: Compare current vs. new architecture
2. Cluster Performance: Test 3, 5, and 10 node clusters
3. Long-Running Tasks: Test with tools that take 5-120 seconds
4. Concurrent Sessions: Test 100, 1000, 10000 active sessions

Expected Metrics

• Latency Reduction: routing overhead
• Throughput Increase: concurrent requests
• Memory Efficiency: reduction in session management overhead
• Cluster Scalability: Linear scaling across nodes

Security Considerations

Session Security

1. Session Isolation: Process boundaries prevent cross-session data leakage
2. Resource Limits: Per-session limits prevent resource exhaustion
3. Timeout Enforcement: Automatic cleanup prevents session accumulation

Distribution Security

1. Cluster Authentication: Erlang distribution security
2. Session Verification: Validate session ownership across nodes
3. Network Encryption: TLS for inter-node communication

Risks and Mitigation

Technical Risks

1. Complexity Increase: Distributed systems are more complex

   • Mitigation: Maintain simple local deployment option
   • Mitigation: Comprehensive testing and documentation

2. Registry Performance: ETS/Horde performance characteristics

   • Mitigation: Benchmarking and tuning during implementation
   • Mitigation: Fallback to simpler registry implementations

3. Session Migration: Handling node failures in clusters

   • Mitigation: Implement session state persistence options
   • Mitigation: Graceful degradation on registry failures

Migration Risks

1. Breaking Changes: Inadvertent API changes

   • Mitigation: Comprehensive test suite
   • Mitigation: Community testing period

2. Performance Regression: New architecture slower in some scenarios

   • Mitigation: Extensive benchmarking
   • Mitigation: Rollback plan

Testing Strategy

Unit Testing

• Registry adapter implementations
• Session request handling
• Router functionality
• Transport layer integration

Integration Testing

• Full MCP protocol compliance
• Multi-transport scenarios
• Session lifecycle management
• Component registration and execution

Performance Testing

• Load testing with various session counts
• Long-running tool execution scenarios
• Memory usage under sustained load
• Cluster failover and recovery

Compatibility Testing

• Existing server implementations
• Different MCP client implementations
• Various deployment configurations

Future Considerations

Post-Implementation Enhancements

1. Advanced Distribution: Support for partitioned deployments
2. Observability: Enhanced telemetry and monitoring
3. Caching Layer: Distributed caching for component definitions
4. Session Persistence: Optional session state persistence

Long-term Evolution

1. Protocol Evolution: Support for future MCP specification versions
2. Performance Optimization: Further latency and throughput improvements
3. Developer Experience: Enhanced debugging and development tools
4. Ecosystem Integration: Better integration with Phoenix, LiveView, etc.

Conclusion

This RFC proposes a comprehensive architecture refactor that addresses the current bottlenecks in Anubis MCP server while maintaining backward compatibility and providing a clear path for horizontal scaling. The session-centric approach aligns with Elixir/OTP best practices and provides natural scaling boundaries.

The implementation strategy is designed to minimize risk through incremental delivery, extensive testing, and community involvement. The proposed architecture should provide significant performance improvements for concurrent workloads while maintaining the excellent developer experience that Anubis currently provides.

By addressing the fundamental architectural limitations identified in community feedback, this refactor positions Anubis MCP as a scalable, production-ready solution suitable for both simple deployments and large-scale distributed systems.

[zoedsoupe]

Key Questions to Refine the Approach:

1. Frame State Loading Strategy

# For the concurrent Task-based dispatcher:
def process_request(request, session_id, context, server_module) do
  frame = load_frame_state(session_id, server_module)  # <-- How do we handle race conditions?
  # Multiple concurrent requests for same session_id
  # Could lead to frame state conflicts
end

Doubt: How should we handle concurrent requests that both want to modify Frame state for the same session? Do we need:

• Optimistic locking on Frame updates?
• Copy-on-write Frame versioning?
• Or accept that Frame updates might be lost/overwritten?

2. Component Registration Strategy

# Current Frame has runtime component registration:
frame = Frame.register_tool(frame, "my_tool", opts)

Question: In a stateless dispatcher model, should component registration:

• Stay in Frame (loaded/saved per request) - could be expensive
• Move to compile-time only via module attributes - less flexible
• Use separate component registry (ETS) - more complexity
• Be cached per server module - hybrid approach

3. Transport Session → Application State Mapping

The MCP spec says servers MAY assign session IDs. In the new architecture:

Proposal:

# Transport level (internal)
StreamableHTTP assigns "mcp_session_123"

# Application level (user-facing)
Frame state keyed by same ID: load_frame_state("mcp_session_123")

Question: Is this 1:1 mapping always correct, or should application state have its own identity separate from transport sessions?

4. "Stateless Mode" Implementation

Doubt: For truly stateless servers, we'd create Frame fresh per request:

def process_request(request, _session_id, context, server_module) do
  frame = Frame.new() |> populate_from_context(context)
  # No state persistence
end

But this breaks component registration and user assigns between requests. How do we handle:

• Server needs to "remember" registered tools between requests
• User wants persistent state (like authentication context)

Should "stateless mode" actually be "session-scoped state" instead?

[byu]

Got the notification of this issue. I will read the RFC and consider it with more care when I free up soon.

[feng19]

This proposal should greatly improve the work I am currently doing.
I have no objection to this and hope the proposal can be completed soon.

[byu]

Just a quick touch-base status. I read the docs. I agree with the direction of the problem statement and architectural approach. I do want to think more about the open questions, so i'll ponder and problem come back soon.

[zoedsoupe]

I'm trying to focus on minor issues now and documentation fixes too, so we can proceed to an all agreed architecture proposal that answers any open questions

[mrluc]

This is super interesting stuff.

I'll try to understand / contribute some rubber-duckying.

1. there's necessarily a long-lived connection: the streaming http connection.
2. but MCP isn't bidirectional on that long-lived connection -- it ain't websockets! Streaming http down, but you POST to an endpoint to take action. That's the source of pain (since MCP is in flux and is also from non-elixir ecosystems, which didn't want to mandate websockets); it would be great if we could reuse that same lovely auth'd open connection for everything, perfect place to keep state -- but we can't.

So, can we phrase the problem like this?

'we need the POST'd messages to be able to wind their way to the process that backs the open connection, which has the necessary connection state, and return a response back out, in a way that doesn't depend on what node the user POSTs to'

In that case, I'd incline towards adding a dependency on phoenix_pubsub if there isn't one, and implementing a Phoenix.Tracker for Hermes/Anubis. Most of Phoenix.PubSub is like critical infra for the community at this point IMO 😅 so I don't think anyone would fault you-all for adding it (might be there already, haven't checked!)

(Also feels like an opportune moment to ask why anubis and hermes are like mirror images at a first glance 😅 I keep bouncing between repos when looking at issues -- everyone but me knows what's up there, I'm sure. Is it a 'bleeding edge/stable', 'upstream/downstream' distinction? -- update for future searchers: commits seem a good deal newer in Anubis as of this writing)

[mrluc]

(forgot to close the loop, my bad, cont'd)...

and the POSTed-to process can use the tracker (ideally configured to be a bit twitchier than defaults) to look up the pid currently registered to the posted session id, maybe with a quick retry if needed, and send a message to that pid; that process would then be able to handle the call with the necessary state and return.

The long-lived connection process could heartbeat to the tracker every so often, but that could possibly be on relatively long timescales like 30s, since probably there won't be contention for keys since the keys would be session ids -- the heartbeating would be so that, if the streaming http connection is broken in a way that doesn't let a message get to Tracker to remove the pid, the old session id would still stay in Tracker until a cleanup is done, but the immediate reconnect and a tool call following quickly on it would all be to a new session id.

obvs this is similar to other options y'all are considering, using Horde, etc. IMO Phoenix.Tracker or Horde would be fine; it's an attractively small amount of data so it may not matter what crdt is used: just 'your_id => pid, last_seen' mappings that get thrown away if the connection times out, so no long-term/complex state to get out of sync, no storage size issues, etc -- and phoenix_pubsub is super widely used as a dependency / decently stable API over a long time now

Re: workaround until this lands: it feels like people don't need to run into this issue unless they're using multiple app nodes in one geographic region. Even with distributed clusters, unless you have a certain scale, routing and cost can combine to make it a non-issue, because you can do a lot with a single beefier elixir node per geographic region, and adding a node in a new region brings immediate latency benefits to users from that area vs. adding in the same region -- so if my app has 6 nodes in 6 regions I think I'd almost never run into the issues that this addresses; my requests will always go to the nearest node, for streaming and posting.

[zoedsoupe]

Also feels like an opportune moment to ask why anubis and hermes are like mirror images at a first glance 😅 I keep bouncing between repos when looking at issues -- everyone but me knows what's up there, I'm sure. Is it a 'bleeding edge/stable', 'upstream/downstream' distinction? -- update for future searchers: commits seem a good deal newer in Anubis as of this writing

thanks for the question! actually, hermes-mcp was the original source while I was working for Cloudwalk. since July I'm not working there anymore and they asked me to pass the hex package ownership for them, as the repository (although public) is already on their ownership

as I can't guarantee they will still have interest on maintaining the SDK, considering that it was born to tackle an internal PoC and product and doesn't have the same priority anymore, I've decided to fork it and follow from where I've stopped, while trying to keep tracking of the open issues/PRs on the "old repo"

I've also took a "rebrand" to avoid any ownership future problema and relicensed the software under LGPL 3.0

I've explained that on the elixirforum thread about the library

[zoedsoupe]

@mrluc, thanks for the deep dive — your framing on the transport asymmetry is spot-on and helps clarify what we're actually solving here.

On the routing problem

You nailed it with:

'we need the POST'd messages to be able to wind their way to the process that backs the open connection, which has the necessary connection state'

But here's the interesting bit: the MCP spec doesn't actually require servers to maintain sessions at all. The Mcp-Session-Id header is optional (servers MAY assign them), which means we need to support both:

• Stateless mode: Fresh Frame per request, no session tracking → no routing problem
• Stateful mode: Persistent state between requests → needs the routing solution you described

So the architecture needs to be flexible enough to handle both patterns, which is why the RFC proposes dual-mode operation.

Phoenix.Tracker vs Horde

I really like the Phoenix.Tracker suggestion. It's battle-tested infrastructure and the heartbeat mechanism fits naturally. The main trade-off I see is that Tracker is optimized for presence, not process registry — we'd need to layer call routing on top of the presence data. Horde gives us direct Registry.lookup/2 → pid resolution, which is cleaner for this use case.

My instinct is to make the registry pluggable:

# Behavior with adapters:
# - Local (ETS, default)
# - Horde (distributed registry)
# - PhoenixTracker (presence-based)

Let users pick based on their stack. Most folks probably don't need distribution anyway, so local ETS should be the default with easy opt-in for clustering.

On your geographic distribution insight

You're absolutely right that this mostly matters for same-region clusters, not global distribution. The actual pain from #179 and #141 is "concurrent requests serializing through Base server" not "requests hitting wrong geographic node." Solving concurrent processing first makes geographic distribution a natural extension later.

Questions I'm wrestling with

1. Heartbeat frequency: You mentioned ~30s. Given MCP connections can be idle between tool calls, should we be more aggressive (15s) or more lazy (60s+)?

2. Session reconnection: MCP spec allows clients to reconnect with Last-Event-ID. Should we persist Frame state for this, or is losing session state on reconnect acceptable?

3. Component registration: In stateless mode, where do runtime-registered components live? Frame (expensive), compile-time only (less flexible), separate cache (more complexity)?

Thoughts on any of these? And would you be interested in prototyping the Phoenix.Tracker adapter to see how it performs in practice?