[FEATURE]: Elicitation pass-through and logging

[crivetimihai]

📨 Feature: Elicitation Pass-Through and Logging

Goal

Complete elicitation pass-through so MCP servers can issue elicitation/create during tool execution, and the gateway forwards those requests to the originating client session, then returns the client response upstream. Add structured logging and metrics for elicitation lifecycle events.

Why Now?

1. MCP Compliance: The 2025-06-18 spec defines elicitation as a core capability for interactive tool workflows
2. Agentic Workflows: AI agents need to request user confirmation or additional input during tool execution
3. Session Correlation: Current implementation may route elicitation to wrong client in multi-user deployments
4. Observability Gap: No structured logging or metrics for elicitation events makes debugging difficult

---

📖 User Stories

US-1: MCP Server - Request User Input During Tool Execution

As an MCP Server developer
I want to send elicitation/create requests during tool execution
So that I can request user confirmation or additional input before proceeding

Acceptance Criteria:

Scenario: Server requests user confirmation via elicitation
  Given an MCP server executing tool "delete_files"
  And the tool needs user confirmation before proceeding
  When the server sends elicitation/create with message "Delete 50 files?"
  Then the gateway should forward the request to the originating client
  And the client should display the confirmation dialog
  And the client response should be returned to the server
  And the tool execution should continue based on user response

Scenario: Elicitation reaches correct client in multi-user deployment
  Given user A is executing tool "sensitive_action" via session S1
  And user B has an active session S2
  When the server sends elicitation/create
  Then the request should be routed ONLY to session S1
  And session S2 should NOT receive the elicitation request

Scenario: Elicitation timeout returns error to server
  Given an elicitation request is pending
  And the client does not respond within timeout period
  When the timeout expires
  Then the server should receive an error response
  And the error should indicate "Elicitation timed out"

Technical Requirements:

• Add elicitation_callback when creating ClientSession in tool_service.py
• Map tool invocation to downstream session ID for routing
• Use RequestContext.meta.related_request_id for correlation

US-2: Gateway - Route Elicitation to Correct Session

As a Platform Administrator
I want elicitations to be routed to the correct client session
So that the right user receives the request in multi-user deployments

Acceptance Criteria:

Scenario: Track session mapping during tool invocation
  Given a tool invocation from client session "sess-123"
  When the upstream MCP server sends elicitation/create
  Then the gateway should lookup the originating session "sess-123"
  And forward the elicitation to that specific session
  And NOT broadcast to all elicitation-capable sessions

Scenario: Handle missing session gracefully
  Given an elicitation request with unknown session mapping
  When the gateway attempts to route the request
  Then a clear error should be returned to the server
  And the error should indicate "No client session available"
  And the error should be logged with context

Technical Requirements:

• Create per-request mapping: tool_call_id -> downstream_session_id
• Store in request context or in-flight registry
• Return -32601 error if no session available

US-3: Client - Advertise Elicitation Capability

As a Client developer
I want my client to advertise elicitation capability
So that servers know they can request user input

Acceptance Criteria:

Scenario: Client without elicitation capability
  Given a client that did not advertise capabilities.elicitation
  When an MCP server sends elicitation/create via gateway
  Then the gateway should return error to server immediately
  And the error code should be -32601 (method not found)
  And no forwarding should be attempted

Scenario: Client with elicitation capability
  Given a client that advertised capabilities.elicitation
  When the client connects and initializes
  Then the session_registry should record elicitation=true
  And elicitation requests should be routable to this client

Technical Requirements:

• Check session.capabilities.elicitation before forwarding
• Return clear error if capability not advertised

US-4: Operator - Monitor Elicitation Events

As a Platform Operator
I want structured logging and metrics for elicitation events
So that I can monitor and debug elicitation workflows

Acceptance Criteria:

Scenario: Log elicitation lifecycle events
  Given structured logging is enabled
  When an elicitation request is created
  Then a log entry should contain:
    | Field | Value |
    | event | elicitation.created |
    | request_id | {request_id} |
    | upstream_session | {server_session} |
    | downstream_session | {client_session} |
    | message | {elicitation_message} |

  When the client responds
  Then a log entry should contain:
    | Field | Value |
    | event | elicitation.completed |
    | action | accept/decline/cancel |
    | duration_ms | {elapsed_time} |

Scenario: Metrics for elicitation events
  Given Prometheus metrics are enabled
  Then the following metrics should be exposed:
    | Metric | Type |
    | elicitation_requests_total | Counter |
    | elicitation_completed_total | Counter (by action) |
    | elicitation_timeout_total | Counter |
    | elicitation_duration_seconds | Histogram |

Technical Requirements:

• Emit structured log events for created, delivered, completed, timeout, error
• Add Prometheus counters and histograms
• Include latency histogram for response time

---

🏗 Architecture

Elicitation Flow with Session Mapping

sequenceDiagram
    participant Client as MCP Client (User A)
    participant Gateway
    participant Registry as SessionRegistry
    participant Server as MCP Server

    Client->>Gateway: Initialize (capabilities.elicitation=true)
    Gateway->>Registry: Store session "sess-123" with elicitation=true
    
    Client->>Gateway: tools/call "delete_files"
    Gateway->>Gateway: Store mapping: call-456 -> sess-123
    Gateway->>Server: Execute tool
    
    Server->>Gateway: elicitation/create
    Gateway->>Gateway: Lookup session for call-456
    Gateway->>Registry: Get session "sess-123"
    Gateway->>Client: Forward elicitation request
    
    Client->>Gateway: Elicitation response (action: "accept")
    Gateway->>Server: ElicitResult
    Server->>Gateway: Tool result
    Gateway->>Client: Tool response

Loading

Current vs Proposed Behavior

Aspect	Current	Proposed
Upstream elicitation callback	Not set (uses default reject)	Custom callback forwarding to gateway
Session routing	First elicitation-capable session	Mapped originating session
Streamable HTTP	Not explicit	Explicit handling in transport
Logging	Basic logger.info	Structured events + metrics

---

📋 Implementation Tasks

Phase 1: Elicitation Callback in Tool Service

• Add elicitation_callback parameter when creating ClientSession in tool_service.py
• Implement callback that uses ElicitationService to forward request
• Store downstream_session_id in request context before tool invocation
• Use RequestContext.meta.related_request_id for correlation

Phase 2: Session Mapping Registry

• Create mapping storage: tool_call_id -> downstream_session_id
• Store mapping at start of tool invocation
• Clean up mapping after tool invocation completes
• Handle concurrent tool calls correctly

Phase 3: Capability Enforcement

• Check session.capabilities.elicitation before forwarding
• Return -32601 error if client lacks capability
• Add clear error message indicating capability requirement
• Log capability mismatch for debugging

Phase 4: Streamable HTTP Support

• Add explicit elicitation handling in streamablehttp_transport.py
• Ensure request/response correlation in HTTP context
• Handle bi-directional message flow

Phase 5: Structured Logging

• Define log event schema for elicitation events
• Emit elicitation.created event with request details
• Emit elicitation.delivered event when forwarded to client
• Emit elicitation.completed event with action and duration
• Emit elicitation.timeout event when timeout occurs
• Emit elicitation.error event for failures

Phase 6: Metrics

• Add elicitation_requests_total counter
• Add elicitation_completed_total counter with action label
• Add elicitation_timeout_total counter
• Add elicitation_duration_seconds histogram
• Expose in /metrics endpoint

Phase 7: Testing

• Unit tests for elicitation callback
• Unit tests for session mapping
• Integration tests for end-to-end elicitation flow
• Integration tests for multi-user routing
• Test timeout handling
• Test capability enforcement

---

⚙️ Configuration Example

# Existing settings (already in config.py)
ELICITATION_TIMEOUT=60              # Default timeout in seconds
ELICITATION_MAX_CONCURRENT=100      # Max concurrent elicitations
ELICITATION_CLEANUP_INTERVAL=300    # Cleanup interval in seconds

---

✅ Success Criteria

• Tool execution can trigger elicitation/create and receive response
• Elicitation routed to correct client session (not broadcast)
• Clients without elicitation capability receive clear error
• Multi-user deployments route correctly
• Structured logs emitted for all lifecycle events
• Prometheus metrics exposed and accurate
• Timeout handling works correctly
• All integration tests pass

---

🏁 Definition of Done

• Elicitation callback added to tool service
• Session mapping implemented
• Capability enforcement working
• Streamable HTTP support added
• Structured logging implemented
• Metrics added and exposed
• Unit tests written and passing
• Integration tests written and passing
• Code passes make verify
• PR reviewed and approved

---

📝 Additional Notes

MCP Specification References

• Elicitation Spec (2025-06-18)
• Notifications Spec

Current Implementation (Gaps)

Component	Status	Gap
ElicitationService	Exists	Works for client→server, needs server→client
session_registry	Exists	Stores capabilities, needs session mapping
tool_service.ClientSession	Exists	Missing elicitation_callback
Streamable HTTP	Exists	No explicit elicitation handling
Logging	Basic	Not structured, no metrics

---

🔗 Related Issues

• ElicitationService exists at mcpgateway/services/elicitation_service.py
• Session registry at mcpgateway/cache/session_registry.py