feat(tools): Elicitation pass-through implementation with session routing and observability

[bogdanmariusc10]

🔗 Related Issue

Closes #1985

---

📝 Summary

This PR implements complete MCP elicitation pass-through functionality per the MCP 2025-06-18 specification, enabling MCP servers to request structured user input during tool execution with proper session routing in multi-user deployments.

Key capabilities:

• MCP servers can issue elicitation/create requests during tool execution
• Gateway forwards requests to the correct originating client session
• Client responses are returned upstream to the server
• Session correlation works correctly in multi-user environments
• Full observability with structured logging and Prometheus metrics

---

🏷️ Type of Change

• Feature / Enhancement
• Bug fix
• Documentation
• Refactor
• Chore (deps, CI, tooling)
• Other (describe below)

---

🧪 Verification

Check	Command	Status
Lint suite	make lint	✅ Pass (1 pre-existing issue in observability.py)
Unit tests	make test	✅ 17,030 passed, 704 skipped, 2 xfailed
Coverage ≥ 80%	make coverage	✅ (13 new integration tests added)

Test Results:

• All 13 elicitation integration tests passing
• Covers all 4 user stories (US-1 through US-4)
• Multi-user routing, timeout handling, capability enforcement verified

---

✅ Checklist

• Code formatted (make black isort pre-commit)
• Tests added/updated for changes
• Documentation updated (if applicable)
• No secrets or credentials committed

---

📓 Implementation Details

Architecture

sequenceDiagram
    participant Client as MCP Client (User A)
    participant Gateway
    participant Registry as ToolCallRegistry
    participant Server as MCP Server

    Client->>Gateway: tools/call "delete_files"
    Gateway->>Registry: Store mapping: call-456 -> sess-123
    Gateway->>Server: Execute tool
    
    Server->>Gateway: elicitation/create
    Gateway->>Registry: Lookup session for call-456
    Gateway->>Client: Forward to sess-123 only
    
    Client->>Gateway: Elicitation response (action: "accept")
    Gateway->>Server: ElicitResult
    Server->>Gateway: Tool result
    Gateway->>Client: Tool response

Loading

New Components

1. ToolCallRegistry (mcpgateway/cache/tool_call_registry.py)

   • Maps tool_call_id → downstream_session_id for session correlation
   • Thread-safe concurrent access with automatic cleanup
   • Lifecycle managed in application startup/shutdown

2. Elicitation Callback (mcpgateway/services/tool_service.py)

   • Custom callback in _create_elicitation_callback_for_tool_invocation()
   • Integrated in both SSE and Streamable HTTP transport paths
   • Uses ElicitationService to forward requests to correct client

3. Capability Enforcement

   • Checks session.capabilities.elicitation before forwarding
   • Returns -32601 error if client lacks capability
   • Clear error messages for debugging

4. Structured Logging

   • 5 lifecycle events: created, delivered, completed, timeout, error
   • Full context: request_id, sessions, message, duration

5. Prometheus Metrics

   • elicitation_requests_total - Counter
   • elicitation_completed_total - Counter (by action)
   • elicitation_timeout_total - Counter
   • elicitation_duration_seconds - Histogram

Modified Files

File	Changes	Lines
mcpgateway/cache/tool_call_registry.py	NEW - Session mapping registry	+183
tests/integration/test_elicitation_passthrough.py	NEW - 13 integration tests	+595
tests/integration/test_elicitation_callback_coverage.py	NEW - 8 coverage tests	+268
tests/unit/test_tool_call_registry_coverage.py	NEW - 13 unit tests	+227
mcpgateway/services/tool_service.py	Elicitation callback integration + linting fixes	+152
mcpgateway/services/elicitation_service.py	Structured logging	+3
mcpgateway/services/metrics.py	Prometheus metrics + import fix	+5
mcpgateway/transports/streamablehttp_transport.py	Elicitation support functions	+65
mcpgateway/main.py	Registry lifecycle management	+18

Configuration

Existing settings in mcpgateway/config.py (no new env vars required):

ELICITATION_TIMEOUT=60              # Default timeout in seconds
ELICITATION_MAX_CONCURRENT=100      # Max concurrent elicitations
ELICITATION_CLEANUP_INTERVAL=300    # Cleanup interval in seconds

Test Coverage

13 comprehensive integration tests covering:

• ✅ US-1: MCP Server - Request User Input During Tool Execution
• ✅ US-2: Gateway - Route Elicitation to Correct Session
• ✅ US-3: Client - Advertise Elicitation Capability
• ✅ US-4: Operator - Monitor Elicitation Events

Test scenarios:

• End-to-end elicitation flow with user confirmation
• Multi-user session routing (correct isolation)
• Timeout handling
• Capability enforcement (reject if not advertised)
• Callback creation with valid/invalid sessions
• Schema validation (primitive types only)
• Concurrent elicitation handling
• Metrics emission verification

Design Decisions

1. Session Mapping Strategy: Used dedicated ToolCallRegistry instead of extending SessionRegistry to maintain separation of concerns and avoid circular dependencies.

2. Callback Integration: Implemented callback at tool invocation time (not transport layer) to ensure proper context and correlation regardless of transport type.

3. Fail-Closed Security: If session mapping is missing or capability check fails, return clear error to server rather than attempting broadcast or fallback.

4. Structured Logging: Emit events at key lifecycle points (created, delivered, completed, timeout, error) with full context for debugging and audit trails.

5. Metrics Granularity: Track both success/failure counts and latency distribution to enable SLO monitoring and performance analysis.

---

📚 References

• MCP Elicitation Spec (2025-06-18)
• MCP Notifications Spec
• Original feature specification: Issue [FEATURE]: Elicitation pass-through and logging #1985

[marekdano]

@bogdanmariusc10 - thanks for the contribution!
Can you please resolve conflicts, fix lint and test failures?