feat(api): comprehensive Virtual Meta-Server with 12 meta-tools

[ecthelion77]

🔗 Related Issue

Closes #2230
Supersedes #3653

📝 Summary

This PR implements the Virtual Meta-Server feature — a comprehensive tool aggregation layer that enables AI agents to discover and invoke thousands of underlying tools through a unified interface. It exposes 12 meta-tools to the agent while completely hiding the complexity of the underlying tool ecosystem.

This is a production-tested alternative to PR #3653, with additional features, security hardening, and 2+ weeks of real-world testing with VS Code Copilot and Copilot Studio.

Problem Statement

Traditional MCP architectures expose all tools directly to AI agents, leading to:

• Context window exhaustion with hundreds or thousands of tools
• Poor agent decision-making due to overwhelming tool choices
• No dynamic tool discovery — agents must know tools upfront
• No OAuth integration — agents cannot authorize gateway access interactively

Solution — 12 Meta-Tools

Core (same as #3653)

• search_tools: hybrid semantic + keyword search with scope filtering
• list_tools: paginated tool listing with sorting and filtering
• describe_tool: detailed tool info with schema and metadata
• execute_tool: tool execution with JSON schema validation and routing
• get_tool_categories: aggregated categories with counts
• get_similar_tools: vector similarity search for related tools

New (beyond #3653)

• authorize_gateway: interactive OAuth authorization with token refresh
• authorize_all_gateways: one-click authorization for all OAuth gateways
• list_resources: paginated MCP resource listing
• read_resource: read MCP resource content by URI
• list_prompts: paginated MCP prompt listing
• get_prompt: prompt template retrieval with optional rendering

Key Improvements Over #3653

Feature	#3653	This PR
Meta-tools	6	12
OAuth integration	❌	✅ (user identity propagation)
Authorize interactively	❌	✅ (authorize_gateway, authorize_all)
Resources + Prompts	❌	✅ (list/read/get)
camelCase normalization	❌	✅ (MCP clients → snake_case)
Copilot Studio compat	❌	✅ (flat argument tolerance)
Observability spans	❌	✅ (prompt.render, resource.read)
JSON serialization	str() ❌	orjson.dumps() ✅
SSRF loopback validation	Removed ❌	Preserved ✅
OAuth config protection	Removed ❌	Preserved ✅
Production tested	❌	✅ (2+ weeks)

Security Fixes (vs #3653)

The original PR #3653 introduced several security regressions that this PR avoids:

1. SSRF prevention preserved: x-forwarded-internally header validation against loopback IP is maintained (the original PR removed this check)
2. OAuth secret protection preserved: protect_oauth_config_for_storage() is called on server updates (the original PR removed this)
3. Proper JSON serialization: Uses orjson.dumps() instead of Python str() which produces invalid JSON with single quotes
4. Missing import fixed: wraps from functools is properly imported (the original PR had a NameError at startup)
5. Login redirect safety: Post-login redirect uses _is_safe_local_path() validation to prevent open redirect attacks

🏷️ Type of Change

• Feature / Enhancement

Test Coverage

• ✅ 2400+ lines of tests covering all 12 meta-tools
• ✅ MetaServerService: tool definitions, meta-tool detection, scope filtering
• ✅ MetaToolService: describe_tool, execute_tool with schema validation
• ✅ All request/response schemas with Pydantic validation
• ✅ camelCase → snake_case argument normalization
• ✅ Scope filtering with AND semantics and edge cases
• ✅ Error handling for all code paths

📓 Notes

Files Changed (26 files, +6584/-23)

New Files

• mcpgateway/meta_server/__init__.py — Package initialization
• mcpgateway/meta_server/schemas.py — Pydantic models for all 12 meta-tools
• mcpgateway/meta_server/service.py — MetaServerService with all handler implementations
• mcpgateway/routers/meta_router.py — HTTP endpoints under /meta/*
• mcpgateway/services/meta_tool_service.py — Business logic for describe/execute
• mcpgateway/services/semantic_search_service.py — Semantic search stub (graceful fallback)
• mcpgateway/services/vector_search_service.py — Vector search with numpy fallback
• mcpgateway/services/embedding_service.py — Embedding stub (no-op without model)
• mcpgateway/utils/pgvector.py — pgvector compatibility shim
• mcpgateway/alembic/versions/5126ced48fd0 — Migration for meta-server fields

Modified Files

• mcpgateway/schemas.py — server_type, hide_underlying_tools, meta_config, meta_scope fields
• mcpgateway/db.py — ToolEmbedding model + meta-server fields on Server
• mcpgateway/transports/streamablehttp_transport.py — Tool hiding + meta-tool routing
• mcpgateway/middleware/rbac.py — Login redirect with ?next= preservation
• mcpgateway/routers/oauth_router.py — Chained OAuth flow + authorize-all endpoint
• mcpgateway/routers/sso.py — Post-login redirect cookie handling
• mcpgateway/services/server_service.py — Meta-server field handling
• mcpgateway/services/tool_service.py — Absolute OAuth authorize URL
• mcpgateway/admin.py — Meta-server fields in admin UI
• mcpgateway/static/admin.js — Meta-server JS config
• mcpgateway/templates/admin.html — Meta-server form fields
• mcpgateway/templates/gateways_partial.html — Authorize button for non-admin users
• mcpgateway/main.py — Router registration
• mcpgateway/config.py — semantic_search_rate_limit setting

Tests

• tests/unit/mcpgateway/test_meta_server.py — 2257 lines covering all meta-tools
• tests/unit/mcpgateway/services/test_meta_tool_service.py — 180 lines for tool service

Migration & Compatibility

• ✅ Backward compatible: existing servers default to server_type="standard"
• ✅ Additive changes: no breaking changes to existing endpoints
• ✅ Opt-in feature: meta-servers explicitly created with server_type="meta"
• ✅ Graceful degradation: semantic search falls back to keyword search

[ecthelion77]

Suggested labels: enhancement, COULD, python, api