Introduce OpenAPI Conformance Testing

[strawgate]

Enhancement

There are repositories of openapi specs in the world and we should add an e2e test which runs our schema compiler on them. We should start by identifying timeouts, memory leaks, crashes and eventually work towards correctness.

[marvin-context-protocol]

Claude finished @strawgate's task in 4m 54s —— View job

---

Triage: Introduce OpenAPI Conformance Testing

• Gather repository context and understand schema compiler
• Identify existing OpenAPI test coverage
• Research public OpenAPI spec repositories
• Investigate how other projects do conformance testing
• Formulate recommendation and detailed action plan

Recommendation

Add a new @pytest.mark.integration test module at tests/server/providers/openapi/test_openapi_conformance.py that downloads real-world OpenAPI specs and runs the full parse_openapi_to_http_routes → FastMCP.from_openapi pipeline, asserting no crashes, timeouts, or unhandled exceptions. The infrastructure already exists: there's a @pytest.mark.integration marker, a CI job that runs integration tests with a 30s timeout per test, and an existing pattern in test_openapi_performance.py that downloads GitHub's API spec at runtime.

The best spec source is a curated list of raw GitHub URLs for well-known APIs, not a dynamic fetch from the APIs.guru catalog (which adds flakiness and includes many Swagger 2.0 specs FastMCP doesn't support). Start with ~15-20 specs covering both OpenAPI 3.0 and 3.1, varying sizes, and diverse schema features.

Findings

Existing schema compiler pipeline: The entry point is parse_openapi_to_http_routes() in src/fastmcp/utilities/openapi/parser.py, which handles both OpenAPI 3.0 and 3.1 via openapi-pydantic. It feeds into _combine_schemas_and_map_params() and convert_openapi_schema_to_json_schema(). The higher-level FastMCP.from_openapi() wraps this in an OpenAPIProvider.

Existing test coverage:

• 26 test files across tests/utilities/openapi/ and tests/server/providers/openapi/ covering specific schema features (circular refs, allOf, nullable fields, transitive refs, etc.)
• test_openapi_performance.py already downloads GitHub's full API spec (~10MB) as an integration test — this is the closest precedent
• All tests use synthetic/handcrafted specs; no broad real-world coverage exists

Test infrastructure:

• @pytest.mark.integration with 30s timeout in CI, run in a separate job
• @pytest.mark.conformance exists for MCP protocol conformance (Node.js-based), separate from what's proposed here
• pytest-xdist available for parallelism; pytest-timeout for per-test timeouts

Spec sources evaluated:

• APIs-guru/openapi-directory — 700+ providers, but many are Swagger 2.0 (not supported by FastMCP). Has an API but adds CI flakiness
• Individual repos: github/rest-api-description, stripe/openapi, openai/openai-openapi, digitalocean/openapi — stable raw URLs, well-maintained
• APIs-guru mirrors of these specs also work via raw GitHub URLs

Detailed Action Plan

Phase 1: Core conformance test (this issue)

1. Create tests/server/providers/openapi/test_openapi_conformance.py

Structure:

"""OpenAPI conformance tests against real-world API specifications.

These tests download public OpenAPI specs and run the full parsing pipeline
to detect crashes, timeouts, and unhandled exceptions.

Mark: pytest -m integration
"""

import httpx
import pytest
from fastmcp import FastMCP

# Curated list: (name, url, openapi_version, expected_min_tools)
OPENAPI_SPECS = [
    # Large 3.0 specs
    ("github-ghes-3.17", "https://raw.githubusercontent.com/github/rest-api-description/.../ghes-3.17.json", "3.0", 500),
    ("digitalocean", "https://raw.githubusercontent.com/digitalocean/openapi/main/specification/DigitalOcean-public.v2.yaml", "3.0", 100),
    # Large 3.1 specs
    ("github-api", "https://raw.githubusercontent.com/github/rest-api-description/.../api.github.com.json", "3.1", 500),
    # Medium specs
    ("openai", "https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml", "3.0", 20),
    ("petstore-3.0", "https://raw.githubusercontent.com/.../petstore.yaml", "3.0", 3),
    ("petstore-3.1", "https://raw.githubusercontent.com/.../petstore.yaml", "3.1", 3),
    # ... more specs covering diverse features
]

@pytest.mark.integration
@pytest.mark.timeout(30)
@pytest.mark.parametrize("name,url,version,min_tools", OPENAPI_SPECS, ids=[s[0] for s in OPENAPI_SPECS])
async def test_openapi_spec_parses_without_error(name, url, version, min_tools):
    """Real-world OpenAPI spec parses without crashing."""
    response = httpx.get(url, timeout=15.0, follow_redirects=True)
    response.raise_for_status()
    
    if url.endswith((".yaml", ".yml")):
        import yaml
        spec = yaml.safe_load(response.text)
    else:
        spec = response.json()
    
    server = FastMCP.from_openapi(spec, httpx.AsyncClient())
    tools = await server.list_tools()
    assert len(tools) >= min_tools, f"{name}: expected >= {min_tools} tools, got {len(tools)}"

2. Curate the spec list — Start with these categories:

Category	Examples	Why
Large 3.0	GitHub GHES, DigitalOcean	Stress test: 500+ endpoints, deep $ref chains
Large 3.1	GitHub api.github.com	3.1-specific features (webhooks, JSON Schema 2020-12)
Complex schemas	Stripe	Deeply nested allOf/oneOf/anyOf, discriminators
Medium	OpenAI, Twilio	Real-world but manageable size
Small canonical	OAI Petstore (3.0 + 3.1)	Baseline sanity check
Edge cases	Kubernetes, Azure	Circular refs, huge component sections

3. Add to CI — No workflow changes needed. The test uses @pytest.mark.integration, which is already run in a separate CI job with 30s per-test timeout.

Phase 2: Expand coverage (follow-up)

• Parametrize failure modes: Add assertions for memory (via tracemalloc), check that no warnings are logged for known-supported features
• Correctness checks: For a subset of specs (Petstore, etc.), assert specific tool names, parameter schemas, and request body shapes
• APIs-guru bulk sweep: A separate, optional test (perhaps @pytest.mark.slow) that downloads the full APIs-guru catalog and runs parse on all 3.0/3.1 specs, collecting failures into a report rather than failing the suite

Phase 3: Regression tracking (future)

• Track parsing time per spec across commits (emit print() or structured output for CI artifacts)
• Add xfail markers for known-broken specs and track progress toward fixing them

Related Issues and Pull Requests

Repository	Issue or PR	Relevance
PrefectHQ/fastmcp	tests/server/providers/openapi/test_openapi_performance.py (existing file)	Already downloads GitHub's API spec as an integration test — direct precedent for this work
PrefectHQ/fastmcp	tests/conformance/test_conformance.py (existing file)	MCP protocol conformance tests — different scope but similar naming; new tests should use integration marker, not conformance

Related Files

Repository	File	Relevance	Key sections
PrefectHQ/fastmcp	src/fastmcp/utilities/openapi/parser.py	Core parsing pipeline — parse_openapi_to_http_routes() is the function under test	L55-107 (entry point), L663-818 (parse loop)
PrefectHQ/fastmcp	src/fastmcp/utilities/openapi/schemas.py	Schema combination and ref rewriting — crash-prone for complex specs	L75-142 (_replace_ref_with_defs), L223-458 (_combine_schemas_and_map_params)
PrefectHQ/fastmcp	src/fastmcp/utilities/openapi/json_schema_converter.py	OpenAPI→JSON Schema conversion — handles nullable, oneOf/anyOf	Full file
PrefectHQ/fastmcp	src/fastmcp/server/providers/openapi/provider.py	OpenAPIProvider — wraps parser output into MCP components	L1-50 (imports and entry)
PrefectHQ/fastmcp	tests/server/providers/openapi/test_openapi_performance.py	Existing pattern for downloading + parsing real specs	L22-67 (GitHub API perf test)
PrefectHQ/fastmcp	.github/actions/run-pytest/action.yml	CI test runner — integration tests already have 30s timeout	Full file

Related Webpages

Name	URL	Relevance
APIs-guru OpenAPI Directory	https://github.com/APIs-guru/openapi-directory	700+ API providers with OpenAPI specs — potential bulk source for Phase 2
GitHub REST API Description	https://github.com/github/rest-api-description	Large, well-maintained 3.0 and 3.1 specs — already used in perf test
Stripe OpenAPI	https://github.com/stripe/openapi	Complex schemas with deep allOf/anyOf/discriminator usage
OpenAI OpenAPI	https://github.com/openai/openai-openapi	Medium-size 3.0 spec with diverse features
DigitalOcean OpenAPI	https://github.com/digitalocean/openapi	Large 3.0 spec
APIs.guru REST API	https://apis.guru/api-doc	Programmatic access to the full spec catalog

---

Marvin Context Protocol | Type /marvin to interact further

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.