Manually run relevant MCP conformance tests

[Azquelt]

Do the necessary setup and run the relevant MCP protocol conformance tests from https://github.com/modelcontextprotocol/conformance locally.

If there are failures when you run the tests, investigate why it's failing and raise issues to fix our implementation.

[martindrozdz]

I got a server-initiaize scenario to work (pictured):

However, when I run any of the other scenarios, they fail.
I will raise new issues to setup conformance tests for the other scenarios.

[martindrozdz]

Apropriate Server MCP tools need to be written first in order for the server side conformance test scenarios in this thread to:

• pass and
• be automated:

The recommended way of testing these scenarios is:

• create the tool
• run the FAT Test that has the tool in debug mode (create a breakpoint on the MCP server)
• run the conformance test
• check that the conformance test passes
• For more details see how I ran the server-initialize test in the comment above.

Below is a guide to the scenarios described.

Example test Category

name-of-example scenario as it appears in the conformance tests
http url showing the description of test scenario as well as any test acceptance criteria

Below are the actual scenarios to be implemented

Server

server-initialize
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/lifecycle.ts#L8

logging-set-level
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/utils.ts#L9

Completion

completion-complete (depends on completion to be implemented)
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/utils.ts#L176

Tools

tools-list
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/tools.ts#L15

tools-call-simple-text
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/tools.ts#L96

tools-call-image
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/tools.ts#L180

tools-call-audio
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/tools.ts#L915

tools-call-embedded-resource
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/tools.ts#L1009

tools-call-mixed-content
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/tools.ts#L267

tools-call-with-logging
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/tools.ts#L367

tools-call-error
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/tools.ts#L455

tools-call-with-progress
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/tools.ts#L539

tools-call-sampling
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/tools.ts#L658

tools-call-elicitation
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/tools.ts#L785

JSON

json-schema-2020-12
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/json-schema-2020-12.ts#L15

Elicitation

elicitation-sep1034-defaults
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/elicitation-defaults.ts#L9

elicitation-sep1330-enums
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/elicitation-enums.ts#L10

Resources

resources-list
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/resources.ts#L12

resources-read-text
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/resources.ts#L93

resources-read-binary
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/resources.ts#L178

resources-templates-read
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/resources.ts#L262

resources-subscribe
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/resources.ts#L363

resources-unsubscribe
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/resources.ts#L433

Prompts

prompts-list
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/prompts.ts#L8

prompts-get-simple
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/prompts.ts#L88

prompts-get-with-args
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/prompts.ts#L172

prompts-get-embedded-resource
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/prompts.ts#L267

prompts-get-with-image
https://github.com/modelcontextprotocol/conformance/blob/e5cdffec37c09fbaa1e4ad1717bb4892033df0c4/src/scenarios/server/prompts.ts#L372