Skip to content

feat: replace mcp-server-tester with RTF smoke tests #727

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
DaleSeo wants to merge 4 commits into
base: main
Choose a base branch
from
Open

feat: replace mcp-server-tester with RTF smoke tests #727

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c3910fc
feat: replace mcp-server-tester with RTF smoke tests
DaleSeo Apr 14, 2026
085742d
fix: use gh cli for authenticated clone of rtf repo in CI
DaleSeo Apr 15, 2026
0353a74
fix: use rtf-run-test-plan action from release-tooling
DaleSeo Apr 15, 2026
e95f4aa
fix: update healthcheck variable to match rtf-mcp scenario changes
DaleSeo Apr 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 17 additions & 12 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,15 +85,20 @@ jobs:

- uses: actions-rust-lang/setup-rust-toolchain@v1

- uses: actions/setup-node@v4
with:
node-version: '20'

- name: Build release binary
run: cargo build --release

- name: Run e2e tests (local-operations)
run: ./e2e/mcp-server-tester/run_tests.sh local-operations

- name: Run e2e tests (pq-manifest)
run: ./e2e/mcp-server-tester/run_tests.sh pq-manifest
- name: Build Docker image from current code
run: docker build -t apollo-mcp-server:local .

- name: Install RTF CLI
env:
GH_TOKEN: ${{ github.token }}
run: |
gh repo clone apollographql/runtime-testing-framework /tmp/rtf -- --depth 1
cargo install --path /tmp/rtf/crates/rtf-cli

- name: Run e2e smoke tests
env:
GITHUB_TOKEN: ${{ github.token }}
run: |
rtf run e2e/rtf/smoke/test-plan.yaml -v \
--var 'mcp_server_image=apollo-mcp-server' \
--var 'mcp_server_tag=local'
Copy link
Copy Markdown

@sminez sminez Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should use the action we provide from release-tools for running the test plan rather than attempting to clone the repo like this. RTF is an internal repo still which is why you're getting that error in your CI run.

Copy link
Copy Markdown
Member Author

@DaleSeo DaleSeo Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's cool, @sminez! Updated the workflow to use it. :)

Copy link
Copy Markdown
Member Author

@DaleSeo DaleSeo Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any ideas about this permission issue in CI?

2026-04-15 at 11 57 17

https://github.com/apollographql/apollo-mcp-server/actions/runs/24463431474/job/71483772955?pr=727

Copy link
Copy Markdown

@sminez sminez Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be resolved now following the chat in Slack you had with @dandersonhall, but you still have an issue with attempting to pull files from the rtf-mcp repo. Like RTF itself, that is also a private repo and the default token that github gives you is only scoped for interactions with this repo I believe?

We normally handle this by creating an appropriately scoped token and storing it as a repo secret.

Loading
Loading