test: add Q CLI and MCP integration testing#6
test: add Q CLI and MCP integration testing#6ezhang6811 wants to merge 28 commits intoaws-actions:mainfrom ezhang6811:integ-test
Conversation
| - cron: '0 */6 * * *' | ||
| push: | ||
| branches: | ||
| - integ-test |
There was a problem hiding this comment.
temporary manual trigger, remove before merging
| aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }} | ||
| aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} |
There was a problem hiding this comment.
TODO: let's change this to use IAM role later
There was a problem hiding this comment.
will address in next PR.
| @@ -0,0 +1,77 @@ | |||
| name: Integration Test | |||
There was a problem hiding this comment.
So this test will be triggered in 2 scenarios?
- PR push
- Periodical check like soaking test? should we emit metrics on result and alarm for any issues
There was a problem hiding this comment.
Added two triggers. One is a main build (PR merged to main), the other is a soak test (every 6 hours but can be reduced). Currently both point to the source code; once we release the initial version of the action I will update the soak test trigger uses: aws-actions/apm-action@v1.
There was a problem hiding this comment.
Added job to publish success metric.
| ALLOWED_NON_WRITE_USERS: ${{ inputs.allowed_non_write_users }} | ||
| GITHUB_RUN_ID: ${{ github.run_id }} | ||
| DEFAULT_WORKFLOW_TOKEN: ${{ github.token }} | ||
| TRACING_MODE: ${{ inputs.tracing_mode }} |
There was a problem hiding this comment.
why tracing mode got enabled?
| const targetBranch = process.env.TARGET_BRANCH || ''; | ||
| const allowedNonWriteUsers = process.env.ALLOWED_NON_WRITE_USERS || ''; | ||
| const customPrompt = process.env.CUSTOM_PROMPT || ''; | ||
| const tracingMode = process.env.TRACING_MODE || 'false'; |
There was a problem hiding this comment.
removed both instances
2 participants
Issue #, if available:
Description of changes:
Created two integration tests that verify the communication between Q CLI and MCP servers.
Both tests pass in a custom prompt that tell Q CLI to call both Application Signals and Cloudwatch MCP tools. In test mode (only runnable from this repo in the main branch), the logic to read the Github context is bypassed and the agent reads the custom prompt directly.
The tests uploads the saved raw output file, and then verifies that the output logs the
list_monitored_servicestool call to the Application Signals MCP server, as well as theget_active_alarmstool call from the CloudWatch MCP server.Successful test run: https://github.com/ezhang6811/application-observability-for-aws/actions/runs/19115766214
Next steps:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.