Skip to content

Commit b6b765b

Browse files
lpcoxCopilot
andcommitted
fix: move smoke-gemini tests into agent container
Previously, smoke-gemini ran all tests (curl connectivity, file write/read, gh pr list) in a host pre-step, then had the agent merely verify pre-computed results. This meant the tests validated host connectivity, not AWF sandbox connectivity. Now the agent performs all tests inside the sandbox (like smoke-claude), properly exercising the firewall's domain allowlist, bash tool, and MCP connectivity from within the container. Changes: - Remove Pre-compute smoke test data host pre-step - Move test requirements into agent prompt - Agent now runs curl, file write/read, and MCP calls itself - Keep post-step safe-output validation unchanged Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent c99032c commit b6b765b

2 files changed

Lines changed: 19 additions & 89 deletions

File tree

.github/workflows/smoke-gemini.lock.yml

Lines changed: 12 additions & 33 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

.github/workflows/smoke-gemini.md

Lines changed: 7 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -44,41 +44,6 @@ sandbox:
4444
agent:
4545
version: v0.25.29
4646
strict: true
47-
steps:
48-
- name: Pre-compute smoke test data
49-
id: smoke-data
50-
run: |
51-
echo "::group::Fetching last 2 merged PRs"
52-
PR_DATA=$(gh pr list --repo "$GITHUB_REPOSITORY" --state merged --limit 2 \
53-
--json number,title,author,mergedAt \
54-
--jq '.[] | "PR #\(.number): \(.title) (by @\(.author.login), merged \(.mergedAt))"')
55-
echo "$PR_DATA"
56-
echo "::endgroup::"
57-
58-
echo "::group::GitHub.com connectivity check"
59-
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 https://github.com)
60-
echo "github.com returned HTTP $HTTP_CODE"
61-
echo "::endgroup::"
62-
63-
echo "::group::File write/read test"
64-
TEST_DIR="/tmp/gh-aw/agent"
65-
TEST_FILE="$TEST_DIR/smoke-test-gemini-${GITHUB_RUN_ID}.txt"
66-
mkdir -p "$TEST_DIR"
67-
echo "Smoke test passed for Gemini at $(date)" > "$TEST_FILE"
68-
FILE_CONTENT=$(cat "$TEST_FILE")
69-
echo "Wrote and read back: $FILE_CONTENT"
70-
echo "::endgroup::"
71-
72-
{
73-
echo "SMOKE_PR_DATA<<SMOKE_EOF"
74-
echo "$PR_DATA"
75-
echo "SMOKE_EOF"
76-
echo "SMOKE_HTTP_CODE=$HTTP_CODE"
77-
echo "SMOKE_FILE_CONTENT=$FILE_CONTENT"
78-
echo "SMOKE_FILE_PATH=$TEST_FILE"
79-
} >> "$GITHUB_OUTPUT"
80-
env:
81-
GH_TOKEN: ${{ github.token }}
8247
post-steps:
8348
- name: Validate safe outputs were invoked
8449
run: |
@@ -103,35 +68,21 @@ post-steps:
10368

10469
**IMPORTANT: Keep all outputs extremely short and concise. Use single-line responses where possible. No verbose explanations.**
10570

106-
## Pre-Computed Test Results
107-
108-
The following tests were already executed in a deterministic pre-agent step. Your job is to verify the results and produce the summary comment.
109-
110-
### 1. GitHub MCP Testing
111-
Verify MCP connectivity by calling `github-list_pull_requests` for ${{ github.repository }} (limit 1, state merged). Confirm the result matches the pre-fetched data below.
112-
113-
### 2. GitHub.com Connectivity
114-
Pre-step result: HTTP ${{ steps.smoke-data.outputs.SMOKE_HTTP_CODE }} from github.com.
115-
✅ if HTTP 200 or 301, ❌ otherwise.
116-
117-
### 3. File Write/Read Test
118-
Pre-step wrote and read back: "${{ steps.smoke-data.outputs.SMOKE_FILE_CONTENT }}"
119-
File path: ${{ steps.smoke-data.outputs.SMOKE_FILE_PATH }}
120-
Verify by running `cat` on the file path using bash to confirm it exists.
71+
> Use `perPage: 2` when listing PRs.
12172
122-
## Pre-Fetched PR Data
73+
## Test Requirements
12374

124-
```
125-
${{ steps.smoke-data.outputs.SMOKE_PR_DATA }}
126-
```
75+
1. **GitHub MCP Testing**: Review the last 2 merged pull requests in ${{ github.repository }}
76+
2. **GitHub.com Connectivity**: Use bash to run `curl -s -o /dev/null -w "%{http_code}" --max-time 10 https://github.com` and verify the HTTP status is 200 or 301
77+
3. **File Writing Testing**: Create a test file `/tmp/gh-aw/agent/smoke-test-gemini-${{ github.run_id }}.txt` with content "Smoke test passed for Gemini at $(date)" (create the directory if it doesn't exist)
78+
4. **Bash Tool Testing**: Execute bash commands to verify file creation was successful (use `cat` to read the file back)
12779

12880
## Output
12981

130-
Add a **very brief** comment (max 5-10 lines) to the current pull request with:
82+
**If triggered by a pull request**, add a **very brief** comment (max 5-10 lines) to the current pull request with:
13183
- PR titles only (no descriptions)
13284
- ✅ or ❌ for each test result
13385
- Overall status: PASS or FAIL
134-
- Mention the pull request author and any assignees
13586

13687
If all tests pass, add the label `smoke-gemini` to the pull request.
13788

0 commit comments

Comments
 (0)