You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Previously, smoke-gemini ran all tests (curl connectivity, file write/read,
gh pr list) in a host pre-step, then had the agent merely verify pre-computed
results. This meant the tests validated host connectivity, not AWF sandbox
connectivity.
Now the agent performs all tests inside the sandbox (like smoke-claude),
properly exercising the firewall's domain allowlist, bash tool, and MCP
connectivity from within the container.
Changes:
- Remove Pre-compute smoke test data host pre-step
- Move test requirements into agent prompt
- Agent now runs curl, file write/read, and MCP calls itself
- Keep post-step safe-output validation unchanged
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
echo "Smoke test passed for Gemini at $(date)" > "$TEST_FILE"
68
-
FILE_CONTENT=$(cat "$TEST_FILE")
69
-
echo "Wrote and read back: $FILE_CONTENT"
70
-
echo "::endgroup::"
71
-
72
-
{
73
-
echo "SMOKE_PR_DATA<<SMOKE_EOF"
74
-
echo "$PR_DATA"
75
-
echo "SMOKE_EOF"
76
-
echo "SMOKE_HTTP_CODE=$HTTP_CODE"
77
-
echo "SMOKE_FILE_CONTENT=$FILE_CONTENT"
78
-
echo "SMOKE_FILE_PATH=$TEST_FILE"
79
-
} >> "$GITHUB_OUTPUT"
80
-
env:
81
-
GH_TOKEN: ${{ github.token }}
82
47
post-steps:
83
48
- name: Validate safe outputs were invoked
84
49
run: |
@@ -103,35 +68,21 @@ post-steps:
103
68
104
69
**IMPORTANT: Keep all outputs extremely short and concise. Use single-line responses where possible. No verbose explanations.**
105
70
106
-
## Pre-Computed Test Results
107
-
108
-
The following tests were already executed in a deterministic pre-agent step. Your job is to verify the results and produce the summary comment.
109
-
110
-
### 1. GitHub MCP Testing
111
-
Verify MCP connectivity by calling `github-list_pull_requests` for ${{ github.repository }} (limit 1, state merged). Confirm the result matches the pre-fetched data below.
112
-
113
-
### 2. GitHub.com Connectivity
114
-
Pre-step result: HTTP ${{ steps.smoke-data.outputs.SMOKE_HTTP_CODE }} from github.com.
115
-
✅ if HTTP 200 or 301, ❌ otherwise.
116
-
117
-
### 3. File Write/Read Test
118
-
Pre-step wrote and read back: "${{ steps.smoke-data.outputs.SMOKE_FILE_CONTENT }}"
Verify by running `cat` on the file path using bash to confirm it exists.
71
+
> Use `perPage: 2` when listing PRs.
121
72
122
-
## Pre-Fetched PR Data
73
+
## Test Requirements
123
74
124
-
```
125
-
${{ steps.smoke-data.outputs.SMOKE_PR_DATA }}
126
-
```
75
+
1.**GitHub MCP Testing**: Review the last 2 merged pull requests in ${{ github.repository }}
76
+
2.**GitHub.com Connectivity**: Use bash to run `curl -s -o /dev/null -w "%{http_code}" --max-time 10 https://github.com` and verify the HTTP status is 200 or 301
77
+
3.**File Writing Testing**: Create a test file `/tmp/gh-aw/agent/smoke-test-gemini-${{ github.run_id }}.txt` with content "Smoke test passed for Gemini at $(date)" (create the directory if it doesn't exist)
78
+
4.**Bash Tool Testing**: Execute bash commands to verify file creation was successful (use `cat` to read the file back)
127
79
128
80
## Output
129
81
130
-
Add a **very brief** comment (max 5-10 lines) to the current pull request with:
82
+
**If triggered by a pull request**, add a **very brief** comment (max 5-10 lines) to the current pull request with:
131
83
- PR titles only (no descriptions)
132
84
- ✅ or ❌ for each test result
133
85
- Overall status: PASS or FAIL
134
-
- Mention the pull request author and any assignees
135
86
136
87
If all tests pass, add the label `smoke-gemini` to the pull request.
0 commit comments