security: fix workflow permission check bypass

[EdwardIrby]

Security Issue: Workflow Permission Check Bypass

Description

GitHub Actions workflows with permission checks that use exit 0 have a security vulnerability: the exit only stops that specific step, but subsequent steps continue to run. This allows unauthorized users to trigger workflow actions.

Vulnerable Pattern

- name: Check author permission
  run: |
    if [[ "$PERMISSION" \!= "admin" ]]; then
      echo "Unauthorized"
      exit 0  # ❌ Only stops THIS step
    fi

- name: Run expensive action  # ⚠️ Still runs\!

Impact

• Unauthorized users can trigger workflows
• Potential cost implications (API calls, compute time)
• Security boundary is not enforced

Fix

Use step outputs and conditional execution:

- name: Check author permission
  id: author-permission
  run: |
    if [[ "$PERMISSION" \!= "admin" ]]; then
      echo "authorized=false" >> "$GITHUB_OUTPUT"
      exit 0
    fi
    echo "authorized=true" >> "$GITHUB_OUTPUT"

- name: Run expensive action
  if: steps.author-permission.outputs.authorized == 'true'

Additional Workflow Enhancement

Also enhance workflow prompts to explicitly gather existing PR feedback before reviewing:

prompt: |
  Before reviewing, gather existing PR feedback:
  - `gh pr view ${{ github.event.pull_request.number }} --repo ${{ github.repository }} --json comments,reviews,reviewThreads`

References

• Original issue: Calibrate grader youdotcom-oss/web-search-agent-evals#10 (comment)
• Fix commit: youdotcom-oss/web-search-agent-evals@6f74272
• PR feedback enhancement: Calibrate grader youdotcom-oss/web-search-agent-evals#10 (comment)

Action Required

1. Audit all workflows with permission checks
2. Add step IDs and output flags
3. Gate subsequent steps with if: conditions
4. Add PR feedback gathering instructions to review workflows