Skip to content

functional-tests: every triage case fails with API Error policy_denied on main since 2026-09-03 #6962

Closed
#6975
Closed
functional-tests: every triage case fails with API Error policy_denied on main since 2026-09-03#6962
#6975
Labels
agent/triageTriage agentbugcomponent/ciCI pipelines and checkspriority/criticalProduction outages or data lossready-to-codeTriggers code agent dispatchtype/bugConfirmed defect in existing behavior

Description

@waynesun09
waynesun09
opened on Sep 3, 2026

What happened

functional-tests fails on main itself: every triage case gets API Error: Error code policy_denied on the very first Vertex call (agent runs ~2 s, apiKeySource: none, $0.00), then validation fails after 2 iterations.

Case output (from the eval-results artifact): → Agent: claude-opus-5 (v2.1.258), two api_retry events with error: unknown, then API Error: Error code policy_denied. One case in one attempt also hit an OpenShell provider race (default credential was modified concurrently during persist credential) with 4 parallel cases.

Suspects in the window (01:43Z → 12:55Z)

Not verified: whether the denied request is the Vertex model call itself or an earlier call the new claude version makes; the gateway's denial log on the runner is not in the artifact.

Validation criteria

workflow_dispatch of functional-tests.yml on main executes the four triage cases and passes.

Activity

  1. github-project-automation
    added this to fullsend-ai Workstreamon Sep 3, 2026
  2. github-project-automation
    moved this to Todo in fullsend-ai Workstreamon Sep 3, 2026

  3. fullsend-ai-triage commented on Sep 3, 2026

    @fullsend-ai-triage
    fullsend-ai-triageBot
    on Sep 3, 2026 – with fullsend-ai-triage

    🤖 Finished Triage · ✅ Success · Started 1:59 PM UTC · Completed 2:04 PM UTC

    Commit: 2a10349 · View workflow run →

    Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $0.84

  4. fullsend-ai-triage
    added
    bug
    type/bugConfirmed defect in existing behavior
    priority/criticalProduction outages or data loss
    component/ciCI pipelines and checks
    agent/triageTriage agent
    ready-to-codeTriggers code agent dispatch
    on Sep 3, 2026

  5. fullsend-ai-triage commented on Sep 3, 2026

    @fullsend-ai-triage
    fullsend-ai-triageBot
    on Sep 3, 2026 – with fullsend-ai-triage

    Triage Summary

    Severity: Critical — all triage functional tests are broken on main, and the breakage was invisible for ~11 hours due to path-filter skipping on PR runs.

    Root cause hypothesis

    The most likely cause is fullsend-ai/agents#1160 (merged 11:40Z), which repinned harness images from 0.37.0 to 0.40.0 (fullsend-sandbox@sha256:f8255971, claude v2.1.258). The functional-tests workflow resolves fullsend-ai/agents@main at runtime, so it immediately picked up the new images. The policy_denied error on the first Vertex call suggests the new claude version or sandbox image makes API requests that violate a Model Armor policy rule — either the model call format changed or the new version makes an additional call the policy doesn't allow.

    The concurrent credential race (default credential was modified concurrently during persist credential) observed in one case under 4-way parallelism may be a separate issue in the 0.40.0 sandbox's credential handling.

    Recommended investigation

    1. Confirm the suspect: Run functional-tests with agents pinned to pre-mint: add retro and prioritize roles to test defaults #1160 (0.37.0 images). If tests pass, the repin is the cause.
    2. Check Vertex gateway logs for the specific Model Armor policy rule being triggered.
    3. Short-term mitigation: Revert agents image pins to 0.37.0 while the policy/image issue is resolved.
    4. Investigate credential race under parallel case execution separately.

    Proposed test case

    Run functional-tests workflow_dispatch on main:
- All 4 triage cases should pass
- Each case should show non-zero cost (real Vertex API usage)
- No policy_denied or api_retry errors in eval-results
- No concurrent credential modification errors

    Related issues

    Labels: CI pipeline bug affecting triage agent functional tests with critical severity — all tests broken on main.

    Next steps:

    • /fs-code — agent creates a PR to implement this issue
    • /fs-code <your instruction> — agent implements with your specific guidance

  6. fullsend-ai-coder commented on Sep 3, 2026

    @fullsend-ai-coder
    fullsend-ai-coderBot
    on Sep 3, 2026 – with fullsend-ai-coder
    Contributor

    🤖 Finished Code · ❌ Failure (post-script /home/runner/work/fullsend/fullsend/.fullsend/.fullsend-cache/resources/sha256/21644e0238d0b83db763fa138c412322223b0ba4a919a542b839e11b6a3d62e7/scripts/post-code.sh failed: exit status 1) · Started 2:06 PM UTC · Completed 2:31 PM UTC

    Commit: 2a10349 · View workflow run →

    Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $10.15

  7. fullsend-ai-coder commented on Sep 3, 2026

    @fullsend-ai-coder
    fullsend-ai-coderBot
    on Sep 3, 2026 – with fullsend-ai-coder
    Contributor

    ⚠️ Post-code script failed — Push rejected — workflows permission (exit code 1)

    The code agent completed, but the post-code script failed before finishing.

    Security boundary: the coder app intentionally lacks workflows write permission. Changes to .github/workflows/ must be made outside the agent (e.g., via a manual PR). Re-run the agent without workflow file changes, or apply those changes separately.

    Workflow run: https://github.com/fullsend-ai/fullsend/actions/runs/33764725541

    Details:
    To https://github.com/fullsend-ai/fullsend.git
    ! [remote rejected] agent/6962-pin-functional-test-model -> agent/6962-pin-functional-test-model (refusing to allow a GitHub App to create or update workflow .github/workflows/functional-tests.yml without workflows permission)
    error: failed to push some refs to 'https://github.com/fullsend-ai/fullsend.git'
    To https://github.com/fullsend-ai/fullsend.git
    ! [remote rejected] agent/6962-pin-functional-test-model -> agent/6962-pin-functional-test-model (refusing to allow a GitHub App to create or update workflow .github/workflows/functional-tests.yml without workflows permission)
    error: failed to push some refs to 'https://github.com/fullsend-ai/fullsend.git'
    Please check the workflow logs for full details and retry with /fs-code if appropriate.

  8. waynesun09 commented on Sep 3, 2026

    @waynesun09
    waynesun09
    on Sep 3, 2026
    MemberAuthor

    Root cause correction: this is not Model Armor and not model authorization. The gateway log in the eval-results artifact of run 33763731846 shows OpenShell's OPA denying the Claude Code binary itself:

    NET:OPEN DENIED /usr/lib/node_modules/@anthropic-ai/claude-code/bin/claude.exe -> sts.googleapis.com:443
reason: binary '.../claude.exe' not allowed in policy '_provider_vertex_ai'

    Claude Code 2.1.2xx is a Bun binary named claude.exe; the 0.40.0 image (2.1.258) runs it, and internal/scaffold/fullsend-repo/profiles/fullsend-vertex-ai.yaml only allowlists **/claude. The fixed profile from fullsend-ai/agents#1118 is fetched by the functional run, but <fullsend-dir>/profiles/ is imported afterwards and silently replaces it (FULLSEND_DIR=internal/scaffold/fullsend-repo). Fix tracked in #6971; pinning the model would not have helped.

  9. waynesun09
    added 2 commits that reference this issue on Sep 3, 2026

    fix(run): warn when a profiles/ directory copy shadows a harness-reso…

    701718b

    fix(run): warn when a profiles/ directory copy shadows a harness-reso…

    e170772
  10. github-project-automation
    moved this from Todo to In progress in fullsend-ai Workstreamon Sep 3, 2026
  11. waynesun09
    added a commit that references this issue on Sep 3, 2026

    fix(run): warn when a profiles/ directory copy shadows a harness-reso…

    089e0ef
  12. waynesun09
    closed this as completedin #6975on Sep 3, 2026
  13. github-project-automation
    moved this from In progress to Done in fullsend-ai Workstreamon Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/triageTriage agentbugcomponent/ciCI pipelines and checkspriority/criticalProduction outages or data lossready-to-codeTriggers code agent dispatchtype/bugConfirmed defect in existing behavior

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions