[awf] agent+api-proxy: P0/P1 failure cluster — api-proxy health check + node not found in copilot container

[lpcox]

Problem

This is a combined P0/P1 failure report covering 28 failed/cancelled runs out of 40 total over the 07:00–13:00 UTC window on 2026-04-28. Two distinct failure clusters were identified:

P0 — awf-api-proxy health check timeout (at least 3 confirmed runs): Affects Sub-Issue Closer (copilot), Daily Team Evolution Insights (claude), Smoke CI (copilot). Container starts but health check times out, blocking docker compose up entirely.

P1 — node: command not found in copilot agent container: At least 2 scheduled copilot-engine workflows fail inside the container because node is not on PATH. Affects: Daily Issues Report Generator, Daily News.

Context

Parent investigation: github/gh-aw#28947. The api-proxy health check failure is also separately tracked.

Root Cause

P0 (api-proxy health check): See separate tracking issue for #28898/#28949 — tight healthcheck timing under runner load.

P1 (node not found): The containers/agent/entrypoint.sh chroots to /host and drops capabilities before running the user command. The PATH inside the chroot may not include the directory where the host's node binary lives (e.g., /usr/local/bin/node, /opt/hostedtoolcache/node/.../bin). The bind mounts include /usr, /bin, /sbin read-only, but /usr/local or nvm/volta paths under /opt may not be fully covered.

Proposed Solution

P1 fix:

1. In containers/agent/entrypoint.sh: After chroot, construct PATH by including common Node.js installation locations: /usr/local/bin, /opt/hostedtoolcache/node/*/bin (glob expanded), /home/runner/.nvm/versions/node/*/bin.
2. Alternatively: Add a --extra-path <dir> CLI flag in src/cli.ts that appends additional directories to the agent container PATH.
3. In src/docker-manager.ts: When generating the Docker Compose env section, include the host's PATH value or a curated superset to ensure tool availability inside the chroot.

P0 fix: Tracked in the api-proxy health check issues.

Generated by Firewall Issue Dispatcher · ● 436.3K · ◷

[github-actions]

👋 Hello! I noticed this issue might be related to existing issues:

P0 — api-proxy health check:

• [awf] api-proxy: Intermittent health check failure blocks agent startup across multiple engines #2295 - Intermittent health check failure blocks agent startup across multiple engines
• [awf] api-proxy: Flaky health check causes agent startup abort in downstream workflows #2293 - Flaky health check causes agent startup abort in downstream workflows
• [awf] api-proxy: Flaky awf-api-proxy unhealthy failure during docker compose up #2266 - Flaky awf-api-proxy unhealthy failure during docker compose up

P1 — node not found in copilot container:

• [awf] Agent + API Proxy: P0/P1 failure report — health check intermittent + node not found in copilot container #2275 - Very similar combined P0/P1 failure report (health check intermittent + node not found in copilot container)
• [awf] Agent Container: intermittent engine binary missing at startup (node/codex not found) #2247 - Intermittent engine binary missing at startup (node/codex not found)
• [awf] Agent container: Copilot CLI fails with node: command not found on runner bootstrap #2159 - Copilot CLI fails with node: command not found on runner bootstrap

In particular, #2275 appears to be a near-identical combined P0/P1 report. Please review the linked issues to see if any already address your concern, or clarify how this report differs (e.g., different time window, affected runs, or root cause findings).

This is an automated message from the issue duplication detector.

Generated by Issue Duplication Detector for issue #2294 · ● 200.3K · ◷

[lpcox]

Closing — both failure clusters already addressed by merged PRs:

P0 (api-proxy health check timeout): Fixed by PR #2255 (merged Apr 28) — increased health check tolerances (start_period 30s, retries 15, interval 2s, timeout 3s).

P1 (node: command not found): Fixed by PR #2160 (merged) — added ~/.nvm to agent bind mounts and AWF_REQUIRE_NODE preflight check for Copilot workflows, ensuring Node.js is discoverable inside the chroot.

Both fixes are included in firewall v0.25.29+.