Skip to content

Log the resolved filesystem.allowWrite boundary before the Cloud Hypervisor guest boots - #7819

Merged
lpcox merged 3 commits into
mainfrom
copilot/awf-repo-memory-fix
Aug 28, 2026
Merged

Log the resolved filesystem.allowWrite boundary before the Cloud Hypervisor guest boots#7819
lpcox merged 3 commits into
mainfrom
copilot/awf-repo-memory-fix

Conversation

Copilot AI commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

An agent could not persist repo-memory: writes to /tmp/gh-aw/repo-memory/default/ failed with EROFS and no indication of why. AWF does not mount that path read-only — /tmp is bound read-write in buildWorkspaceMounts(). The read-only view came from the caller's filesystem.allowWrite policy, which the failing run emitted as:

"filesystem":{"allowWrite":["/tmp/gh-aw/agent","/workspace","/workspace/.awf-home","/tmp/gh-aw/cache-memory"]}

/tmp/gh-aw is one virtio-fs export; under a policy its host backing tree is staged read-only with writable overlays only for listed paths, so repo-memory is correctly read-only. Widening the allowlist belongs to the gh-aw compiler (as was done for cache-memory). What is missing on the AWF side is any way to see the boundary, so the failure surfaced as a bare EROFS.

Changes

  • src/cloud-hypervisor/filesystem-write-policy.ts — adds pure summarizeCloudHypervisorFilesystemWriteBoundary(plan), rendering one line per export from the existing plan (writable / read-only / selective with named overlays). Returns [] when unrestricted.

  • src/cloud-hypervisor/filesystem-write-enforcement.ts — surfaces the summary as writeBoundary. The unrestricted return still omits the mountEnforcement key, so legacy virtiofsd staging stays byte-identical.

  • src/cloud-hypervisor-runtime-backend.ts — logs the boundary once at the existing filesystem-write-policy stage, before virtiofsd, the VMM, or the guest start:

    [cloud-hypervisor] stage=filesystem-write-policy boundary /workspace=ro /tmp/gh-aw=ro except /tmp/gh-aw/agent (writes outside these paths fail with EROFS; widen filesystem.allowWrite to permit them)
    
  • docs/troubleshooting.md — new "Read-Only Filesystem (EROFS) Inside the Sandbox" entry under Permission Issues, using repo-memory as the worked example and noting that gh-aw-launched runs must declare the path in sandbox.agent.config.filesystem.allowWrite.

  • docs/awf-config-spec.md §4.1 — states that caller-owned directories under /tmp/gh-aw (e.g. /tmp/gh-aw/repo-memory) must appear in allowWrite, and that the boundary is reported before boot.

Notes for reviewers

No behavioral change to mounting or policy enforcement — only a derived summary and one log line. Tests cover the summary shape, the unrestricted (silent) case, and the backend log line.

Copilot AI changed the title [WIP] Fix repo-memory persistence issue in AWF sandbox Log the resolved filesystem.allowWrite boundary before the Cloud Hypervisor guest boots Aug 28, 2026
Copilot AI requested a review from lpcox August 28, 2026 04:21
@lpcox
lpcox marked this pull request as ready for review August 28, 2026 04:25
Copilot AI balanced review requested due to automatic review settings August 28, 2026 04:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds pre-boot diagnostics for Cloud Hypervisor filesystem write boundaries to explain EROFS failures.

Changes:

  • Derives and logs effective per-export write boundaries.
  • Adds unit coverage for restricted and unrestricted policies.
  • Documents filesystem.allowWrite troubleshooting and requirements.
Show a summary per file
File Description
src/cloud-hypervisor/filesystem-write-policy.ts Adds boundary summarization.
src/cloud-hypervisor/filesystem-write-policy.test.ts Tests summary formats.
src/cloud-hypervisor/filesystem-write-enforcement.ts Exposes the resolved boundary.
src/cloud-hypervisor/filesystem-write-enforcement.test.ts Tests enforcement summaries.
src/cloud-hypervisor-runtime-backend.ts Logs the boundary before boot.
src/cloud-hypervisor-runtime-backend.test.ts Tests logging and silence when unrestricted.
docs/troubleshooting.md Adds EROFS troubleshooting guidance.
docs/awf-config-spec.md Documents caller-owned writable paths.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

  • Files reviewed: 8/8 changed files
  • Comments generated: 2
  • Review effort level: Balanced

Comment thread src/cloud-hypervisor-runtime-backend.ts Outdated
Comment on lines +274 to +275
`[cloud-hypervisor] stage=filesystem-write-policy boundary ${writeBoundary.join(' ')} ` +
'(writes outside these paths fail with EROFS; widen filesystem.allowWrite to permit them)',
Comment thread docs/troubleshooting.md Outdated
Comment on lines +220 to +224
**Cause:** AWF does not mount any `/tmp` subdirectory read-only on its own. A
read-only view of such a path comes from a `filesystem.allowWrite` policy in the
AWF config file: when that key is present, every writable host mount is narrowed
to the listed guest-visible paths and everything else becomes read-only (see
[awf-config-spec.md §4.1](./awf-config-spec.md#41-filesystem-write-boundary)).
@lpcox

lpcox commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

@copilot address review feedback

Copilot AI commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

@copilot address review feedback

Addressed both review threads in 8622a1b: the log now limits widening guidance to paths under originally writable exports, and the troubleshooting cause is scoped to the Cloud Hypervisor repo-memory case.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@copilot Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

@github-actions

Copy link
Copy Markdown
Contributor

Documentation Preview

Documentation build failed for this PR. View logs.

Built from commit 2a23a50

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Copilot

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed...

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

🔌 Service connectivity validated by Smoke Services

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Build Test Suite completed successfully!

Generated by Build Test Suite for #7819

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

Tested by Smoke Chroot

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Docker Sbx has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Docker Sbx

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (Entra) reports failed. AOAI BYOK (Entra) mode investigation needed...

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Contribution Check completed successfully!

PR #7819 follows CONTRIBUTING.md: it includes tests for the new boundary-summary behavior and updates docs for the new EROFS troubleshooting guidance. The PR description is clear and references the related issue (#7811). No contribution-guidelines issues found.

Generated by Contribution Check for #7819

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

📡 OTel tracing validated by Smoke OTel Tracing

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Smoke Gemini reports failed. Facets need polishing...

💎 Faceted by Smoke Gemini

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Smoke Claude passed

Generated by Smoke Claude for #7819

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Security Guard has started processing this pull request

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

🔑 BYOK report filed by Smoke Copilot BYOK

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Docker Sbx@lpcox

Overall: PASS

📰 BREAKING: Report filed by Smoke Docker Sbx
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test Results @lpcox

Overall: PASS

📰 BREAKING: Report filed by Smoke Copilot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Direct) Mode ✅

Test Result
GitHub MCP ✅ (PR list verified)
github.com connectivity ✅ (HTTP 200)
File write/read ✅ (smoke-test-copilot-byok.txt)
BYOK inference ✅ (responding in direct mode)

Mode: Direct BYOK (COPILOT_PROVIDER_API_KEY) via api-proxy → api.githubcopilot.com

Status: PASS

🔑 BYOK report filed by Smoke Copilot BYOK
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

EGRESS_RESULT allow=pass deny=pass

✅ Allowed domain (github.com) reachable: allowed=200
✅ Blocked domain (example.com) denied: CONNECT tunnel failed, 403

Overall status: PASS

cc @lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation
Add label ready-for-aw to run again

@github-actions github-actions Bot added the smoke-copilot-network-isolation Copilot network-isolation egress smoke test label Aug 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Claude Engine Validation

Check Result
API status ✅ PASS
gh check ✅ PASS
File status ✅ PASS

Overall result: PASS

Generated by Smoke Claude for #7819 · claude · haiku45 · 32.7 AIC · ⊞ 4.5K ·
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Chroot Version Comparison

Runtime Host Version Chroot Version Match?
Python Python 3.12.14 Python 3.12.14 ✅ YES
Node.js v24.19.0 v22.23.2 ❌ NO
Go go1.22.12 go1.22.12 ✅ YES

Result: Not all versions matched (Node.js differs), so the smoke-chroot label was not added.

Tested by Smoke Chroot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: GitHub Actions Services Connectivity

  • Redis PING: ❌ (Temporary failure in name resolution)
  • Postgres pg_isready: ❌ (no response)
  • Postgres SELECT 1: ❌ (could not translate host name)

Overall: FAILhost.docker.internal did not resolve inside the AWF sandbox.

🔌 Service connectivity validated by Smoke Services
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

PR titles: unavailable (safeinputs-gh missing)
Merged PR review: ❌
Safe-input PR query: ❌
Playwright GitHub title: ✅
Temp file write/read: ✅
Discussion lookup/comment: ❌
Build (npm ci && npm run build): ✅
Overall: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A ✅ PASS
.NET json-parse N/A ✅ PASS
Go color 1/1 passed ✅ PASS
Go env 1/1 passed ✅ PASS
Go uuid 1/1 passed ✅ PASS
Java gson 1/1 passed ✅ PASS
Java caffeine 1/1 passed ✅ PASS
Node.js clsx passed ✅ PASS
Node.js execa passed ✅ PASS
Node.js p-limit passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — PASS

Notes:

  • All repositories cloned successfully through the AWF firewall's domain allowlist.
  • Java builds required using a custom -Dmaven.repo.local path (~/.m2 was root-owned and not writable by the runner user in this sandbox) — Maven proxy settings via ~/.m2/settings.xml worked correctly once the local repo path was set.
  • No network egress issues encountered for any ecosystem (npm, cargo, go modules, deno.land, bun, maven via squid-proxy).

Generated by Build Test Suite for #7819 · copilot · auto · 60.9 AIC · ⊞ 12K ·
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: API Proxy OpenTelemetry Tracing — Results

Scenario Result
1. Module Loading (otel.js) ✅ Loaded successfully. isEnabled: true. Exports: startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled, _provider, _ProxyAwareOtlpExporter, _FileSpanExporter, _FanOutSpanExporter, _parseEndpoints, _parseOtlpHeaders, _buildResourceSpans, _createOtlpWorkloadIdentity
2. Test Suite (otel*.test.js) ✅ 3 suites / 68 tests passed, 0 failed
3. Env Var Forwarding GITHUB_AW_OTEL_TRACE_ID / GITHUB_AW_OTEL_PARENT_SPAN_ID forwarded in env-passthrough.ts; GH_AW_OTLP_ENDPOINTS, OTEL_EXPORTER_OTLP_ENDPOINT, plus trace/span IDs forwarded in api-proxy-env-config.ts
4. Token Tracker Integration onUsage callback present in token-tracker-http.js as the OTEL hook point
5. OTEL Diagnostics ⚠️ No otel.jsonl span file found under api-proxy-logs/ this run (no OTLP endpoint configured for this workflow invocation) — this is expected graceful degradation. token-usage.jsonl shows 19 records tracked normally, confirming the proxy operated correctly without OTEL export enabled.

Overall: ✅ Pass — module, tests, wiring, and hook points all validated; no unexpected errors or regressions. Span export path itself was not exercised because no OTLP collector endpoint was configured for this run (expected during development / not yet enabled here).

📡 OTel tracing validated by Smoke OTel Tracing
Add label ready-for-aw to run again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[awf] repo-memory: mount is read-only inside AWF sandbox, blocking persistence of audit findings

3 participants