feat(runtime): host-side entrypoint log tailer for Cloud Run sandboxes - #1325
Open
ptone wants to merge 4 commits into
Open
feat(runtime): host-side entrypoint log tailer for Cloud Run sandboxes#1325ptone wants to merge 4 commits into
ptone wants to merge 4 commits into
Conversation
added 4 commits
August 28, 2026 19:21
…boxes (#112) Ship sandbox startup output to Cloud Logging by tailing the entrypoint log and re-emitting each line to host stdout as GCP-structured JSON. Cloud Run captures host stdout automatically — no client library, no service account, no new dependency. Key behaviors: - Stat-before-launch offset: skips prior-run content when PR 1323 (append-mode) is present. - Partial final line flush: on any terminal exit the unterminated buffer is emitted with partial:true — the crash message is never swallowed. - File-never-created detection: if the sandbox terminates without creating its entrypoint log, emits a single ERROR with file_never_created:true. - Truncation detection: if file size < offset (PR 1323 reverted or file replaced), emits WARNING and reads from start. - Buffer cap: 64KB max; lines exceeding it emit with truncated_line:true. - Backoff: 250ms x10, 1s x10, then 5s; resets on new data. Tunable constants (not measured against production): - 64KB buffer cap (generous; longest plausible line is ~4-8KB) - Backoff step counts (shape is load-bearing, counts are arbitrary) - 8s open-retry window (bounded by liveness-probe ladder reasoning)
Finding 1: openWithRetry now exits with ERROR after retry schedule is exhausted for persistent non-ENOENT errors (permission denied, I/O), instead of polling indefinitely. ENOENT continues polling at 5s as designed (sandbox may be slow to start). Finding 2: eofCount resets when truncation is detected during the EOF backoff phase, so fresh content after truncation is read at 250ms instead of the plateau 5s interval. Finding 3: Added TestTailer_FileDeletedMidTailPartialFlush covering the file-deletion exit path with buffered partial content. Note: on Linux, os.File.Read on a deleted file returns EOF (not ENOENT) due to Unix fd semantics, so the test exercises the cancel path which is the practical mechanism on POSIX. Finding 4a: emitCompleteLines now uses bytes.IndexByte instead of manual byte scan — idiomatic Go, assembly-optimized. Finding 4b: Removed tailerWriter type alias; parameter uses io.Writer directly. Finding 4c: Test cleanup sleep acknowledged as acceptable.
… F3 test F1 (openWithRetry design deviation): REVERTED the behavioral change that exited on persistent non-ENOENT errors. Restored infinite polling with WARNING — noisy beats absent, and watchCtx provides bounded lifetime. Added comment documenting this as a deliberate deviation from the design and why. F2 (backoff reset on truncation): Already fixed in prior commit — eofCount resets when truncation is detected so fresh content is read at 250ms. F3 (ENOENT mid-tail partial flush test): The read-error exit path is unreachable on Linux (os.File.Read on a deleted file returns EOF, not ENOENT — Unix fd semantics keep the inode alive). Integration test exercises the practical path (file delete + context cancel). Added TestTailer_FlushPartialUnit to directly prove flushPartial correctness independent of the trigger mechanism. F4 (style nits): bytes.IndexByte and io.Writer changes kept from prior commit — already applied, not worth reverting.
Remove the errors.Is(readErr, fs.ErrNotExist) sub-branch from exit path 2. read() operates on a file descriptor, not a path — it never returns ENOENT. File deletion is invisible to the reader on POSIX (the fd keeps the inode alive via the open descriptor). The branch was not dead weight but a false affordance: a reader sees it and concludes the deletion case is handled; it is not handled, it cannot occur, and the code asserts otherwise. Dead code that describes an impossible world is worse than absent code, because it answers a question nobody then asks again. Add TestTailer_ReadErrorExitPath2: drives exit path 2 end-to-end using the EISDIR trick — os.Open on a directory succeeds, Read returns EISDIR (a real non-EOF syscall error, same shape as EIO from disk corruption). Proves exit path 2 is reachable and the goroutine exits cleanly without hanging or requiring context cancellation. Limitation documented in the test: EISDIR fires on the first Read before any bytes enter lineBuf, so flushPartial is called with an empty buffer (a no-op). The test does not go red if flushPartial is deleted. Getting content into lineBuf AND triggering a non-EOF read error on the same fd requires io.Reader injection (refactoring the read loop), which was ruled out for this PR. flushPartial itself is mutation-tested by TestTailer_FlushPartialUnit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the host-side entrypoint log tailer for Cloud Run sandboxes (#112). One goroutine per sandbox tails
.scion-entrypoint.logand re-emits each line to the host process stdout as GCP-structured JSON. Cloud Run captures host stdout and forwards it to Cloud Logging automatically. No client library, no service account, no new module dependency.This makes startup failures visible. It does not make runtime failures visible. Silence from this channel after a successful startup means "init completed and handed off to tmux", not "the agent is healthy."
Key behaviors
partial:true. The crash message that names the cause of death is never swallowedwatchCtxis cancelled and the file was never opened, emits one ERROR withfile_never_created:true— the only way to distinguish "died before it could say anything" from "started fine and went quiet"truncation_detected:trueand reads from start — duplicated output with an explanation, not silencetruncated_line:truePR 1323 interaction
Works with or without PR 1323 (append-mode entrypoint redirect). With it: clean per-run separation. Without it: truncation warning fires and still ships the output. Degradation is visible, not silent.
Design deviations
openWithRetry (F1): the design says persistent non-ENOENT open errors should emit ERROR and exit after the retry schedule is exhausted. This implementation keeps polling with WARNING every 5s instead. Rationale: noisy beats absent. A permission error will complain visibly until the sandbox is torn down (watchCtx cancelled). The error may be transient, and exiting permanently silences the channel on an assumption. watchCtx provides the bounded lifetime regardless. Documented in a comment on
openWithRetry.Test coverage limitation
Exit path 2 (non-EOF read error in the read loop) is defensive code, reachable only through infrastructure-level errors such as EIO from disk corruption.
TestTailer_ReadErrorExitPath2proves the path is reachable using a real EISDIR syscall error and that the goroutine exits cleanly, but does not exercise theflushPartialcall at that site. EISDIR fires on the first Read before any bytes enter the line buffer, soflushPartialis called with an empty buffer (a no-op). Getting content into the buffer AND triggering a non-EOF read error on the same fd would require io.Reader injection, which was ruled out as a refactoring change on an already-reviewed read loop. TheflushPartialfunction itself is mutation-tested byTestTailer_FlushPartialUnit.Rendering note
This PR captures entrypoint log output to Cloud Logging. The captured lines do not yet appear in the operator's
/cloud-logstab — the logquery filter whitelists specific logName values and the tailer's output arrives under Cloud Run's platform stdout logName. Rendering is a separate change tologquery.goand is not in scope for this PR.Tunable constants (not measured against production)
debug.Stack()trace (~4-8KB)Files changed
pkg/runtime/cloudrun_sandbox_tailer.go— new file:tailEntrypointLoggoroutine and helperspkg/runtime/cloudrun_sandbox_tailer_test.go— new file: 14 unit tests against real temp filespkg/runtime/cloudrun_sandbox_runtime.go— stat-before-launch (L743-758) and goroutine launch (L885)pkg/runtime/cloudrun_sandbox_runtime_test.go— cleanup for watchCtx in existing Run testTest plan
partial:truefile_never_createdERROR emitted exactly oncetruncated_line:true, no unbounded growthlogging.googleapis.com/labelsflushPartialunit test: emitspartial:truewhen buffer non-empty, no-op when empty