Skip to content

feat(runtime): host-side entrypoint log tailer for Cloud Run sandboxes - #1325

Open
ptone wants to merge 4 commits into
mainfrom
scion/sn-tailer-dev
Open

feat(runtime): host-side entrypoint log tailer for Cloud Run sandboxes#1325
ptone wants to merge 4 commits into
mainfrom
scion/sn-tailer-dev

Conversation

@ptone

@ptone ptone commented Aug 28, 2026

Copy link
Copy Markdown
Owner

Summary

Implements the host-side entrypoint log tailer for Cloud Run sandboxes (#112). One goroutine per sandbox tails .scion-entrypoint.log and re-emits each line to the host process stdout as GCP-structured JSON. Cloud Run captures host stdout and forwards it to Cloud Logging automatically. No client library, no service account, no new module dependency.

This makes startup failures visible. It does not make runtime failures visible. Silence from this channel after a successful startup means "init completed and handed off to tmux", not "the agent is healthy."

Key behaviors

  • Stat-before-launch offset: captures file size before sandbox creation; skips prior-run content when PR 1323 (append-mode) is present
  • Partial final line flush: on any terminal exit of the goroutine, the unterminated buffer is emitted with partial:true. The crash message that names the cause of death is never swallowed
  • File-never-created detection: if watchCtx is cancelled and the file was never opened, emits one ERROR with file_never_created:true — the only way to distinguish "died before it could say anything" from "started fine and went quiet"
  • Truncation detection: if file size < expected offset (PR 1323 reverted or file replaced), emits WARNING with truncation_detected:true and reads from start — duplicated output with an explanation, not silence
  • Buffer cap: 64KB max partial-line buffer; lines exceeding it emit with truncated_line:true
  • Backoff: 250ms x10, 1s x10, then 5s; resets to 250ms on new data (including on truncation detection)

PR 1323 interaction

Works with or without PR 1323 (append-mode entrypoint redirect). With it: clean per-run separation. Without it: truncation warning fires and still ships the output. Degradation is visible, not silent.

Design deviations

openWithRetry (F1): the design says persistent non-ENOENT open errors should emit ERROR and exit after the retry schedule is exhausted. This implementation keeps polling with WARNING every 5s instead. Rationale: noisy beats absent. A permission error will complain visibly until the sandbox is torn down (watchCtx cancelled). The error may be transient, and exiting permanently silences the channel on an assumption. watchCtx provides the bounded lifetime regardless. Documented in a comment on openWithRetry.

Test coverage limitation

Exit path 2 (non-EOF read error in the read loop) is defensive code, reachable only through infrastructure-level errors such as EIO from disk corruption. TestTailer_ReadErrorExitPath2 proves the path is reachable using a real EISDIR syscall error and that the goroutine exits cleanly, but does not exercise the flushPartial call at that site. EISDIR fires on the first Read before any bytes enter the line buffer, so flushPartial is called with an empty buffer (a no-op). Getting content into the buffer AND triggering a non-EOF read error on the same fd would require io.Reader injection, which was ruled out as a refactoring change on an already-reviewed read loop. The flushPartial function itself is mutation-tested by TestTailer_FlushPartialUnit.

Rendering note

This PR captures entrypoint log output to Cloud Logging. The captured lines do not yet appear in the operator's /cloud-logs tab — the logquery filter whitelists specific logName values and the tailer's output arrives under Cloud Run's platform stdout logName. Rendering is a separate change to logquery.go and is not in scope for this PR.

Tunable constants (not measured against production)

  • 64KB buffer cap: generous; longest plausible line is a debug.Stack() trace (~4-8KB)
  • Backoff step counts (10/10): the shape (fast, slow, plateau, reset on data) is load-bearing; counts are arbitrary
  • 8s open-retry window: bounded by liveness-probe ladder reasoning (250+500+1000+2000+4000 ms)

Files changed

  • pkg/runtime/cloudrun_sandbox_tailer.go — new file: tailEntrypointLog goroutine and helpers
  • pkg/runtime/cloudrun_sandbox_tailer_test.go — new file: 14 unit tests against real temp files
  • pkg/runtime/cloudrun_sandbox_runtime.go — stat-before-launch (L743-758) and goroutine launch (L885)
  • pkg/runtime/cloudrun_sandbox_runtime_test.go — cleanup for watchCtx in existing Run test

Test plan

  • Normal append from offset, prior content not re-shipped
  • Unterminated final line emitted with partial:true
  • Truncation mid-tail: WARNING emitted, reading resumes from 0
  • File never created + context cancelled: file_never_created ERROR emitted exactly once
  • Context cancellation with partial line pending: partial emitted, clean exit
  • Buffer cap exceeded: truncated_line:true, no unbounded growth
  • Goroutine exits on cancellation, no leak
  • First run (no prior file): offset 0, picks up content when file appears
  • Truncation at startup (offset > file size): WARNING + content from start
  • Live append: tailer picks up new content as it is written
  • JSON format matches GCP structured logging with logging.googleapis.com/labels
  • Empty lines skipped
  • Exit path 2 reachable via EISDIR: goroutine exits cleanly on real read error
  • flushPartial unit test: emits partial:true when buffer non-empty, no-op when empty
  • Race detector clean
  • gofmt clean
  • go vet clean
  • Existing package tests all pass

Scion Agent (sn-tailer-dev) added 4 commits August 28, 2026 19:21
…boxes (#112)

Ship sandbox startup output to Cloud Logging by tailing the entrypoint
log and re-emitting each line to host stdout as GCP-structured JSON.
Cloud Run captures host stdout automatically — no client library, no
service account, no new dependency.

Key behaviors:
- Stat-before-launch offset: skips prior-run content when PR 1323
  (append-mode) is present.
- Partial final line flush: on any terminal exit the unterminated
  buffer is emitted with partial:true — the crash message is never
  swallowed.
- File-never-created detection: if the sandbox terminates without
  creating its entrypoint log, emits a single ERROR with
  file_never_created:true.
- Truncation detection: if file size < offset (PR 1323 reverted or
  file replaced), emits WARNING and reads from start.
- Buffer cap: 64KB max; lines exceeding it emit with
  truncated_line:true.
- Backoff: 250ms x10, 1s x10, then 5s; resets on new data.

Tunable constants (not measured against production):
- 64KB buffer cap (generous; longest plausible line is ~4-8KB)
- Backoff step counts (shape is load-bearing, counts are arbitrary)
- 8s open-retry window (bounded by liveness-probe ladder reasoning)
Finding 1: openWithRetry now exits with ERROR after retry schedule is
exhausted for persistent non-ENOENT errors (permission denied, I/O),
instead of polling indefinitely. ENOENT continues polling at 5s as
designed (sandbox may be slow to start).

Finding 2: eofCount resets when truncation is detected during the
EOF backoff phase, so fresh content after truncation is read at
250ms instead of the plateau 5s interval.

Finding 3: Added TestTailer_FileDeletedMidTailPartialFlush covering
the file-deletion exit path with buffered partial content. Note: on
Linux, os.File.Read on a deleted file returns EOF (not ENOENT) due
to Unix fd semantics, so the test exercises the cancel path which
is the practical mechanism on POSIX.

Finding 4a: emitCompleteLines now uses bytes.IndexByte instead of
manual byte scan — idiomatic Go, assembly-optimized.
Finding 4b: Removed tailerWriter type alias; parameter uses io.Writer
directly.
Finding 4c: Test cleanup sleep acknowledged as acceptable.
… F3 test

F1 (openWithRetry design deviation): REVERTED the behavioral change that
exited on persistent non-ENOENT errors. Restored infinite polling with
WARNING — noisy beats absent, and watchCtx provides bounded lifetime.
Added comment documenting this as a deliberate deviation from the design
and why.

F2 (backoff reset on truncation): Already fixed in prior commit — eofCount
resets when truncation is detected so fresh content is read at 250ms.

F3 (ENOENT mid-tail partial flush test): The read-error exit path is
unreachable on Linux (os.File.Read on a deleted file returns EOF, not
ENOENT — Unix fd semantics keep the inode alive). Integration test
exercises the practical path (file delete + context cancel). Added
TestTailer_FlushPartialUnit to directly prove flushPartial correctness
independent of the trigger mechanism.

F4 (style nits): bytes.IndexByte and io.Writer changes kept from prior
commit — already applied, not worth reverting.
Remove the errors.Is(readErr, fs.ErrNotExist) sub-branch from exit
path 2. read() operates on a file descriptor, not a path — it never
returns ENOENT. File deletion is invisible to the reader on POSIX
(the fd keeps the inode alive via the open descriptor). The branch
was not dead weight but a false affordance: a reader sees it and
concludes the deletion case is handled; it is not handled, it cannot
occur, and the code asserts otherwise. Dead code that describes an
impossible world is worse than absent code, because it answers a
question nobody then asks again.

Add TestTailer_ReadErrorExitPath2: drives exit path 2 end-to-end
using the EISDIR trick — os.Open on a directory succeeds, Read
returns EISDIR (a real non-EOF syscall error, same shape as EIO
from disk corruption). Proves exit path 2 is reachable and the
goroutine exits cleanly without hanging or requiring context
cancellation.

Limitation documented in the test: EISDIR fires on the first Read
before any bytes enter lineBuf, so flushPartial is called with an
empty buffer (a no-op). The test does not go red if flushPartial is
deleted. Getting content into lineBuf AND triggering a non-EOF read
error on the same fd requires io.Reader injection (refactoring the
read loop), which was ruled out for this PR. flushPartial itself is
mutation-tested by TestTailer_FlushPartialUnit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant