Skip to content

GUEST_READY_TIMEOUT is a hard-coded 30s with no override #1111

Description

@DorianZheng

Summary

GUEST_READY_TIMEOUT is a hard-coded 30 s with no environment or config override, so a box
that boots slowly on a contended host fails permanently with no available mitigation:

Box uPKTB5hOHtZf failed to start: timeout after 30s

Evidence at T+30s:
• shim_alive          = true
• console_bytes       = 0
• ready_socket_exists = true
• likely_cause        = guest agent never wrote to console (init or vsock plumbing broken)

src/boxlite/src/litebox/init/tasks/guest_connect.rs:110:

const GUEST_READY_TIMEOUT: Duration = Duration::from_secs(30);

It has exactly two references — the definition and the call site at :85. There is no env var,
no AdvancedOptions field, and no per-box knob, so a caller on a loaded machine has no way to
raise it. The doc comment says it is "exposed as a constant so tests can call
wait_for_guest_ready with a short timeout", i.e. tests can shorten it but production callers
cannot lengthen it.

How it showed up

Running the full Python SDK integration suite on macOS (make test:integration:python, which
the pre-push hook runs), tests/test_tcp_filter.py::TestEdgeCases::test_multiple_allowed_hosts
intermittently fails with the message above. It failed 3 times during full-suite runs.

Reproduction attempts — what does and does not trigger it

I could not reproduce it in isolation. Recording the negative results so the next person does
not repeat them:

Condition Result
Standalone, idle machine (3 consecutive runs) pass — 15.9 s, 15.6 s, 17.6 s
Standalone, 24 CPU spinners (load avg 20.6 / 18 cores) pass — 28.4 s
Standalone, 40 CPU spinners + 4 dd I/O loops (load avg 52.9) pass — 69.2 s
Whole test_tcp_filter.py file, 21 boxes back to back pass — 21/21, 179 s
During / immediately after the full integration suite fails, 3 occurrences

Two things worth noting:

  • Wall time degrades sharply with host load (15.9 s idle → 28.4 s → 69.2 s), so box startup is
    clearly contention-sensitive; but pure CPU/IO pressure alone did not push the guest-ready
    handshake
    past 30 s here.
  • The failure signature is console_bytes = 0 — the guest wrote nothing, which does not look
    like ordinary slowness. Something about running the whole suite (hundreds of boxes) is the
    trigger; I did not isolate what.

Why the constant is worth fixing regardless

Whatever the underlying trigger turns out to be, a user or CI job that hits it today has no
escape hatch: they cannot raise the deadline to confirm "slow" versus "hung", and the same
30 s applies to a warm laptop and a heavily loaded CI box alike.

Suggested: make it overridable (env var and/or an AdvancedOptions field) with the current
30 s as the default, keeping the existing diagnostic block on expiry.

Related

Environment

  • macOS 27.0.0, arm64, 18 cores
  • boxlite 0.9.7, local (non-REST) runtime, sdks/python built from source

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingrustPull requests that update rust code

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions