Skip to content

Derive the start-service launch budget from agent configuration instead of hardcoded constants #13683

Description

@jopemachine

Goal

Replace the two hardcoded service-launch timeouts with a single operator-tunable value, so the agent and the kernel runner always derive their budgets from the same number instead of two constants that can silently drift apart.

Background

BA-7258 restored the ordering by raising the agent's reply timeout to a constant that exceeds the kernel runner's launch timeout. That is the minimal, backportable fix, but it leaves two problems open:

  • The budget cannot be tuned per deployment. Sites whose images bind slowly under CPU or IO contention have no knob.
  • The two constants live in packages that cannot import each other (ai.backend.kernel runs inside the container), so nothing prevents a future edit to one of them from re-inverting the pair.

The proposed shape, from the BA-7258 report: add a key to the existing [kernel-lifecycles] section of agent.toml, send it in the start-service payload, and have the kernel runner honour it. Older runners ignore the extra key, so this stays backward compatible.

Acceptance criteria

  • The launch budget is configurable through [kernel-lifecycles] in agent.toml and is etcd-overridable like its neighbours.
  • The agent and the kernel runner derive their budgets from that one value, with the agent's reply timeout strictly greater than the runner's launch timeout.
  • A regression test asserts the ordering so the pair cannot silently invert.
  • A start-service payload without the key keeps the pre-existing runner behaviour, so containers created before the upgrade are unaffected.
  • The generated configs/agent/sample.toml documents the new key.

Notes

  • The kernel runner terminates the service process when its launch timeout expires, so lowering the runner's budget is a behaviour change, not just a tuning knob. Pick the default with that in mind.
  • The kernel runner ships bind-mounted from the agent's own venv, so no image rebuild is needed, but the mount is fixed at container creation: the change only reaches newly created containers.
  • The effective ceiling on the agent's side is the client's own request timeout, which is outside this repository.

Related

  • BA-7258 — the minimal fix this builds on (backported to 26.8 and 26.4).
  • BA-7260 — start-service replies are not correlated with requests; touches feed_start_service in the same file.

Story points

1

JIRA Issue: BA-7322

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions