Skip to content

fix: make turns, shutdown, and model requests recoverable - #384

Merged
Haz3-jolt merged 3 commits into
Observal:mainfrom
Haz3-jolt:fix/turn-completion-recovery
Sep 6, 2026
Merged

fix: make turns, shutdown, and model requests recoverable#384
Haz3-jolt merged 3 commits into
Observal:mainfrom
Haz3-jolt:fix/turn-completion-recovery

Conversation

@Haz3-jolt

@Haz3-jolt Haz3-jolt commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Purpose

Fix four dogfood failures that could stop long tool turns, leave the TUI in a stale Working state, strand an incompatible daemon after upgrade, or dispatch model requests without explicit output and transport-idle configuration.

Fixes

Fixes #39
Fixes #381
Fixes #382
Fixes #383

Approach

  • Remove the numerical per-turn model-call ceiling and preserve cancellation through normal terminal events.
  • Synchronize TUI activity from the SDK projection so observers recover after errors, cancellation, and long tool loops.
  • Separate /detach from /quit. Quit closes admission, interrupts active work, drains accepted request outcomes and canonical history, shuts down the daemon, and exits. Shared sessions and clients require a generation-bound confirmation.
  • Add independently versioned host control for trusted process hosts, plus explicit axl daemon status, stop, and restart actions. Automatic version-mismatch replacement remains forbidden, and force termination is available only after graceful shutdown starts or fails.
  • Default ordinary model calls to the advertised model output maximum, clamp to estimated available context with a 4,096-token reserve, and send the effective ceiling explicitly.
  • Preserve selected reasoning effort and reserve 1,024 answer tokens when fitting explicit thinking-token budgets.
  • Add a configurable five-minute HTTP header/body idle timeout through Undici. Response bytes, including SSE heartbeats, refresh it. Zero disables it. There is no absolute request or turn deadline.
  • Record selected and effective request configuration canonically, expose it through the SDK projector, and show or change it with /request and /status.
  • Advance the exact session wire to version 11. Host-control remains independently versioned at 1.
  • Fix imported operationless canonical events so new configuration events remain portable.

Security impact: lifecycle authority remains in trusted process hosts. Renderers receive only typed intent. Shutdown never signals a PID read from storage, never replaces an incompatible daemon automatically, and fails loudly when graceful cleanup cannot complete.

How was this tested?

  • pnpm check: passed. 590 tests passed and 8 environment-dependent integration tests skipped. Skips were Podman unavailable, the pinned Docker image unavailable, and Seatbelt unavailable on Linux.
  • Focused protocol, AI, kernel, daemon, SDK, CLI, and TUI tests: passed, including 60-round turns, cancellation, state recovery, shutdown races, incompatible daemons, stale sockets, import/replay, context fitting, reasoning answer room, HTTP header/body inactivity, heartbeat refresh, disabled timeout, and subsequent prompts.
  • pnpm audit --audit-level high: passed with no known vulnerabilities.
  • git diff --check: passed.
  • pnpm check:boundaries: passed.
  • pnpm check:generated: passed.
  • reuse lint over each staged tree: passed. The final tree reports 322 of 322 files compliant.
  • Live Azure OpenAI smoke test with gpt-5.6-luna, low reasoning, built artifacts, Bubblewrap, and an isolated daemon/workspace: passed 52 tool rounds, cancellation recovery, provider-error recovery, and subsequent prompts.
  • Live lifecycle smoke test: /quit cancelled a Bubblewrap tool, persisted an aborted outcome, released the socket, and a fresh daemon resumed the session and completed another Azure request.
  • Final live request-configuration smoke test: the default effective output ceiling was the advertised 128,000 tokens, an explicit 2,048-token ceiling was honored, the setting survived daemon restart, another Azure request completed, sandbox enforcement remained active, canonical history contained no credential, and the saved credential file was unchanged.
  • An initial local full run hit one transient descendant-process abort assertion. The focused rerun passed, and subsequent full pnpm check runs passed.
  • The first GitHub Build and test run exposed an interrupt-admission race: optimistic Working could appear before daemon operation ownership, so immediate Escape could arrive early. The shared TUI interrupt path now preserves intent across that window. The focused Escape test passed 20 consecutive runs, shell and compaction Escape tests passed, and the full suite passed afterward.
  • macOS and Seatbelt were not available on this Linux host.

Learning

Pi at badlogic/pi-mono@6c87d9a02 was the behavioral reference for output fitting, answer reservation, byte-idle HTTP timeouts, and interactive shutdown. Axl implements the behavior independently within its daemon, protocol, SDK, process-host, and thin-client boundaries.

Checklist

  • I reviewed the complete diff.
  • I added or updated the smallest relevant test for behavior changes.
  • I ran the relevant formatting, lint, type-check, test, boundary, and license checks.
  • Every new file has SPDX metadata, directly or through REUSE.toml.
  • Every commit has a matching DCO Signed-off-by trailer.
  • UI changes include screenshots attached to the pull request, not committed to the repository.

No screenshot is attached. The UI changes are terminal command output and confirmation behavior covered by deterministic virtual-terminal tests.

Licenses

Resource Source and version License Use
Pi https://github.com/badlogic/pi-mono/tree/6c87d9a02 MIT Read-only behavioral reference; no source copied or translated line by line
Undici https://github.com/nodejs/undici, 8.10.2 MIT Model-only HTTP transport with header and body idle timeout support

AI assistance

  • Generative AI materially assisted this change. Tool and model/version: pi coding agent; model metadata was unavailable in the session environment.
  • I manually reviewed, understood, and tested the generated work.

Allow model turns to continue without an arbitrary call-count limit. Derive TUI activity from the SDK projection and record a terminal aborted message after tool cancellation without another model request.

Add long-turn and multi-attachment recovery regressions. Verified 52 sequential tool rounds against Azure gpt-5.6-luna in Bubblewrap, plus interruption and provider-error recovery.

Fixes Observal#383
Fixes Observal#381

Signed-off-by: Hari Srinivasan <harisrini21@gmail.com>
Separate quit from detach, coordinate cancellation and durable cleanup, and add independently versioned host control with safe CLI lifecycle actions.

Fixes Observal#382

Refs Observal#107

Signed-off-by: Hari Srinivasan <harisrini21@gmail.com>
Use advertised model output limits, fit requests to available context, preserve answer room for thinking budgets, and record effective request settings. Add configurable byte-idle transport timeouts without an absolute turn deadline.

Fixes Observal#39

Signed-off-by: Hari Srinivasan <harisrini21@gmail.com>
@Haz3-jolt
Haz3-jolt force-pushed the fix/turn-completion-recovery branch from 55a3467 to d52faf5 Compare September 6, 2026 10:24
@Haz3-jolt
Haz3-jolt merged commit 48fd229 into Observal:main Sep 6, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant