feat(terminal): kitty graphics state, replay, child input encoding, and an honest scrollback

[schickling-assistant]

Problem

The session's terminal has no image state and no child input encoder, and both
gaps push work onto consumers that consumers cannot do correctly.

Images are lost by the replay. A child can transmit a kitty image and place
it, and a client connected at that moment sees the bytes. A client that
attaches afterwards does not: the SCREEN payload keeps the placeholder cells
(they are ordinary text with a foreground colour) and nothing else. On main,
with a child that transmitted a 1x1 PNG as image 4242 and placed it virtually:

attach after the write, then ask what the session holds
  images:      —          (no API exists)
  placements:  —          (no API exists)
  screen:      "drawn" + U+10EEEE cell naming image 4242

The placeholder cell survives and names an image nobody has. No API can even
report that, because there is no image state to report.

The 10,000-line scrollback promise was 7% true. libghostty's
Options::max_scrollback is documented as "maximum number of lines to keep in
scrollback history" and is in fact a byte budget for the history page list.
Passing 10_000 for "10,000 lines" bought one page. Fed 10,008 short,
non-wrapping lines on main:

columns	rows retained	oldest line surviving
20	3,310	L6698
80	745	L9263
200	456	L9552
400	149	L9859

Retention scales inversely with the width, which a line count cannot do, and
doubling the number to 20,000 changes nothing at 80 columns — both are smaller
than one page, and one page is the floor. scrollback_capacity() meanwhile
reported 24 + 10_000, describing nothing that existed.

Input encoding has nowhere correct to live. What bytes a key becomes
depends on modes the child set — DECCKM, the keypad mode, modifyOtherKeys,
the kitty keyboard flags — and what a mouse event becomes depends on the
tracking mode and report format. Modes did not even track ?1 (DECCKM) or
?9 (X10), so a consumer could not encode an arrow key or tell press-only X10
from wheel-reporting ?1000. The alternative is a second encoder outside the
terminal, which means a second implementation of the kitty keyboard protocol.

Goal

• Embedded graphics are queryable as durable image bytes plus placement
  identity, source crop, and position — after a live write, and after a replay
  or reconnect, from a client that never saw the original DATA.
• A terminal asked for 10,000 lines of history retains 10,000 lines, and the
  numbers it reports describe the terminal rather than the request.
• Nothing the terminal accepts can be unreachable by a late client.
• One child input encoder, inside the terminal, fed semantic events.
• Cell pixel metrics come from whoever draws the cells, not from a guess.

Decisions

Graphics live in the terminal, and the replay carries them. Two other
shapes were considered and rejected. Passing the child's graphics bytes through
to an outer terminal cannot work for an embedder that draws a sub-rectangle:
the child's coordinates are its own, and the embedder clips, pans, and draws
chrome around it. Keeping the state per client cannot work either, because a
late client never sees the DATA. Re-emitting the storage in the protocol the
child used is the smallest thing that makes a late client and a live client
hold the same images.

One bound for the state and the wire. graphics::MAX_STORAGE_BYTES
(32 MiB) is clamped to by enable_graphics / set_graphics_storage_limit, and
a replay carries all of it. A separate, smaller replay cap was tried first and
was wrong: it made a supported state — an image the terminal accepted and
reports through image_bytes — one that a late client could never be given.

Virtual placements are located by their cells. libghostty reports no
viewport position for a virtual placement, correctly: it has none, it is
wherever its placeholder cells are. Those cells each name their own image row
and column, so graphics decodes them from the grid (one bounded pass over the
requested window, never the scrollback). That is what makes a partially
scrolled image answerable, and it is also why a virtual placement can be
replayed regardless of where it currently shows.

Positions resolve against the window that was asked for. Direct placements
use the placement's own rectangle in screen space rather than
viewport_pos, which answers only for the live viewport. Otherwise a
scrolled-back reader would see virtual images and lose cursor-positioned ones.

Input events are dumb; encoding is not. input::KeyEvent carries the
logical key, the associated text, and the unshifted codepoint separately,
because the kitty protocol reports them separately — folding shift into one
character upstream loses the shifted alternate. The encoders themselves are
libghostty's, configured from the live terminal. Key/Mods/actions are
re-exported rather than re-declared: a second key enum would be a translation
table that silently drifts.

Cell metrics travel on the wire as an optional suffix. Four bytes appended
after the existing rows/cols on ATTACH and RESIZE. Every reader of a size
payload takes rows and cols from the first four bytes and the frame carries its
own length, so an older daemon — the Node one included — reads exactly the size
it always read. Undeclared is explicit (GraphicsState::cell_declared), with a
deterministic 8x16 fallback, rather than a silent guess.

The scrollback line count is converted, not passed through. The
alternative was to weaken the promise to "up to N lines" — which is what an
assertion of used <= capacity amounts to, since it passes when 6,398 of
10,008 lines have been thrown away. A replay window is a product promise: a
consumer decides what history to show on the basis of that number, so the
honest options were to meet it or to publish a smaller one. The conversion is
256 + 16 * cols bytes per line against a measured ~838 bytes per row at 80
columns and ~1,804 at 200 — about 1.8x headroom, because the page list rounds
up to whole pages and a styled or multi-codepoint row costs more than a plain
one — capped at 64 MiB. libghostty exposes the budget only in Options, with
no setter, which is the source of the one residual gap (see Concerns).

Modes gains ?1 and ?9 as separate facts. Neither joins
mouse_tracking() (X10 reports no wheel, and a consumer must be able to tell
that from no tracking at all) and neither enters the Node-parity mode prefix.

Verification

nix flake check — the required gate, which builds the package and runs the
workspace suite:

✅ checks.x86_64-linux.pty
✅ checks.x86_64-linux.completions
✅ checks.x86_64-linux.extension-forwarding
✅ checks.x86_64-linux.help

Focused suites, 420 passing, 0 failing across pty-terminal and pty-core
(graphics 26, handle 10, input 13, replay 21, scrollback 6,
protocol 27, rest unchanged):

cargo test -p pty-terminal -p pty-core     420 passed; 0 failed
cargo check -p pty -p pty-tui -p pty-testkit -p pty-conformance    0 errors

Scrollback, tests/scrollback.rs — 10,008 short non-wrapping lines into a
24x80 terminal with the default 10,000-line scrollback, so nothing depends on
reflow:

before   rows retained 745        oldest surviving line L9263
after    rows retained 10009      oldest surviving line L0

The same at every width the per-row cost differs across
(the_promise_holds_at_every_width):

          before   after
20 cols    3310    10009
80 cols     745    10009
200 cols    456    10009
400 cols    149    10009

The behaviour change, end to end against a real session daemon
(tests/handle.rs::a_late_attach_gets_the_image_the_child_drew_before_it_connected):
a child transmits a PNG, places it virtually, writes a placeholder cell; a
handle attaches 400 ms later, so it never sees that DATA.

before   image bytes:  —                     (no state to ask)
         placement:    —
         screen:       placeholder cell naming image 4242
after    image bytes:  [255, 0, 0, 255]      decoded, from the SCREEN alone
         placement:    image 4242, placement 7, virtual, at its cell
         reconnect:    same bytes, same cell

Rebuilding the daemon without its enable_graphics call makes exactly that
test fail with "the replay carried no placement", which is the negative control
for the daemon-side ownership.

Cell geometry, same rig
(a_client_declares_its_cell_size_and_the_session_geometry_follows) — a 16x16
image placed with no c=/r=:

declared 16x16 cell     cell_size (1, 1)
declared 8x16 cell      cell_size (2, 1)     via RESIZE, on the session's own terminal

Input encoding, from tests/input.rs:

arrow up, DECCKM off    ESC [ A
arrow up, DECCKM on     ESC O A
shift+a, kitty 17u      ESC [ 97;2;65u       base key, shift, associated text "A"
shift+a, kitty 5u       …97:65…              base:shifted alternate
wheel, no tracking      None                 the surface keeps it
wheel, ?9 (X10)         None                 press-only mode reports no wheel
wheel, ?1000            ESC [ M ` $ %
press, ?1006            ESC [ <0;4;5M

Replay fidelity has a case per failure mode it can have, each of which fails on
the shape that preceded it: crop preserved through a replay, virtual placement
scrolled into history still replayed, partially scrolled direct placement
replayed clipped, grayscale PNG stored as RGBA, palette foreground recognised,
bare continuation cells inheriting row and id high byte, storage limit raising
that keeps the cell size and the PNG decoder, and the normal screen's images
landing on the client's normal screen after a full-screen program exits.

Complexity

Two new modules in an existing crate, one new dependency, one protocol suffix.

• graphics.rs is large because the protocol is: image storage, placements,
  crop resolution, placeholder-cell decoding (the diacritic table is 297
  entries of data), and replay emission. It has no abstraction over
  libghostty — it is the boundary that makes libghostty's borrowed handles
  owned and Send.
• png is a new dependency. libghostty rejects PNG transmissions without a
  decoder, and its own RustPngDecoder is unusable (no constructor, and its
  buffer is reserved but never sized). Most senders use f=100.
• The cell-size protocol suffix adds four bytes to two message types and no
  new message type.

Concerns

• A session can now hold up to 32 MiB of images. The limit is a cap, not an
  allocation — a child that never transmits holds nothing — but a child that
  wants to can make its session hold that much, and a replay then carries it.
  That is the deliberate cost of a lossless replay; the alternative was
  dropping images silently.
• The graphics block widens the SCREEN payload for graphics sessions. It
  is empty for a session that never sent an image, so non-graphics sessions are
  byte-identical to before (pinned by test).
• Cell metrics are newest-declaration-wins, not negotiated. Two clients
  with different fonts cannot both be right about an implicit placement, and
  unlike rows and cols there is nothing to reconcile — the metrics change no
  bytes and no client's screen. Recorded in the decision record.
• History memory per session rose from ~1 MiB of budget to 14.6 MiB at 80
  columns (33 MiB at 200, 63.5 MiB at 400), capped at 64 MiB. The budget is
  address space the page list fills only as history accumulates, so an idle
  session pays nothing — but a fleet of sessions that all fill their history
  now costs what a 10,000-line window actually costs. That is the price of the
  promise being true; the alternative was publishing a smaller window.
• Widening a terminal lowers the line count it retains. libghostty takes
  the byte budget in Options and exposes no setter, so a terminal created at
  80 columns and widened to 400 holds about a fifth of the lines.
  scrollback() and scrollback_capacity() follow it down and
  scrollback_request() still reports what was asked for, so the number is
  never a lie — but the gap is real. Closing it needs either a libghostty
  setter (upstream) or budgeting for the widest plausible width up front
  (10,000 lines at 1,000 columns is 154 MiB per session, which is not worth
  it). Called out as a follow-up.
• scrollback_capacity() is now a floor, not a ceiling. libghostty never
  holds less than one page, so a small scrollback retains more than it
  promised (a 100-line request at 80 columns keeps ~1,000 rows). Node's number
  is a ceiling. Code that treated it as an upper bound on scrollback_used
  has to stop.
• pty-terminal is doing more than it did. Graphics state and input
  encoding are both genuinely terminal state, but the crate is now the place
  where a consumer's whole terminal contract lives.
• Overlap with open PRs. feat: persist PTY output activity evidence #5 and feat(gc): expire keep-tagged exited sessions after a max age (parity with pty#173) #12 also touch crates/pty/src/daemon/;
  this PR's daemon changes are lifecycle.rs::terminal_actor and
  clients.rs::adopt_cell_size. A textual conflict is possible, semantically
  independent.

Friction & bottlenecks

• libghostty_vt::kitty::graphics::RustPngDecoder cannot be constructed
  (private field, no new, no Default) and would not work if it could — it
  reserves capacity for the decode buffer but never sets its length, so
  next_frame gets an empty slice. Worked around with a local decoder.
• png::Transformations::ALPHA is documented as expanding paletted images,
  and there is no grayscale-to-RGB transformation at all, so
  EXPAND | ALPHA | STRIP_16 does not produce RGBA for a grayscale PNG the way
  its name suggests. The expansion has to be done by hand.
• libghostty's viewport_pos and its placement rect answer in different
  coordinate spaces, and only the latter can answer for a window other than the
  live viewport. Not documented as a choice; found by testing.
• libghostty_vt::terminal::Options::max_scrollback is documented as a line
  count and is a byte budget. The discrepancy is invisible until you count
  retained rows at two different widths, and it silently degrades any consumer
  that trusts the doc comment. Worth an upstream report.
• crates/pty/tests/daemon_geometry.rs has 3 failures on this machine that
  reproduce identically with this PR's daemon change reverted, so they are
  pre-existing and unrelated. nix flake check is green, so they appear to be
  environment-specific.

Follow-ups

• A libghostty setter for the history byte budget would let a resize keep the
  line promise; without it, a widened terminal retains fewer lines (see
  Concerns). Worth raising upstream together with the max_scrollback doc
  discrepancy.
• No gated Node/Rust conformance fixture for the cell-size suffix — the Node
  daemon's tolerance of it is argued from its decode offsets and the explicit
  frame length, not proven against the binary.
• SIXEL and the iTerm2 protocol are still not read. Neither offers a placement
  contract an embedder can reproject into a sub-rectangle, so a child using
  them still shows nothing.
• An image transmitted and never placed is not listed by GraphicsState:
  libghostty exposes lookup by id, not enumeration. Such an image is also not
  drawable.
• The 3 pre-existing daemon_geometry failures noted above deserve their own
  investigation.

References

Refs #3 — one embedded terminal API for attached and
spawned PTYs (this adds the graphics and input halves of that interface).
Refs schickling/dotfiles#1378 — the Fractal terminal-core convergence that
consumes it.
docs/vrs/01-images/requirements.md — the durable terminal-image contract.
docs/vrs/01-images/spec.md — the Kitty graphics mechanism and validation index.
docs/decisions/0012-kitty-graphics-replay.md — the graphics deviations, the
bounds, and the test index.
docs/decisions/0013-scrollback-is-a-line-promise.md — the scrollback
measurements, the conversion, the memory cost, and the residual gap.

Posted on behalf of @schickling

field	value
agent_identity	unknown
session	unknown
agent_persona	unknown
agent_supervisor	unavailable
agent_tool	unknown
agent_tool_version	unknown
agent_runtime	unknown
tooling_profile	dotfiles@7534055