feat(display): capture the device's own screen with capture_display

[jamesarich]

Adds capture_display — a screenshot of the device's own screen, read from the framebuffer the firmware is rendering instead of photographed off the panel with a webcam.

This is the on-device counterpart to capture_screen. The pixels are exact rather than a camera's view of a lit LCD, which also makes OCR reliable, and it needs no camera rig — the bench's webcam becomes optional for "what is on the screen?" questions.

Draft: needs unreleased firmware

The firmware side is meshtastic/firmware#11681 and the wire contract is meshtastic/protobufs#1054, neither merged. Older firmware ignores the get_display_frame_request admin verb silently, so the tool times out with a message naming the requirement rather than looking like a hung device. Marking ready once those land.

Same reason the admin verb and the DisplayFrame payload are hand-encoded and hand-parsed: the released meshtastic package's generated bindings do not carry these fields yet. display_mirror.py is written so those helpers can be swapped for the generated messages with no change to the tool surface.

What it does

Handles both wire formats:

• MONO_VLSB — BaseUI's 1bpp framebuffer, chunked by offset.
• RGB565 — MUI (device-ui/LVGL) dirty rects, composited onto a persistent canvas.

PNG encoding is pure stdlib, so the core stays dependency-light — no Pillow.

Arm, request, disarm. The one-shot get_display_frame_request looked preferable — the firmware clears it once the frame drains, so it cannot leave the device streaming — but on the MUI (LVGL) path it does not reliably produce a frame. On a T-Deck, arm-plus-request returns a rect every time where request-alone times out, minutes apart on the same device. The firmware source says when the one-shot flag is cleared, not whether a repaint is ever produced. So this arms, requests, and disarms in a finally; PhoneAPI::close() disarms again when the connection drops, so an abandoned call cannot leave the device armed.

Two related details, both found only on hardware: connect() yields the instant the interface is constructed, and an admin message sent into the tail of the config handshake is silently dropped — so the capture settles before speaking, and re-sends if nothing arrives. The retry re-sends both messages, since retrying the request alone reproduces exactly the sequence that does not work.

Collection continues until the stream goes quiet, because a full repaint arrives as several rects — nine for the T-Deck home screen — and stopping at the first completion captures a fragment.

Blank captures are labelled. A panel that has hit display.screen_on_secs mirrors as one flat colour, which otherwise reads as a mirroring failure. The result carries blank plus the device's own screen_on_secs so the caller knows to wake it with send_input_event and capture again. Blankness is tested as "all one colour", not "all black" — an inverted or e-ink panel blanks to white.

Validating device-controlled input

total_size and the rect geometry come off the wire, so every field is checked before anything is allocated:

• rect geometry is latched at offset == 0, so a later chunk cannot transpose an in-flight blit;
• a continuation must match the buffer actually allocated, not merely its own total_size claim;
• bounds use the subtraction form (rect_x <= width - rect_width) because the addition overflows for hostile values;
• rect_width/rect_height of 0 means "the whole panel in that axis", not "empty";
• mono page math rounds up ((h + 7) // 8), and the validator and renderer agree on it.

Each of these is pinned by a unit test.

Annotations

openWorldHint — the device's screen renders mesh-sourced text (a remote node's long_name, a received message), so a capture can carry a prompt-injection payload exactly like android_screenshot. destructiveHint matches android_screenshot too: it drives the device and overwrites the PNG at its fixed path.

Testing

• 38 new unit tests, all pure — no device, no serial port. Gates green: ruff check, ruff format --check, mypy (no new ignores), pytest tests/unit at 802 passed / 54 skipped.
• Hardware-verified on a LILYGO T-Deck running MUI (displaymode = COLOR) at firmware 2.8.0.07cfb8f: 320×240 RGB565, full colour, correct screen. The blank path was verified too — a timed-out panel returned blank: true with screen_on_secs: 30, and the same capture after a send_input_event wake returned the live screen.
• The palette path is hardware-verified too: switching the T-Deck to displaymode = DEFAULT (BaseUI on its colour panel) returns colorized: true on a MONO_VLSB frame, and the PNG carries 4 distinct colours with 75.8% non-greyscale pixels — a monochrome render would be exactly 2 colours and 0%.
• Repeat-verified after the arming fix: three consecutive MUI captures succeed where all three previously timed out.

Colour on both UIs

MUI streams RGB565 rects, so it is full colour directly. A colour BaseUI device instead streams its 1bpp frame plus the region palette its panel paints with (DisplayPalette, FromRadio tag 21), and that is now applied here — those devices render in their true colours at 1bpp bandwidth rather than black-and-white.

Region precedence follows the proto: a higher table index overrides lower-indexed regions where they overlap, and a pixel outside every region takes the palette defaults. colorized in the result reports whether a palette was actually matched and applied — a frame naming a signature we do not hold, or none at all, renders monochrome, as does a half-received region table (colouring some regions and not others would be worse than not colouring at all).

Mono rendering is deferred to the end of a capture rather than done per chunk, because the palette may arrive after the frame it colours — the proto allows display_palette to interleave.

Feature spec and cross-platform discussion: meshtastic/design#142.

---

Feature stack

Five repos, and the release order runs top to bottom. Nothing below can ship until the piece above it lands.

Repo	PR	Role
meshtastic/design	#142	Cross-platform feature spec
meshtastic/device-ui	#394	MUI flush observer + opt-in input injection
meshtastic/protobufs	#1054	DisplayFrame / DisplayPalette / DisplayInfo + the two admin verbs
meshtastic/firmware	#11681	Producer: streams the framebuffer, bridges input
meshtastic/Meshtastic-Android	#6987	Mirror tab with remote control
meshtastic/meshtastic-mcp	#78	capture_display tooling

device-ui#394 is the current blocker: without the flush observer the MUI/colour path compiles out of every committed firmware configuration, leaving only the 1bpp mono path.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}