feat(debug): live device screen mirror with remote control (protobufs #1054) - #6987
Draft
jamesarich wants to merge 14 commits into
Draft
feat(debug): live device screen mirror with remote control (protobufs #1054)#6987jamesarich wants to merge 14 commits into
jamesarich wants to merge 14 commits into
Conversation
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Collaborator
Author
|
Cross-platform feature spec (client-agnostic wire contract, reassembly rules, input mapping, UX and device requirements): meshtastic/design#142 |
|
Consumes the DisplayFrame PoC protobufs (2.99.0-screen-mirror-poc-SNAPSHOT from mavenLocal, useMavenLocal committed on this branch): DisplayMirrorManager reassembles FromRadio.display_frame chunks into MirrorFrame snapshots, AdminController gains setDisplayMirror + sendInputEvent, and the Debug panel gains a Mirror tab rendering the device's 1bpp framebuffer live with a D-pad driving the firmware InputBroker. PoC: tab strings deliberately unlocalized. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cap the mirror canvas at 512.dp (4x for a 128px OLED) instead of filling the window width, fold the frame info into the toggle row, and pair Down/Back so the controls fit without scrolling on desktop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Manager: validate format (MONO_VLSB only) and geometry (width/height must describe exactly total_size bytes) so malformed device input can't reach aspectRatio and crash; unit tests for reassembly, drops, restarts. AdminController: setDisplayMirror/sendInputEvent become non-suspend immediate sends (teardown-safe, input never queues behind bulk admin) and gain requestDisplayFrame. Mirror tab: render each frame once into a 1:1 ImageBitmap scaled with FilterQuality.None (kills the fractional-scale seams), gate controls on connection state, reset the toggle on disconnect, stop the stream on tab dispose and ViewModel clear, add a one-shot Refresh, inject AdminController/ConnectionStateProvider instead of RadioController. Dispatch: display_frame test coverage incl. revoked-session. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Populates the new DisplayInfo during the handshake — dimensions from the live display device, MONO_VLSB frame format, PanelClass from the panel defines (EINK/HUB75/TFT/LCD/OLED), has_touch from HAS_TOUCHSCREEN — so clients can gate mirroring UI and adapt to the panel. Absent on display-less builds. Regenerates protos (DisplayInfo, DisplayFrame rect fields reserved for dirty-rect streaming). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the flat button stack with the researched control surface: cross-shaped 5-way cluster (56dp circular keys, 8dp gaps, Material Symbols chevrons with contentDescriptions), directions auto-repeat on hold (500ms then 10Hz — slow enough for the radio to render), OK long-press sends SELECT_LONG, Back sits below per TV-remote convention. The mirror itself becomes an input surface: click to capture the keyboard (arrows / Enter / Esc, with a visible focus ring and hint) and swipe in a cardinal direction to navigate. Consumes the new DeviceMetadata.display: shows an e-ink hint steering toward one-shot Refresh. Adds keyboard_arrow_left/right icons (Material Symbols rounded, transposed from the existing up glyph). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DisplayMirrorManager reassembles FromRadio.display_palette chunks (signature-keyed, defaults captured on the offset-0 chunk) into a MirrorPalette; frames carry palette_signature and the renderer resolves each pixel through the region table (last overlapping region wins, matching firmware precedence) with RGB565 conversion, falling back to monochrome when the referenced palette hasn't arrived. Renderer split into MirrorFrameRenderer.kt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Controls sit beside the mirror when the content width allows (>=760dp, BoxWithConstraints) and stack below it otherwise; tapping the mirror now toggles keyboard capture instead of only acquiring it, with the hint text naming the way out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Circular D-pad ring replaces the cross cluster (wedge hit-testing with a radial dead band, hold-to-repeat, per-wedge semantics, haptics), keyboard capture becomes an explicit chip with tap-to-toggle, and taps on touch devices forward real panel coordinates (long-press = SELECT, matching the physical touch driver). Manager gains reset() (wired to ViewModel creation and disconnect), a palette region cap, and first-chunk geometry pinning; renderer uses bit-replicated RGB565 and run-length row drawing with the pixel resolver extracted pure and tested; KeyUp of captured keys is consumed; ViewModel moved to its own file; all strings localized; the stray .bak test file is gone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DisplayMirrorManager reassembles RGB565 rect chunks (sequence, geometry, and panel-bound validation) and composites them into a persistent little-endian RGB565 canvas emitted per completed rect; the renderer gains a true-color branch using the same run-length row drawing. Mono frames are untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RGB565 frames identify LVGL/MUI devices, whose UIs read their own input drivers rather than the InputBroker remote events inject into — so the mirror hides the D-pad, keyboard, swipe, and touch affordances and says so, until an input-injection path exists. Utility helpers and the input codes move to DisplayMirrorUtils.kt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
device-ui now accepts injected input (virtual LVGL pointer + group-attached keypad bridged from the InputBroker), so the view-only treatment goes away: the D-pad navigates MUI widgets, taps forward panel coordinates, and the keyboard types. The mirror_view_only string stays for the future capability-flag path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mirror's keyboard capture only mapped navigation keys and silently dropped everything else, so text entry did nothing: printable characters never left the client even though the firmware already injects kb_char. Characters now go out as ANYKEY with the code point, Backspace goes as a character (so text fields delete rather than navigating back, which stays Esc's job), and the handling moves into a Modifier.remoteKeyInput extension. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Third adversarial review pass over the client side. Rect reassembly moves out of DisplayMirrorManagerImpl into MirrorRectCompositor: the manager was carrying two unrelated state machines and 14 methods. Every rect field is now validated before anything is allocated (total_size is device-controlled), geometry is latched from the chunk at offset 0 so a later chunk cannot transpose the blit, continuations must match the buffer that was actually allocated rather than their own claim, and bounds use the subtraction form because rect_x + rect_width overflows for hostile values. A rect that omits width or height covers the full panel in that axis. remoteKeyInput no longer swallows modified keys, so Cmd+Q, Ctrl+C and Alt+Tab stay with the desktop instead of being forwarded to the device. MirrorFrameRenderer tracks colorized runs as a raw Int with a sentinel rather than a boxed Color?, and MirrorFrame.hashCode includes the format so a mono and an RGB565 frame of the same size cannot collide. Drops mirror_view_only, orphaned since MUI gained remote input. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jamesarich
force-pushed
the
screen-mirror-poc
branch
from
September 8, 2026 21:03
c9d6548 to
af3e1bc
Compare
The Debug Panel is the one Settings route deliberately exempt from the `connected && !isManaged` gate every sibling carries, justified by a comment saying it "reads app-local logs only". The Mirror tab put an admin write (set_display_mirror) and arbitrary input injection on that route, so managed mode - which exists to lock a device out of configuration - was bypassed entirely. Enforced in the ViewModel, because the route will not do it: arming, one-shot frame requests, key, character and touch sends are all refused while managed. Disarming still goes through, so a stream already running is torn down when managed mode arrives mid-session. The UI gates on `active = connected && !managed` rather than offering controls that silently do nothing, and the RadioConfig comment now says what the route actually guarantees, since its old wording is what let this through. MirrorHeaderRow is split out of DisplayMirrorContent, which was at detekt's LongMethod limit and could not absorb another line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a live Mirror tab to the Debug panel: the connected device's screen rendered in real time — in its actual theme colors on color-TFT devices — with full remote control.
Verified on hardware against RAK4631 (mono OLED) and LILYGO T-Deck (color, touch). Read the T-Deck result with care: the colour path is MUI/LVGL dirty rects, which need a
device-uicarrying a display flush observer. That is not in any released device-ui, so on this branch the whole MUI path compiles out (HAS_MUI_MIRRORis 0 unlessMESHTASTIC_MUI_MIRRORis defined) and the T-Deck run used a locally built device-ui. A reviewer building from these branches gets the mono path only.🌟 New Features
FromRadio.display_framechunks reassembled into frames and rendered once per frame into a 1:1ImageBitmap, upscaled with nearest-neighbor filtering (crisp device pixels). Color devices stream a signature-keyedDisplayPalette; the renderer applies per-region RGB565 colors with the firmware's last-region-wins precedence, and a frame arriving before its palette self-heals to color when the palette lands.SELECT_LONG, haptics, per-wedge semantics for screen readers); desktop keyboard capture on the mirror (chip- or click-toggled, visible focus ring; arrows/Enter/Esc,KeyUpconsumed so keys never leak to the scroll container); swipe-to-navigate on the image, and onhas_touchdevices tap/long-press forwards real panel coordinates viasend_input_event.DeviceMetadata.displaygates behavior — e-ink devices get a hint steering toward one-shot Refresh; touch capability enables tap-to-touch.Changes
core/repository:DisplayMirrorManager(+MirrorFrame/MirrorPalette),AdminController.setDisplayMirror/requestDisplayFrame/sendInputEvent— non-suspend immediate sends so input never queues behind bulk admin traffic and lifecycle teardown can always stop the stream.core/data: reassembly state machine — contiguous-offset chunks only, geometry pinned at the first chunk, format/size/region-cap validation before any allocation,reset()so a previous device's last screen is never shown as live. Unit-tested (frames, palettes, drops, restarts, caps).core/service: admin senders viasendAdminImmediate.feature/settings: the Mirror tab (DisplayMirror.kt,DisplayMirrorViewModel.kt,DpadRing.kt,MirrorFrameRenderer.kt) — stream stopped on tab dispose, ViewModel clear, and disconnect; toggle tracks connection state. Renderer resolver extracted pure and unit-tested (region precedence, RGB565 bit-replication).core/resources), two Material Symbols chevrons added.🤖 Generated with Claude Code
Status
Rebased onto
main(2026-09-08). Baseline green:spotlessApply spotlessCheck detekt assembleDebug test allTests kmpSmokeCompile, 43 mirror tests passing.Blocking, in release order:
screen-mirror-pocon a fork). Until it lands and a pin bumps, the colour/MUI path is compiled out everywhere.useMavenLocal=trueand the2.99.0-screen-mirror-poc-SNAPSHOTpin. Notemainalready tracks a protobufs develop-SNAPSHOT, so this may need no re-pin at all rather than a version bump.Fixed since the last review pass:
connected && !isManagedgate, on the grounds that it "reads app-local logs only" — and the Mirror tab put an admin write and arbitrary input injection on it. Now refused in the ViewModel (disarm still allowed, so a live stream tears down when managed arrives mid-session) and gated in the UI.FromRadio.display_frame20 → 21 anddisplay_palette21 → 22, which moved off tags open PRs Show full connectivity info in first line #980 and chore(deps): update coroutines_version to v1.8.1 #1030 already claimed.Known gaps, not yet addressed: a cross-repo audit raised further findings on this branch — frame-render cost on the composition thread, and rect-reassembly panel-geometry stability across continuation chunks. Both are recorded and unfixed here.
Screenshots still to attach.
Feature stack
Five repos, and the release order runs top to bottom. Nothing below can ship until the piece above it lands.
DisplayFrame/DisplayPalette/DisplayInfo+ the two admin verbscapture_displaytoolingdevice-ui#394 is the current blocker: without the flush observer the MUI/colour path compiles out of every committed firmware configuration, leaving only the 1bpp mono path.