Skip to content

feat(driver): display flush observer and opt-in remote input injection - #394

Draft
jamesarich wants to merge 6 commits into
meshtastic:masterfrom
jamesarich:screen-mirror-poc
Draft

feat(driver): display flush observer and opt-in remote input injection#394
jamesarich wants to merge 6 commits into
meshtastic:masterfrom
jamesarich:screen-mirror-poc

Conversation

@jamesarich

@jamesarich jamesarich commented Sep 8, 2026

Copy link
Copy Markdown

Adds the two seams MUI needs so a host can mirror the panel and drive it remotely. Both are inert unless a host opts in — no default build changes behaviour.

This is the missing piece of the screen-mirroring stack: firmware currently consumes it through a symlink:// override in platformio.ini, because no released device-ui carries the flush observer. Until this lands and a pin bumps, the whole MUI/colour path in meshtastic/firmware#11681 compiles out (HAS_MUI_MIRROR is 0) and only the 1bpp mono path works.

What this adds

A display flush observer (DisplayDriver::flushObserver, an atomic). Called on every LVGL flush before panel byte-swapping, so the observer sees the dirty area (x, y, w, h) and pixels as native little-endian RGB565 with rows tightly packed. Null by default; loaded with acquire semantics on the flush path.

A cross-thread full-refresh request. A host streaming the screen needs to force a complete repaint from another thread. fullRefreshRequested is honoured on the LVGL thread in task_handler, and invalidates the active screen plus lv_layer_top() and lv_layer_sys() — overlay content (clock, notifications) lives on those layers, so a full sync has to repaint them too or the client sees a stale overlay.

A wake request. Injected input must wake a slept panel and still act. Without this the first remote event is swallowed as a wake — which is every event, since remote control is exactly the case where nobody is physically at the device.

Opt-in remote input injection. A host calls InputDriver::enableInjection() before init() to add three LVGL virtual input devices — pointer, keypad, and encoder — fed from a 16-entry single-producer/single-consumer ring. injectTouch() supports a hold duration so long-press works; injectEncoder() maps rotation onto MUI's trackball semantics. Nothing is registered unless enableInjection() was called.

Scope and safety

  • Default builds are unaffected: the observer is a null atomic and injectionEnabled is false, so neither path is reachable without a host explicitly wiring it.
  • No new dependencies; <atomic> is the only added include.
  • The consuming side deliberately keeps mirroring local-connection-only — firmware drops the arm request if it arrives over the mesh, because frames ride FromRadio and honouring a remote arm would stream the screen to whatever local client happens to be attached.

Feature stack

Five repos, and the release order runs top to bottom. Nothing below can ship until the piece above it lands.

Repo PR Role
meshtastic/design #142 Cross-platform feature spec
meshtastic/device-ui this PR MUI flush observer + opt-in input injection
meshtastic/protobufs #1054 DisplayFrame / DisplayPalette / DisplayInfo + the two admin verbs
meshtastic/firmware #11681 Producer: streams the framebuffer, bridges input
meshtastic/Meshtastic-Android #6987 Mirror tab with remote control
meshtastic/meshtastic-mcp #78 capture_display tooling

device-ui#394 is the current blocker: without the flush observer the MUI/colour path compiles out of every committed firmware configuration, leaving only the 1bpp mono path.

Testing

Exercised on a LILYGO T-Deck via meshtastic/firmware#11681 with a local symlink:// override — live mirror plus D-pad, keyboard and touch control from Meshtastic-Android. Draft until reviewers weigh in on the seam shape; the firmware side is the only consumer today.

jamesarich and others added 5 commits September 8, 2026 15:58
DisplayDriver gains a static flush observer invoked from the LGFX flush
callback before the in-place panel byte-swap, so observers receive native
little-endian RGB565 dirty rects (LVGL thread; copy and return), plus a
thread-safe requestFullRefresh drained in task_handler so a newly attached
observer can synchronize the full frame. Both flush variants stay in sync.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
lv_scr_act covers only the active screen; clock and notification overlays
live on the top and system layers, so a full-frame sync for a flush
observer must repaint those too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
InputDriver gains injectTouch/injectKey statics callable from any thread:
events land in lock-free SPSC rings drained by two always-present virtual
LVGL devices created in the (previously empty) base init — a pointer whose
read callback holds PRESSED for a requested duration (taps and synthesized
long presses) and a keypad attached to the default group, which init now
guarantees exists even on boards with no physical input so injected keys
can navigate widgets. Mirrors the flush-observer pattern on the output side.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two fixes for remote control. First: a slept panel swallowed injected
input as a wake event, which is every event when nobody is physically at
the device — inject* now raises a wake request drained on the UI thread
(force wakeup when powersaving, otherwise reset the inactivity timer),
distinct from toggleDisplay which would sleep an awake screen. Second: add
a virtual encoder device, because LVGL moves focus between widgets on
encoder rotation while keypad UP/DOWN go to the already-focused widget —
so arrow keys silently did nothing while typed characters worked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Congratulations for your first pull request

@jamesarich

Copy link
Copy Markdown
Author

build-and-test is red, and I do not believe it is this branch. Recording the evidence so a reviewer does not have to re-derive it.

The failure is in generating cannedmessages.pb.cpp and admin.pb.cpp:

AttributeError: module 'proto.nanopb_pb2' has no attribute 'IS_8'
--nanopb_out: protoc-gen-nanopb: Plugin failed with status code 1.

This diff is five C++ files (DisplayDriver, LGFXDriver, InputDriver) and adds no .proto and no build-system change, so it cannot reach the proto-generation step.

What it looks like instead: requirements.txt is a single unpinned grpcio-tools, which drags in whatever protobuf runtime is current at CI time. nanopb_pb2 missing an attribute the runtime expects is the usual shape of that drift. The last green build-and-test runs were #388/#390/#391 on 2026-09-06/07; this ran on 2026-09-08.

I have not proven it by reproducing locally, so treat that as a strong hypothesis rather than a finding. If it holds, it will redden every PR until grpcio-tools (or protobuf) is pinned, and it is worth fixing on master independently of this branch — happy to open that separately.

Note #393, opened the same day, shows no build-and-test check at all, so it is not a counter-example.

Creating the default input group unconditionally would enrol every
focusable widget on touch-only boards that have no focus concept, so
injection is now opt-in: a host calls enableInjection() before init(),
and init() is otherwise inert as before. The flush observer becomes
atomic with a documented detach-quiesce contract, its comment records
that the tightly-packed pixel guarantee is PARTIAL-render-mode only and
that LGFXDriver is the only driver honouring it, the injector contract
narrows to single-producer (the rings are SPSC), and encoder steps clamp
instead of truncating through int8.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jamesarich

Copy link
Copy Markdown
Author

Correcting my own comment above: the timing evidence I gave for the grpcio-tools theory was wrong.

I said the last green build-and-test was 2026-09-07. It was not — input-policy went green at 2026-09-08 06:50, about 17 hours before this branch failed at 23:29, on the same master tip (7bdde1f, this branch is 0 behind). So "broken since the 7th" does not hold, and the unpinned-grpcio-tools drift theory is weaker than I made it sound.

What still stands: this diff is five C++ files under include/graphics/driver/, include/input/ and source/, with no .proto, no requirements.txt and no build-system change, and the failure is a Python-side nanopb_pb2 AttributeError during proto generation for cannedmessages/admin. I cannot see a mechanism by which this branch reaches that step.

I have re-triggered CI to get a second datapoint. Treat the cause as unknown until that comes back, not as diagnosed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant