feat: add StudyOverlay AI tutor demo using Moss semantic memory - #369
Open
AyanBhardwaj1 wants to merge 2 commits into
Open
feat: add StudyOverlay AI tutor demo using Moss semantic memory#369AyanBhardwaj1 wants to merge 2 commits into
AyanBhardwaj1 wants to merge 2 commits into
Conversation
|
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new StudyOverlay community demo under moss-live-labs/ showcasing a desktop “always-on-top” study assistant that captures the screen, calls an OpenRouter vision-capable chat model for a tutor-style explanation, and uses Moss as local semantic session memory (JS sidecar by default, Python fallback).
Changes:
- Introduces a Python + pywebview overlay app with global hotkey capture and a settings UI.
- Adds Moss “session memory” integration via a Node sidecar (
@moss-dev/moss) plus a Python SDK fallback. - Adds local settings persistence and demo packaging files for Python and Node dependencies.
Reviewed changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| moss-live-labs/community-demos/study-overlay/settings.py | Implements config storage/loading and exposes settings to the UI. |
| moss-live-labs/community-demos/study-overlay/requirements.txt | Python runtime dependencies for the overlay app. |
| moss-live-labs/community-demos/study-overlay/README.md | Demo setup + usage instructions. |
| moss-live-labs/community-demos/study-overlay/pnpm-lock.yaml | Locks Node dependencies for the Moss JS sidecar (and other declared deps). |
| moss-live-labs/community-demos/study-overlay/package.json | Node dependency manifest used to install @moss-dev/moss for the sidecar. |
| moss-live-labs/community-demos/study-overlay/overlay.js | Frontend logic for the overlay UI and settings form. |
| moss-live-labs/community-demos/study-overlay/overlay.html | Overlay UI markup + external script/style includes. |
| moss-live-labs/community-demos/study-overlay/overlay.css | Overlay UI styling. |
| moss-live-labs/community-demos/study-overlay/moss_sidecar.mjs | Node sidecar process that queries/adds docs to Moss. |
| moss-live-labs/community-demos/study-overlay/memory.py | Moss memory abstraction with JS sidecar and Python fallback backends. |
| moss-live-labs/community-demos/study-overlay/main.py | App entrypoint: window creation, hotkey listener, capture flow wiring. |
| moss-live-labs/community-demos/study-overlay/capture.py | Screen capture helper producing PNG/base64/data URL. |
| moss-live-labs/community-demos/study-overlay/ai.py | OpenRouter client + prompt composition for screenshot explanations. |
| moss-live-labs/community-demos/study-overlay/.gitignore | Ignores local build artifacts and local config. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+70
to
+83
| def public_settings(settings: dict[str, str] | None = None) -> dict[str, Any]: | ||
| data = settings if settings is not None else load_settings() | ||
| return { | ||
| "config_path": str(config_path()), | ||
| "missing": missing_required(data), | ||
| "has_openrouter": bool(data.get("OPENROUTER_API_KEY")), | ||
| "has_moss_project_id": bool(data.get("MOSS_PROJECT_ID")), | ||
| "has_moss_project_key": bool(data.get("MOSS_PROJECT_KEY")), | ||
| "values": { | ||
| "OPENROUTER_API_KEY": data.get("OPENROUTER_API_KEY", ""), | ||
| "MOSS_PROJECT_ID": data.get("MOSS_PROJECT_ID", ""), | ||
| "MOSS_PROJECT_KEY": data.get("MOSS_PROJECT_KEY", ""), | ||
| }, | ||
| } |
Comment on lines
+163
to
+167
| deadline = time.time() + timeout | ||
| while time.time() < deadline: | ||
| line = self._process.stdout.readline() | ||
| if not line: | ||
| if self._process.poll() is not None: |
Comment on lines
+8
to
+12
| <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.17.0/dist/katex.min.css" crossorigin="anonymous" /> | ||
| <script defer src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script> | ||
| <script defer src="https://cdn.jsdelivr.net/npm/katex@0.17.0/dist/katex.min.js" crossorigin="anonymous"></script> | ||
| <script defer src="https://cdn.jsdelivr.net/npm/katex@0.17.0/dist/contrib/auto-render.min.js" crossorigin="anonymous"></script> | ||
| <script defer src="overlay.js"></script> |
| @@ -0,0 +1,27 @@ | |||
| { | |||
| "name": "moss-meeting-copilot-overlay", | |||
Comment on lines
+10
to
+14
| "dependencies": { | ||
| "@moss-dev/moss": "^1.0.0", | ||
| "next": "^15.3.0", | ||
| "react": "^19.0.0", | ||
| "react-dom": "^19.0.0" |
Comment on lines
+28
to
+32
| def __init__(self) -> None: | ||
| self.window: webview.Window | None = None | ||
| self.memory = StudyMemory() | ||
| self.busy = False | ||
|
|
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Author
|
Committed changes from Copilot, please merge. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| <title>StudyOverlay</title> | ||
| <link rel="stylesheet" href="overlay.css" /> | ||
| <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.17.0/dist/katex.min.css" crossorigin="anonymous" /> | ||
| <script defer src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script> |
There was a problem hiding this comment.
BLOCKING ```html
<script defer src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script>This privileged pywebview page exposes `window.pywebview.api`, including screen capture and settings APIs, but it executes remote CDN scripts. A CDN compromise or unpinned `marked` update can call `run_capture()` or read settings from the renderer. Bundle/vendor these assets locally and load only `self` scripts, ideally with a CSP; avoid giving third-party code access to the pywebview bridge.
| "has_moss_project_id": bool(data.get("MOSS_PROJECT_ID")), | ||
| "has_moss_project_key": bool(data.get("MOSS_PROJECT_KEY")), | ||
| "values": { | ||
| "OPENROUTER_API_KEY": data.get("OPENROUTER_API_KEY", ""), |
There was a problem hiding this comment.
BLOCKING ```py
"OPENROUTER_API_KEY": data.get("OPENROUTER_API_KEY", ""),
`public_settings()` sends raw API keys back into the webview, so any renderer XSS or third-party script can exfiltrate OpenRouter and Moss credentials. Return only booleans/masked labels, leave password fields blank on edit, and make save semantics preserve existing secrets unless the user supplies a replacement or explicitly clears them.
| const source = markdown || ""; | ||
| const escaped = source.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">"); | ||
| if (window.marked) { | ||
| responseEl.innerHTML = window.marked.parse(escaped, { mangle: false, headerIds: false }); |
There was a problem hiding this comment.
BLOCKING ```js
responseEl.innerHTML = window.marked.parse(escaped, { mangle: false, headerIds: false });
The model response is untrusted input, and rendering Marked output directly into `innerHTML` leaves HTML/URL injection paths in the same privileged renderer that can reach `window.pywebview.api`. Sanitize the generated HTML with DOMPurify or render a safe Markdown subset, and reject dangerous link/image protocols before assigning to `innerHTML`.
|
|
||
| deadline = time.time() + timeout | ||
| while time.time() < deadline: | ||
| line = self._process.stdout.readline() |
There was a problem hiding this comment.
CONSIDER ```py
line = self._process.stdout.readline()
This blocking `readline()` means the surrounding deadline is ineffective: if the sidecar hangs without emitting a newline, startup or capture can wait forever instead of disabling memory. Read stdout on a background thread into a `Queue` and use `queue.get(timeout=...)`, or use nonblocking I/O/select; on timeout, kill/restart the sidecar and fall back to no memory.
| self.window.show() | ||
|
|
||
| def run_capture(self, user_prompt: str = "") -> dict[str, Any]: | ||
| if self.busy: |
There was a problem hiding this comment.
CONSIDER ```py
if self.busy:
return {"ok": False, "error": "StudyOverlay is already working on a capture."}
`busy` is a plain check-then-set flag shared by pywebview calls and hotkey worker threads, so two near-simultaneous captures can both pass the check and run, racing window hide/show and Moss sidecar access. Replace it with a `threading.Lock` or `Semaphore(1)` and acquire non-blocking around the whole capture flow.
Codex reviewThe PR adds a functional desktop overlay demo, but the renderer/security boundary is too loose for an app that captures the screen and stores API credentials. I also found a capture concurrency race and a sidecar timeout that can hang instead of falling back cleanly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Checklist
Please ensure that your PR meets the following requirements:
Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context.
Fixes # (issue number)
Type of Change