Skip to content

fix(enhance): include note images in summaries#5422

Merged
ComputelessComputer merged 1 commit into
mainfrom
fix/enhance-image-context
Jun 1, 2026
Merged

fix(enhance): include note images in summaries#5422
ComputelessComputer merged 1 commit into
mainfrom
fix/enhance-image-context

Conversation

@ComputelessComputer
Copy link
Copy Markdown
Collaborator

@ComputelessComputer ComputelessComputer commented Jun 1, 2026

Read referenced image attachments for multimodal summary models and send them as AI SDK image parts. Add fs-sync attachment reading support and coverage.


Note

Medium Risk
Adds local attachment reads and sends image bytes to external LLM APIs when vision models are selected; mitigated by model gating and size/count limits but still affects cost, latency, and note content exposure.

Overview
Enhance can now use note images as multimodal context when the selected LLM supports vision.

When the current provider/model passes modelSupportsImageInput, the transform step gathers images from pre-meeting transcript memos and post-meeting session raw_md. References are parsed from TipTap JSON and markdown (asset:// / local attachment paths and inline data: URLs). Referenced session attachments are loaded via a new attachment_read fs-sync API (Rust core, Tauri command, JS bindings, default permissions), with caps of 10 images and 8MB per image.

The enhance workflow attaches those images as AI SDK ImageParts alongside the text prompt (template discovery and summary streaming) and adds a short instruction that attached images are visual context. Text-only models skip collection entirely.

Tests cover model gating, image reference parsing, attachment reads, and transform wiring.

Reviewed by Cursor Bugbot for commit 5a040b2. Bugbot is set up for automated code reviews on this repo. Configure here.

@netlify
Copy link
Copy Markdown

netlify Bot commented Jun 1, 2026

Deploy Preview for old-char canceled.

Name Link
🔨 Latest commit 5a040b2
🔍 Latest deploy log https://app.netlify.com/projects/old-char/deploys/6a1d0a8bd176940008dea827

Comment thread apps/desktop/src/settings/ai/shared/model-capabilities.ts Outdated
@ComputelessComputer ComputelessComputer force-pushed the fix/enhance-image-context branch from 8018a76 to 9d66acf Compare June 1, 2026 03:58
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 9d66acf. Configure here.

Comment thread apps/desktop/src/store/zustand/ai-task/task-configs/enhance-transform.ts Outdated
Read referenced image attachments for multimodal summary models and send them as AI SDK image parts. Add fs-sync attachment reading support and coverage.
@ComputelessComputer ComputelessComputer force-pushed the fix/enhance-image-context branch from 9d66acf to 5a040b2 Compare June 1, 2026 04:28
@ComputelessComputer ComputelessComputer merged commit ab66f33 into main Jun 1, 2026
10 checks passed
@ComputelessComputer ComputelessComputer deleted the fix/enhance-image-context branch June 1, 2026 04:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant