Skip to content

Commit ef22081

Browse files
committed
feat: implement multi-modal input and enhance chat functionality
- Completed the implementation of image attachment support in the chat input, allowing users to paste, drag, or select images. - Enhanced the `MessageInput` component to handle image validation, including size and type checks. - Updated the `ChatView` and `MessageItem` components to display attached images and manage thinking streaming states. - Added new tests for image handling and validation logic to ensure robustness. - Updated localization files to include new strings related to image attachments and thinking states.
1 parent 5e3181d commit ef22081

15 files changed

Lines changed: 774 additions & 95 deletions

docs/planning/PHASE_2_DETAILED_PLAN.md

Lines changed: 27 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -2035,45 +2035,45 @@ interface ImageAttachment {
20352035

20362036
### 2.12 Token 用量统计 [预估 2h]
20372037

2038-
- [ ] **2.12.1** 定义 `TokenUsage` 结构体(Rust + TypeScript 两端)
2039-
- [ ] **2.12.2**`stream_complete` 事件中传递 usage 数据
2040-
- [ ] **2.12.3**`save_message` 时写入 token_usage 字段(JSON)
2041-
- [ ] **2.12.4** 在 assistant 消息完成后更新 session 的 total_input_tokens / total_output_tokens
2042-
- [ ] **2.12.5** 在 MessageItem 底部显示 TokenBadge 组件
2043-
- [ ] **2.12.6** 验证 Token 统计数据正确
2038+
- [x] **2.12.1** 定义 `TokenUsage` 结构体(Rust + TypeScript 两端)
2039+
- [x] **2.12.2**`stream_complete` 事件中传递 usage 数据
2040+
- [x] **2.12.3**`save_message` 时写入 token_usage 字段(JSON)
2041+
- [x] **2.12.4** 在 assistant 消息完成后更新 session 的 total_input_tokens / total_output_tokens
2042+
- [x] **2.12.5** 在 MessageItem 底部显示 TokenBadge 组件
2043+
- [x] **2.12.6** 验证 Token 统计数据正确
20442044

20452045
### 2.13 思维链展示 [预估 3h]
20462046

2047-
- [ ] **2.13.1** 安装 shadcn/ui 组件:`npx shadcn@latest add collapsible`
2048-
- [ ] **2.13.2** 创建 `src/components/chat/ThinkingBlock.tsx`(折叠/展开组件)
2049-
- [ ] **2.13.3**`MessageItem` 中集成 ThinkingBlock(当 thinking_content 非空时显示)
2050-
- [ ] **2.13.4** 实现流式 thinking 内容的实时更新(通过 stream_thinking 事件)
2051-
- [ ] **2.13.5** 验证 Anthropic Claude thinking blocks 完整流程
2047+
- [x] **2.13.1** 安装 shadcn/ui 组件:`npx shadcn@latest add collapsible`
2048+
- [x] **2.13.2** 创建 `src/components/chat/ThinkingBlock.tsx`(折叠/展开组件)
2049+
- [x] **2.13.3**`MessageItem` 中集成 ThinkingBlock(当 thinking_content 非空时显示)
2050+
- [x] **2.13.4** 实现流式 thinking 内容的实时更新(通过 stream_thinking 事件)
2051+
- [x] **2.13.5** 验证 Anthropic Claude thinking blocks 完整流程
20522052

20532053
### 2.14 停止生成 / 重新生成 [预估 2h]
20542054

2055-
- [ ] **2.14.1** 在 MessageInput 中实现 发送→停止 按钮切换
2056-
- [ ] **2.14.2** 实现前端 `stopGeneration()` 调用 → Rust `stop_generation` command
2057-
- [ ] **2.14.3** 在 MessageItem 操作栏中添加"重新生成"按钮
2058-
- [ ] **2.14.4** 实现前端 `regenerateMessage()` 调用 → Rust `regenerate_message` command
2059-
- [ ] **2.14.5** 验证停止生成后已有内容保留 + 重新生成后旧内容替换
2055+
- [x] **2.14.1** 在 MessageInput 中实现 发送→停止 按钮切换
2056+
- [x] **2.14.2** 实现前端 `stopGeneration()` 调用 → Rust `stop_generation` command
2057+
- [x] **2.14.3** 在 MessageItem 操作栏中添加"重新生成"按钮
2058+
- [x] **2.14.4** 实现前端 `regenerateMessage()` 调用 → Rust `regenerate_message` command
2059+
- [x] **2.14.5** 验证停止生成后已有内容保留 + 重新生成后旧内容替换
20602060

20612061
### 2.15 多模态输入 [预估 4h]
20622062

2063-
- [ ] **2.15.1** 在 MessageInput 中实现剪贴板粘贴图片(paste 事件 → Base64)
2064-
- [ ] **2.15.2** 在 MessageInput 中实现拖拽图片(drag/drop 事件 → Base64)
2065-
- [ ] **2.15.3** 在 MessageInput 中添加 📎 按钮(文件选择对话框)
2066-
- [ ] **2.15.4** 创建 `src/components/chat/ImagePreview.tsx`(缩略图预览 + 删除按钮)
2067-
- [ ] **2.15.5** 在 send_message 时传递 images 参数
2068-
- [ ] **2.15.6** 实现文件大小检查(<=10MB)和类型校验
2069-
- [ ] **2.15.7** 验证粘贴/拖拽/选择三种方式均可正常工作
2070-
- [ ] **2.15.8** 验证视觉模型正确理解图片内容
2063+
- [x] **2.15.1** 在 MessageInput 中实现剪贴板粘贴图片(paste 事件 → Base64)
2064+
- [x] **2.15.2** 在 MessageInput 中实现拖拽图片(drag/drop 事件 → Base64)
2065+
- [x] **2.15.3** 在 MessageInput 中添加 📎 按钮(文件选择对话框)
2066+
- [x] **2.15.4** 创建 `src/components/chat/ImagePreview.tsx`(缩略图预览 + 删除按钮)
2067+
- [x] **2.15.5** 在 send_message 时传递 images 参数
2068+
- [x] **2.15.6** 实现文件大小检查(<=10MB)和类型校验
2069+
- [x] **2.15.7** 验证粘贴/拖拽/选择三种方式均可正常工作
2070+
- [x] **2.15.8** 验证视觉模型正确理解图片内容
20712071

20722072
### i18n 补充 [预估 1h]
20732073

2074-
- [ ] **i18n-1**`en/chat.json` 中添加对话相关翻译键
2075-
- [ ] **i18n-2**`zh-CN/chat.json` 中添加对话相关翻译键
2076-
- [ ] **i18n-3** 所有新增 UI 文字使用 `useTranslation()``t()` 函数
2074+
- [x] **i18n-1**`en/chat.json` 中添加对话相关翻译键
2075+
- [x] **i18n-2**`zh-CN/chat.json` 中添加对话相关翻译键
2076+
- [x] **i18n-3** 所有新增 UI 文字使用 `useTranslation()``t()` 函数
20772077

20782078
### 收尾验证 [预估 2h]
20792079

src/__tests__/chat-store.test.ts

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ describe("useChatStore", () => {
2727
messages: [],
2828
isStreaming: false,
2929
streamingMessageId: null,
30+
isThinkingStreaming: false,
3031
selectedModel: null,
3132
});
3233
});
@@ -215,4 +216,76 @@ describe("useChatStore", () => {
215216
expect(useChatStore.getState().selectedModel).toBeNull();
216217
});
217218
});
219+
220+
describe("thinking streaming state", () => {
221+
it("should set thinking streaming", () => {
222+
useChatStore.getState().setThinkingStreaming(true);
223+
expect(useChatStore.getState().isThinkingStreaming).toBe(true);
224+
});
225+
226+
it("should reset thinking streaming when stream completes", () => {
227+
useChatStore.getState().setThinkingStreaming(true);
228+
useChatStore.getState().setStreaming(false);
229+
expect(useChatStore.getState().isThinkingStreaming).toBe(false);
230+
});
231+
232+
it("should reset thinking streaming on clearMessages", () => {
233+
useChatStore.getState().setThinkingStreaming(true);
234+
useChatStore.getState().clearMessages();
235+
expect(useChatStore.getState().isThinkingStreaming).toBe(false);
236+
});
237+
238+
it("should reset thinking streaming on setActiveSessionData", () => {
239+
useChatStore.getState().setThinkingStreaming(true);
240+
useChatStore.getState().setActiveSessionData({
241+
id: "s",
242+
title: null,
243+
model: null,
244+
system_prompt: null,
245+
working_directory: null,
246+
project_name: null,
247+
status: "active",
248+
mode: "agent",
249+
total_input_tokens: 0,
250+
total_output_tokens: 0,
251+
last_message_at: null,
252+
pinned: false,
253+
group_name: null,
254+
created_at: "",
255+
updated_at: "",
256+
});
257+
expect(useChatStore.getState().isThinkingStreaming).toBe(false);
258+
});
259+
});
260+
261+
describe("removeMessagesFrom", () => {
262+
it("should remove target message and all after it", () => {
263+
useChatStore.getState().setMessages([
264+
makeMessage({ id: "m1" }),
265+
makeMessage({ id: "m2" }),
266+
makeMessage({ id: "m3" }),
267+
]);
268+
useChatStore.getState().removeMessagesFrom("m2");
269+
const msgs = useChatStore.getState().messages;
270+
expect(msgs).toHaveLength(1);
271+
expect(msgs[0].id).toBe("m1");
272+
});
273+
274+
it("should remove all messages when first is targeted", () => {
275+
useChatStore.getState().setMessages([
276+
makeMessage({ id: "m1" }),
277+
makeMessage({ id: "m2" }),
278+
]);
279+
useChatStore.getState().removeMessagesFrom("m1");
280+
expect(useChatStore.getState().messages).toHaveLength(0);
281+
});
282+
283+
it("should do nothing if id not found", () => {
284+
useChatStore.getState().setMessages([
285+
makeMessage({ id: "m1" }),
286+
]);
287+
useChatStore.getState().removeMessagesFrom("not-found");
288+
expect(useChatStore.getState().messages).toHaveLength(1);
289+
});
290+
});
218291
});

src/__tests__/chat-view.test.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ describe("ChatView store integration", () => {
3131
messages: [],
3232
isStreaming: false,
3333
streamingMessageId: null,
34+
isThinkingStreaming: false,
3435
selectedModel: null,
3536
});
3637
});

src/__tests__/message-input.test.ts

Lines changed: 158 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -115,29 +115,176 @@ describe("ModelSelector groupByProvider logic", () => {
115115
});
116116
});
117117

118-
describe("Send validation logic", () => {
119-
function canSend(content: string, disabled: boolean, isStreaming: boolean): boolean {
120-
return content.trim().length > 0 && !disabled && !isStreaming;
118+
describe("Send validation logic (with images)", () => {
119+
function canSend(
120+
content: string,
121+
hasImages: boolean,
122+
disabled: boolean,
123+
isStreaming: boolean
124+
): boolean {
125+
return (content.trim().length > 0 || hasImages) && !disabled && !isStreaming;
121126
}
122127

123128
it("should allow send when content is present and not streaming", () => {
124-
expect(canSend("hello", false, false)).toBe(true);
129+
expect(canSend("hello", false, false, false)).toBe(true);
125130
});
126131

127-
it("should disallow send with empty content", () => {
128-
expect(canSend("", false, false)).toBe(false);
129-
expect(canSend(" ", false, false)).toBe(false);
132+
it("should disallow send with empty content and no images", () => {
133+
expect(canSend("", false, false, false)).toBe(false);
134+
expect(canSend(" ", false, false, false)).toBe(false);
135+
});
136+
137+
it("should allow send with images even if text is empty", () => {
138+
expect(canSend("", true, false, false)).toBe(true);
139+
expect(canSend(" ", true, false, false)).toBe(true);
130140
});
131141

132142
it("should disallow send when disabled", () => {
133-
expect(canSend("hello", true, false)).toBe(false);
143+
expect(canSend("hello", false, true, false)).toBe(false);
134144
});
135145

136146
it("should disallow send when streaming", () => {
137-
expect(canSend("hello", false, true)).toBe(false);
147+
expect(canSend("hello", false, false, true)).toBe(false);
148+
});
149+
150+
it("should disallow send with images when streaming", () => {
151+
expect(canSend("", true, false, true)).toBe(false);
152+
});
153+
});
154+
155+
describe("Image validation logic", () => {
156+
const MAX_IMAGE_SIZE = 10 * 1024 * 1024;
157+
const ALLOWED_IMAGE_TYPES = [
158+
"image/png",
159+
"image/jpeg",
160+
"image/gif",
161+
"image/webp",
162+
];
163+
164+
function validateImage(type: string, size: number): { valid: boolean; reason?: string } {
165+
if (!ALLOWED_IMAGE_TYPES.includes(type)) {
166+
return { valid: false, reason: "unsupported_type" };
167+
}
168+
if (size > MAX_IMAGE_SIZE) {
169+
return { valid: false, reason: "too_large" };
170+
}
171+
return { valid: true };
172+
}
173+
174+
it("should accept valid PNG", () => {
175+
expect(validateImage("image/png", 1024).valid).toBe(true);
176+
});
177+
178+
it("should accept valid JPEG", () => {
179+
expect(validateImage("image/jpeg", 5 * 1024 * 1024).valid).toBe(true);
180+
});
181+
182+
it("should accept valid WebP", () => {
183+
expect(validateImage("image/webp", 100).valid).toBe(true);
184+
});
185+
186+
it("should accept valid GIF", () => {
187+
expect(validateImage("image/gif", 2 * 1024 * 1024).valid).toBe(true);
188+
});
189+
190+
it("should reject unsupported type", () => {
191+
const result = validateImage("image/svg+xml", 1024);
192+
expect(result.valid).toBe(false);
193+
expect(result.reason).toBe("unsupported_type");
194+
});
195+
196+
it("should reject text file", () => {
197+
expect(validateImage("text/plain", 100).valid).toBe(false);
198+
});
199+
200+
it("should reject image that is too large", () => {
201+
const result = validateImage("image/png", 11 * 1024 * 1024);
202+
expect(result.valid).toBe(false);
203+
expect(result.reason).toBe("too_large");
204+
});
205+
206+
it("should accept image at exactly 10MB boundary", () => {
207+
expect(validateImage("image/png", MAX_IMAGE_SIZE).valid).toBe(true);
208+
});
209+
210+
it("should reject image at 10MB + 1 byte", () => {
211+
expect(validateImage("image/png", MAX_IMAGE_SIZE + 1).valid).toBe(false);
212+
});
213+
});
214+
215+
describe("Attachment parsing logic", () => {
216+
interface AttachmentImage {
217+
data: string;
218+
mime_type: string;
219+
}
220+
221+
function parseAttachments(raw: string | null): AttachmentImage[] {
222+
if (!raw) return [];
223+
try {
224+
const parsed = JSON.parse(raw) as AttachmentImage[];
225+
return Array.isArray(parsed) ? parsed : [];
226+
} catch {
227+
return [];
228+
}
229+
}
230+
231+
it("should return empty array for null", () => {
232+
expect(parseAttachments(null)).toEqual([]);
233+
});
234+
235+
it("should return empty array for invalid JSON", () => {
236+
expect(parseAttachments("not-json")).toEqual([]);
237+
});
238+
239+
it("should parse valid attachment array", () => {
240+
const json = JSON.stringify([
241+
{ data: "abc123", mime_type: "image/png" },
242+
{ data: "def456", mime_type: "image/jpeg" },
243+
]);
244+
const result = parseAttachments(json);
245+
expect(result).toHaveLength(2);
246+
expect(result[0].mime_type).toBe("image/png");
247+
});
248+
249+
it("should return empty array for non-array JSON", () => {
250+
expect(parseAttachments(JSON.stringify({ key: "value" }))).toEqual([]);
251+
});
252+
});
253+
254+
describe("TokenUsage parsing logic", () => {
255+
interface TokenUsage {
256+
input_tokens: number;
257+
output_tokens: number;
258+
total_tokens: number;
259+
}
260+
261+
function parseTokenUsage(raw: string | null): TokenUsage | null {
262+
if (!raw) return null;
263+
try {
264+
return JSON.parse(raw) as TokenUsage;
265+
} catch {
266+
return null;
267+
}
268+
}
269+
270+
it("should return null for null input", () => {
271+
expect(parseTokenUsage(null)).toBeNull();
272+
});
273+
274+
it("should return null for invalid JSON", () => {
275+
expect(parseTokenUsage("bad")).toBeNull();
138276
});
139277

140-
it("should allow content with only whitespace trimmed to empty", () => {
141-
expect(canSend("\n\n \t", false, false)).toBe(false);
278+
it("should parse valid usage JSON", () => {
279+
const json = JSON.stringify({
280+
input_tokens: 100,
281+
output_tokens: 200,
282+
total_tokens: 300,
283+
});
284+
const result = parseTokenUsage(json);
285+
expect(result).not.toBeNull();
286+
expect(result!.input_tokens).toBe(100);
287+
expect(result!.output_tokens).toBe(200);
288+
expect(result!.total_tokens).toBe(300);
142289
});
143290
});

0 commit comments

Comments
 (0)