Skip to content

feat: add image context model for Claude proxy#3771

Open
Pswzy wants to merge 2 commits into
farion1231:mainfrom
Pswzy:claude-image-context-model
Open

feat: add image context model for Claude proxy#3771
Pswzy wants to merge 2 commits into
farion1231:mainfrom
Pswzy:claude-image-context-model

Conversation

@Pswzy
Copy link
Copy Markdown

@Pswzy Pswzy commented Jun 5, 2026

Summary

  • Add an optional image processing model for Claude and Claude Desktop proxy providers.
  • When a request contains images, the proxy first asks the configured image model to extract image context, then removes image blocks and injects the extracted text back into the original message order before forwarding to the main mapped model.
  • Keep existing unsupported-image fallback behavior when no image model is configured.

Details

  • Adds ProviderMeta.imageModel and matching frontend provider form fields with i18n strings.
  • Supports Anthropic image blocks plus OpenAI-style input_image/image_url data URL blocks and nested image MIME metadata.
  • Avoids recursive image preprocessing/model mapping for the image-analysis request.
  • Uses an English internal analysis protocol while preserving parsing compatibility for Chinese image-analysis section labels.
  • Automatically enables Claude Code proxy takeover when the active provider has an image model, and restores it when switching to providers without one.

Tests

  • pnpm typecheck
  • pnpm test:unit
  • cargo +stable test image_context::tests
  • cargo +stable test media_sanitizer::tests
  • cargo +stable test provider_meta_serializes_image_model

@farion1231
Copy link
Copy Markdown
Owner

@codex review

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 633c7c18ed

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/hooks/useProviderActions.ts Outdated
Comment on lines +176 to +177
} else if (activeApp === "claude" && isProxyTakeover) {
await proxyApi.setProxyTakeoverForApp("claude", false);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Only disable Claude takeover for the provider being edited

When editing any Claude provider that does not have an imageModel, this branch turns off proxy takeover whenever isProxyTakeover is true, without checking whether the edited provider is the current provider. If the current provider still relies on an image model and the user edits an inactive text-only provider, Claude Code gets restored away from the proxy and image-context routing stops working for the active provider. The disable path should mirror the enable path by first confirming the updated/original id is the current provider.

Useful? React with 👍 / 👎.

@Pswzy
Copy link
Copy Markdown
Author

Pswzy commented Jun 6, 2026

Updated the PR to address the automated review and CI formatting failure.\n\nChanges in the follow-up commit:\n- Guard the Claude proxy takeover disable path so it only runs when the edited provider is the current provider, matching the enable path.\n- Added a focused useProviderActions test covering inactive text-only provider edits while Claude proxy takeover is active.\n- Ran Prettier on the files flagged by CI.\n\nVerified locally:\n- pnpm format:check\n- pnpm typecheck\n- pnpm exec vitest run tests/hooks/useProviderActions.test.tsx\n- pnpm test:unit

@makoMakoGo
Copy link
Copy Markdown
Collaborator

makoMakoGo commented Jun 7, 2026

Updated the PR to address the automated review and CI formatting failure.\n\nChanges in the follow-up commit:\n- Guard the Claude proxy takeover disable path so it only runs when the edited provider is the current provider, matching the enable path.\n- Added a focused useProviderActions test covering inactive text-only provider edits while Claude proxy takeover is active.\n- Ran Prettier on the files flagged by CI.\n\nVerified locally:\n- pnpm format:check\n- pnpm typecheck\n- pnpm exec vitest run tests/hooks/useProviderActions.test.tsx\n- pnpm test:unit

Thanks for the pull request. When using AI to create pull requests and comments, please conduct a self-check first. Line breaks and escape characters were not properly handled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants