fix(sdk): resize Anthropic many-image inputs by Zheng-Lu · Pull Request #2552 · OpenHands/software-agent-sdk

Zheng-Lu · 2026-03-23T22:51:01Z

Summary

Reproduces and fixes the Anthropic many-image failure by resizing oversized base64 images during LLM message formatting.

What Changed

Added an Anthropic-only resize path in LLM.format_messages_for_llm
Resize only triggers when the outgoing request crosses the many-image threshold
Preserves aspect ratio and leaves URL images unchanged
Added pillow as a runtime dependency for in-memory image resizing

Validation

pytest tests/sdk/llm/test_llm_image_resizing.py Passed
pytest tests/sdk/llm/test_llm_image_resizing.py tests/sdk/llm/test_vision_support.py Passed
ruff check openhands-sdk/openhands/sdk/llm/llm.py tests/sdk/llm/test_llm_image_resizing.py Passed
pyright openhands-sdk/openhands/sdk/llm/llm.py tests/sdk/llm/test_llm_image_resizing.py Passed

Proof

Now the multiple-images request with at least one image > 2000px doesn't throw the error litellm.BadRequestError

Co-authored-by: openhands <openhands@all-hands.dev>

xingyaoww · 2026-03-27T06:46:54Z

@OpenHands pls merge from main, resolve all conflicts. Then do /codereview-roasted /github-pr-review

openhands-ai · 2026-03-27T06:47:16Z

I'm on it! xingyaoww can track my progress at all-hands.dev

Co-authored-by: openhands <openhands@all-hands.dev>

xingyaoww

Taste Rating: 🟡 Acceptable — Works, but the structure needs improvement

Linus's Three Questions:

Is this solving a real problem? — Yes. Anthropic's many-image limit is a real production failure.
Is there a simpler way? — Yes. This is ~80 lines of image manipulation code jammed into a 1500-line god-class. Extract it.
What will this break? — Adding pillow as a hard runtime dependency to the core SDK is the biggest concern. Every user now pays for PIL whether they use images or not.

VERDICT:
❌ Needs rework — The fix is directionally correct, but the dependency strategy and code placement need redesign before merging.

KEY INSIGHT:
The core problem is treating PIL as a hard SDK dependency and stuffing image-processing plumbing into the LLM class, when this should be a lazy-loaded utility module.

openhands-ai · 2026-03-27T06:53:31Z

OpenHands encountered an error: Request timeout after 30 seconds to https://ypvqnvwqmvbumzax.prod-runtime.all-hands.dev/api/conversations/9650ebc3-b326-46ce-88ee-b683ce63e259/ask_agent

See the conversation for more information.

Co-authored-by: openhands <openhands@all-hands.dev>

xingyaoww · 2026-04-10T15:35:53Z

@OpenHands do /codereview-roasted /github-pr-review

openhands-ai · 2026-04-10T15:36:19Z

I'm on it! xingyaoww can track my progress at all-hands.dev

xingyaoww

🟡 Acceptable — Core logic is correct and the tests are solid, but there are design issues worth addressing before merge.

Linus-Style Analysis:

The fundamental idea is sound: intercept oversized base64 images before they hit Anthropic's API limits. The Anthropic docs research is correct (20-image threshold, 2000px vs 8000px caps), and the resize logic itself is clean.

But the plumbing around it — threading a PIL module as Any through three functions, silent in-place mutation — adds unnecessary complexity that a simpler design would eliminate entirely.

⚠️ Eval-risk note: This PR changes image processing in the LLM message pipeline, which could affect agent behavior on vision benchmarks. Flagging for a human maintainer to decide after running lightweight evals.

This review was generated by an AI agent (OpenHands).

openhands-ai · 2026-04-10T15:40:40Z

Summary

The request was to perform a /codereview-roasted and /github-pr-review on PR #2552 ("fix(sdk): resize Anthropic many-image inputs"). Here's what was done:

Checklist

✅ Retrieved PR diff and context — Fetched the full diff (5 files changed), PR description, and linked issue SDK should auto-resize oversized images before sending to LLM providers #2467
✅ Performed roasted code review — Analyzed the code through the Linus Torvalds-style lens focusing on data structures, complexity, pragmatism, and breaking changes
✅ Posted structured GitHub PR review — Submitted a single COMMENT review with 6 inline comments using priority labels (🟠🟠🟡🟡🟢🟢) via the GitHub API
✅ Used correct review event — Left as COMMENT (not APPROVE) because the PR changes image processing in the LLM message pipeline, which falls under the eval-risk policy requiring human maintainer sign-off
✅ No code modifications made — Review-only, as required by the codereview skill

Key Findings Posted

🟠 image_module: Any threading — Unnecessary complexity; Python's sys.modules cache makes repeated imports free
🟠 Silent in-place mutation — maybe_resize_messages_for_provider mutates inputs with no return value, relying on an invisible deepcopy contract
🟡 pillow>=12.1.1 floor too high — The APIs used are stable since Pillow 9.1+
🟡 Leaky public API — resize_base64_data_url exposes an Any-typed PIL parameter
🟢 Good dimension logic — Clean early returns, correct Anthropic doc mirroring
🟢 Solid tests — Real image creation and dimension assertions, not mock-only

No extraneous changes were made — this was purely a review action with no code modifications.

all-hands-bot · 2026-04-16T12:39:15Z

[Automatic Post]: This PR seems to be currently waiting for review. @xingyaoww @Zheng-Lu @openhands-ai[bot], could you please take a look when you have a chance?

Co-authored-by: openhands <openhands@all-hands.dev>

all-hands-bot · 2026-04-22T12:37:27Z

[Automatic Post]: It has been a while since there was any activity on this PR. @Zheng-Lu, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

all-hands-bot · 2026-04-26T12:28:39Z

[Automatic Post]: This PR seems to be currently waiting for review. @xingyaoww, could you please take a look when you have a chance?

xingyaoww · 2026-04-27T14:05:12Z

@OpenHands in the existing example file that tests for vision support, can you add another send_message that sends image inputs that would properly exercise the stuff we've introduced in this PR

openhands-ai · 2026-04-27T14:05:44Z

I'm on it! xingyaoww can track my progress at all-hands.dev

Add a second send_message in 17_image_input.py that sends 21 oversized base64 images (2500×100 px each), triggering the Anthropic many-image threshold (>20 images → 2000 px cap). This exercises the image resizing path introduced in this PR end-to-end. Co-authored-by: openhands <openhands@all-hands.dev>

openhands-ai · 2026-04-27T14:54:56Z

I'm on it! xingyaoww can track my progress at all-hands.dev

The wait-on-check-action was using github.event.pull_request.head.ref (branch name) to look up check runs. For fork PRs, the branch name doesn't exist in the upstream repo, causing a 422 error: 'No commit found for SHA: fix/2467-image-downscale' Switch to github.event.pull_request.head.sha which always resolves correctly regardless of whether the PR is from a fork. Co-authored-by: openhands <openhands@all-hands.dev>

The Build & Push workflow doesn't run for fork PRs, so the wait-on-check-action was timing out and failing with 'The requested check was never run against this ref'. Set fail-on-no-checks: false so the wait step is a no-op when the check doesn't exist (fork PRs) while still waiting properly when it does (non-fork PRs). Co-authored-by: openhands <openhands@all-hands.dev>

Fork PRs have restricted GITHUB_TOKEN permissions and cannot write comments on the upstream repo's PR. The update_comment function was calling exit 1 on failure, which aborted the entire test run. Change to emit warnings and continue when comment creation fails, clearing API_URL to skip subsequent comment attempts. Co-authored-by: openhands <openhands@all-hands.dev>

openhands-ai · 2026-04-27T15:21:32Z

Summary

The user asked me to fix the CI failure at https://github.com/OpenHands/software-agent-sdk/actions/runs/25000384169/job/73209940791?pr=2552 for PR #2552 on a fork branch.

Was the request addressed?

✅ Identified the root cause: The test-examples workflow had three separate issues preventing it from running on fork PRs.
✅ Fix 1 — head.ref → head.sha: The wait-on-check-action used the branch name to look up check runs, which doesn't exist in the upstream repo for fork PRs (422 error). Changed to use the commit SHA.
✅ Fix 2 — fail-on-no-checks: false: The Build & Push (python-amd64) check doesn't run for fork PRs, causing a timeout. Made the wait step tolerate missing checks.
✅ Fix 3 — Non-fatal comment posting: Fork PRs lack write permissions to post PR comments (403 error). Changed exit 1 to warnings + return so test execution isn't aborted.
✅ Merged main into the branch to bring it up to date (clean merge, no conflicts).
✅ Pushed all changes to the fork branch and re-triggered the workflow.
✅ Verified the wait step now passes successfully and the workflow progresses to running actual tests.

Are the changes concise?

Yes — all three changes are to .github/workflows/run-examples.yml and are minimal, targeted fixes for fork PR compatibility. No extraneous changes were introduced. The merge from main was also requested context (branch was behind).

Remaining limitation

The 48/53 example test failures after the infrastructure fixes are due to GitHub Actions not exposing repository secrets (LLM_API_KEY, etc.) to fork PR workflows. This is expected security behavior affecting all fork PRs and cannot be fixed from the PR side — a maintainer would need to re-run from the upstream context.

all-hands-bot · 2026-05-04T12:50:15Z

[Automatic Post]: It has been a while since there was any activity on this PR. @Zheng-Lu, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

xingyaoww · 2026-05-05T15:00:14Z

@OpenHands please merge from main and resolve conflicts

openhands-ai · 2026-05-05T15:00:33Z

I'm on it! xingyaoww can track my progress at all-hands.dev

…lict Keep main's litellm>=1.83.7 and PR's pillow>=12.1.1 dependency. Co-authored-by: openhands <openhands@all-hands.dev>

openhands-ai · 2026-05-05T15:02:14Z

Summary

The request to merge from main and resolve conflicts on PR #2552 has been fully addressed:

Checklist

✅ Fetched the PR branch from the fork (Zheng-Lu/software-agent-sdk)
✅ Merged origin/main into fix/2467-image-downscale
✅ Resolved the single conflict in openhands-sdk/pyproject.toml:
- Kept main's updated litellm>=1.83.7 (replacing the PR's older litellm>=1.82.6,!=1.82.7,!=1.82.8)
- Preserved the PR's new pillow>=12.1.1 dependency (needed for image resizing)
✅ Verified no other conflicts — llm.py auto-merged cleanly
✅ Committed and pushed the merge to the fork remote

Conciseness

The changes are minimal — only the merge commit resolving one conflict. No extraneous changes were made.

xingyaoww

Thanks! Sorry for taking so long :(

Co-authored-by: openhands <openhands@all-hands.dev> Co-authored-by: Xingyao Wang <xingyao@all-hands.dev> Co-authored-by: Xingyao Wang <xingyaoww@gmail.com>

fix(sdk): resize Anthropic many-image inputs

19c998b

Co-authored-by: openhands <openhands@all-hands.dev>

Merge branch 'main' into fix/2467-image-downscale

b3f67af

Co-authored-by: openhands <openhands@all-hands.dev>

xingyaoww reviewed Mar 27, 2026

View reviewed changes

Zheng-Lu and others added 3 commits March 28, 2026 00:29

Merge branch 'OpenHands:main' into fix/2467-image-downscale

7dbe9e0

Merge branch 'OpenHands:main' into fix/2467-image-downscale

079183f

fix(sdk): handle Anthropic single-image limits

4d047ed

Co-authored-by: openhands <openhands@all-hands.dev>

Zheng-Lu requested a review from xingyaoww April 6, 2026 22:31

Zheng-Lu and others added 2 commits April 6, 2026 23:32

Merge branch 'OpenHands:main' into fix/2467-image-downscale

487da93

Merge branch 'main' into fix/2467-image-downscale

82447d2

xingyaoww reviewed Apr 10, 2026

View reviewed changes

Merge branch 'main' into fix/2467-image-downscale

aef2b73

xingyaoww reviewed Apr 16, 2026

View reviewed changes

Comment thread openhands-sdk/openhands/sdk/llm/utils/image_resize.py Outdated

Zheng-Lu and others added 2 commits April 19, 2026 00:08

fix(sdk): simplify image resize utilities

e2ed9aa

Co-authored-by: openhands <openhands@all-hands.dev>

Merge branch 'OpenHands:main' into fix/2467-image-downscale

7fce88f

Zheng-Lu requested a review from xingyaoww April 19, 2026 00:00

fix(sdk): require pillow for image resizing

64f0104

Co-authored-by: openhands <openhands@all-hands.dev>

Merge branch 'main' into fix/2467-image-downscale

5394fef

Merge branch 'main' into fix/2467-image-downscale

c400168

openhands-agent added 2 commits April 27, 2026 14:57

Merge remote-tracking branch 'origin/main' into fix/2467-image-downscale

813919d

xingyaoww removed the test-examples Run all applicable "examples/" files. Expensive operation. label Apr 27, 2026

xingyaoww added the test-examples Run all applicable "examples/" files. Expensive operation. label Apr 27, 2026 — with OpenHands AI

xingyaoww removed the test-examples Run all applicable "examples/" files. Expensive operation. label Apr 27, 2026

xingyaoww added the test-examples Run all applicable "examples/" files. Expensive operation. label Apr 27, 2026 — with OpenHands AI

xingyaoww removed the test-examples Run all applicable "examples/" files. Expensive operation. label Apr 27, 2026

xingyaoww added the test-examples Run all applicable "examples/" files. Expensive operation. label Apr 27, 2026 — with OpenHands AI

Merge branch 'main' into fix/2467-image-downscale

fc2bd09

xingyaoww added test-examples Run all applicable "examples/" files. Expensive operation. and removed test-examples Run all applicable "examples/" files. Expensive operation. labels Apr 27, 2026

Merge branch 'main' into fix/2467-image-downscale

2c4287e

Merge main into fix/2467-image-downscale, resolve pyproject.toml conf…

09e84fa

…lict Keep main's litellm>=1.83.7 and PR's pillow>=12.1.1 dependency. Co-authored-by: openhands <openhands@all-hands.dev>

xingyaoww added 2 commits May 5, 2026 11:17

fix(sdk): resize proxy Anthropic images

75390e7

chore: remove run examples workflow changes

1e420e2

xingyaoww approved these changes May 5, 2026

View reviewed changes

Merge branch 'main' into fix/2467-image-downscale

574aa42

xingyaoww enabled auto-merge (squash) May 5, 2026 15:38

Merge branch 'main' into fix/2467-image-downscale

172e0ee

xingyaoww merged commit 1d932c8 into OpenHands:main May 5, 2026
26 checks passed

jamiechicago312 mentioned this pull request May 6, 2026

SDK should auto-resize oversized images before sending to LLM providers #2467

Closed

Conversation

Zheng-Lu commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What Changed

Validation

Proof

Uh oh!

xingyaoww commented Mar 27, 2026

Uh oh!

openhands-ai Bot commented Mar 27, 2026

Uh oh!

xingyaoww left a comment

Choose a reason for hiding this comment

Taste Rating: 🟡 Acceptable — Works, but the structure needs improvement

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

openhands-ai Bot commented Mar 27, 2026

Uh oh!

xingyaoww commented Apr 10, 2026

Uh oh!

openhands-ai Bot commented Apr 10, 2026

Uh oh!

xingyaoww left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

openhands-ai Bot commented Apr 10, 2026

Summary

Checklist

Key Findings Posted

Uh oh!

all-hands-bot commented Apr 16, 2026

Uh oh!

Uh oh!

all-hands-bot commented Apr 22, 2026

Uh oh!

all-hands-bot commented Apr 26, 2026

Uh oh!

xingyaoww commented Apr 27, 2026

Uh oh!

openhands-ai Bot commented Apr 27, 2026

Uh oh!

openhands-ai Bot commented Apr 27, 2026

Uh oh!

openhands-ai Bot commented Apr 27, 2026

Summary

Was the request addressed?

Are the changes concise?

Remaining limitation

Uh oh!

all-hands-bot commented May 4, 2026

Uh oh!

xingyaoww commented May 5, 2026

Uh oh!

openhands-ai Bot commented May 5, 2026

Uh oh!

openhands-ai Bot commented May 5, 2026

Summary

Checklist

Conciseness

Uh oh!

xingyaoww left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Zheng-Lu commented Mar 23, 2026 •

edited

Loading