Bug: Vertex AI incorrectly rejecting ~2MB requests with 413 '32MB limit exceeded' error

[abhiwebshar]

Bug Report: Vertex AI incorrectly rejecting requests well under 32MB limit with 413 error

Summary

Vertex AI is incorrectly rejecting requests with 413 "Request exceeds the maximum allowed number of bytes. The maximum request size is 32 MB" errors when the actual request size is only ~2MB, well under the documented 32MB limit.

Environment

• SDK Version: anthropic-sdk-python (latest)
• Model: claude-sonnet-4@20250514
• API: Vertex AI (not direct Anthropic API)
• Region: us-east5-aiplatform.googleapis.com, asia-east1-aiplatform.googleapis.com
• Platform: Python 3.11, macOS

Bug Description

Issue

When sending requests to Claude Sonnet 4 via Vertex AI with document payloads of ~800KB - 2MB, we consistently get:

Error code: 413 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'Prompt is too long'}, 'request_id': 'req_vrtx_...'}

Evidence This is a Bug

1. Request size is well under limit: Our payloads are ~2MB, far below the documented 32MB limit
2. Inconsistent behavior: Similar payloads work fine with prompt caching enabled
3. Rate limit test works: We successfully processed 376,926 cached tokens without issues
4. Direct API comparison needed: This appears to be Vertex AI specific

Actual vs Expected Behavior

Expected: Requests under 32MB should be accepted
Actual: Requests as small as 2MB are rejected with 413 errors

Reproduction Details

Working Case (Cache Rate Limit Test)

# This works fine - 15 documents, ~400KB total
documents = load_test_documents(15)
response = client.messages.create(
    model="claude-sonnet-4@20250514",
    max_tokens=100,
    temperature=0,
    system=system_prompt,
    messages=[{"role": "user", "content": content_blocks}]
)
# Result: 100% success, 376,926 tokens from cache

Failing Case (Direct Document Analysis)

# This fails with 413 - 32 documents, ~2MB total
documents = load_documents(32)  # ~800KB-2MB payload
response = client.messages.create(
    model="claude-sonnet-4@20250514",
    max_tokens=8000,
    temperature=0,
    system=system_prompt,
    messages=[{"role": "user", "content": content_blocks}]
)
# Result: 413 "Request exceeds 32MB limit" - but payload is only ~2MB!

Test Results

From our payload progression test:

Document Count	Payload Size	Result
5 docs	~147KB	❌ 413 Error
10 docs	~272KB	❌ 413 Error
15 docs	~384KB	❌ 413 Error
32 docs	~808KB	❌ 413 Error

All payloads are well under 32MB but getting rejected

Expected Fix

Vertex AI should accept requests up to the documented 32MB limit, not reject requests at ~2MB.

Impact

This bug prevents legitimate use of Claude Sonnet 4's large context capabilities via Vertex AI, forcing workarounds or fallback to direct Anthropic API.

Workaround

Currently using progressive caching strategies, but this shouldn't be necessary for payloads well under the documented limits.

Additional Context

• Progressive caching works (suggests the issue is with direct document analysis)
• Same code works on direct Anthropic API (suggests Vertex AI specific issue)
• Error message claims 32MB limit but rejects much smaller payloads
• Consistent across multiple regions (us-east5, asia-east1)

Request

Please investigate Vertex AI's payload size validation logic and fix the incorrect rejection of requests well under the 32MB limit.

[abhiwebshar]

🔬 COMPREHENSIVE ROOT CAUSE ANALYSIS

TL;DR: Vertex AI incorrectly reports Citations API rate limit violations as 413 payload errors instead of proper 429 rate limit errors, making debugging extremely difficult.

---

📊 Complete Test Matrix

We conducted systematic testing across both APIs with identical payloads:

Test Case	Vertex AI Result	Direct API Result	Payload Size
Basic documents (32 files)	✅ SUCCESS (188k tokens)	❌ 429 rate_limit_error	~0.77 MB
Citations API (32 files)	❌ 413 'Prompt is too long'	❌ 429 rate_limit_error	~0.77 MB
Prompt Caching (32 files)	✅ SUCCESS (187k cached)	Not tested	~0.77 MB

---

🎯 Key Findings

1. Vertex AI Error Misreporting (Primary Bug)

Issue: Vertex AI incorrectly maps rate limit violations to 413 payload errors

• Expected: Citations API failure should return
• Actual: Returns
• Impact: Developers waste time optimizing payload size instead of rate limits

2. Citations API Resource Overhead (Secondary Issue)

Issue: Citations API document format requires significantly more processing

• Basic text format: Works fine on Vertex AI (188k tokens)
• Citations format: Triggers rate limits with same content (197k tokens)
• Overhead: ~9k token equivalent in processing load

3. Vertex AI Actually More Permissive

Surprising finding: Vertex AI handles larger payloads better than Direct API

• Vertex AI: Processes 188k basic tokens successfully
• Direct API: Rate limited even with basic documents (~197k tokens)

---

🔬 Technical Details

Test Environment

• Model: claude-sonnet-4@20250514 (Vertex AI), claude-3-5-sonnet-20241022 (Direct)
• Region: asia-east1-aiplatform.googleapis.com
• SDK: anthropic-sdk-python (latest)
• Payload: 32 markdown documents, 806,299 bytes (0.77 MB)

Citations API Format That Triggers Issue

{
  "type": "document",
  "source": {
    "type": "text",
    "media_type": "text/plain", 
    "data": "document content here"
  },
  "title": "Document Title",
  "citations": {"enabled": true}
}

Basic Format That Works Fine

{
  "type": "text",
  "text": "document content here"
}

---

🔧 Reproduction Steps

Step 1: Test Basic Documents on Vertex AI (Should Work)

content_blocks = [{"type": "text", "text": doc_content} for doc_content in documents]
response = vertex_client.messages.create(
    model="claude-sonnet-4@20250514",
    max_tokens=500,
    messages=[{"role": "user", "content": content_blocks}]
)
# Result: SUCCESS with ~188k tokens

Step 2: Test Citations API on Vertex AI (Should Fail with 413)

content_blocks = [{
    "type": "document",
    "source": {"type": "text", "media_type": "text/plain", "data": doc},
    "title": f"Doc {i}",
    "citations": {"enabled": True}
} for i, doc in enumerate(documents)]

response = vertex_client.messages.create(
    model="claude-sonnet-4@20250514", 
    max_tokens=500,
    messages=[{"role": "user", "content": content_blocks}]
)
# Result: 413 'Prompt is too long' (INCORRECT - should be 429)

Step 3: Test Citations API on Direct API (Should Fail with 429)

# Same exact content_blocks as Step 2
response = direct_client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=500, 
    messages=[{"role": "user", "content": content_blocks}]
)
# Result: 429 rate_limit_error (CORRECT)

---

🚀 Recommended Fixes

1. Fix Error Reporting (High Priority)

Vertex AI should return consistent error codes with Direct API:

• Current: Citations API rate limits → 413 'Prompt is too long'
• Should be: Citations API rate limits → 429 'rate_limit_error'

2. Document Citations API Overhead (Medium Priority)

Update documentation to indicate Citations API format has higher processing overhead:

• Requires ~5% more processing per token
• May hit rate limits sooner than basic text format
• Recommend progressive batching for large document sets

3. Consistent Rate Limit Implementation (Low Priority)

Consider aligning rate limits between Vertex AI and Direct API for consistent developer experience.

---

💡 Developer Impact

This bug causes significant developer confusion:

• ❌ Developers optimize for wrong constraint (payload size vs rate limits)
• ❌ Unnecessary chunking strategies implemented
• ❌ Time wasted debugging non-existent payload issues
• ✅ Workaround: Progressive caching/batching resolves the underlying rate limit issue

---

📋 Test Data Available

We have complete test scripts and data available demonstrating:

• Exact payload sizes and token counts
• Identical requests showing different error codes
• Working workarounds using progressive processing

Happy to provide additional technical details or test data as needed.

[abhiwebshar]

Hi @andrewedstrom can you check this? anthropic vertexai gives 403 error, at around 200k tokens. There is no way to use claude-sonnet-4 1m context window.