OpenAI Responses image input succeeds via raw HTTP but fails via Symfony AI Agent->call() with Object not found

[tacman]

Draft Issue: Symfony AI Responses Image Input Fails While Raw OpenAI Call Succeeds

Suggested repo: symfony/ai

Title

OpenAI Responses image input succeeds via raw HTTP but fails via Symfony AI Agent->call() with Object not found

Summary

When using Symfony AI with the OpenAI Responses API and image input, a request that succeeds via direct HTTP fails via Agent->call() with:

Object not found

The same public image URL and the same model work correctly through a raw curl request to the OpenAI Responses API.

Packages

Observed with:

• symfony/ai-platform v0.8.1
• symfony/ai-open-ai-platform v0.8.0
• symfony/ai-bundle v0.8.0
• symfony/ai-agent v0.8.0

What Works

A direct call to the OpenAI Responses API succeeds:

curl https://api.openai.com/v1/responses \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.4",
    "input": [{
      "role": "user",
      "content": [
        {"type": "input_text", "text": "Briefly describe this image."},
        {"type": "input_image", "image_url": "https://ssai.fsn1.your-objectstorage.com/tac-shack/001/derivatives/0001.jpg"}
      ]
    }]
  }'

This returns a normal completed response with error: null.

What Fails

The equivalent request through Symfony AI fails in the OpenAI bridge/result path. The OpenAI HTTP request returns 200, but Symfony surfaces:

Object not found

Stack trace included:

#0 Symfony\Component\HttpClient\Response\AsyncResponse::stream()
#1 Symfony\Component\HttpClient\Response\CommonResponseTrait::...
#2 Symfony\Component\HttpClient\Response\AsyncResponse::initialize()
#3 Symfony\AI\Platform\Bridge\OpenAi\Gpt\ResultConverter->convert()
#4 Symfony\AI\Platform\Result\DeferredResult->getResult()
#5 Symfony\AI\Agent\Agent->call()

Important Observations

• The public image URL is valid and reachable by OpenAI.
• The same image URL succeeds with raw curl.
• This was reproducible on production.
• Local behavior differed from production during debugging, but the core symptom remained that Symfony AI collapsed the provider-side failure to Object not found.
• Removing structured response_format was not sufficient to resolve the issue.

Expected Behavior

One of:

1. Symfony AI should successfully handle the response when the equivalent raw OpenAI Responses API call succeeds.
2. If the provider returns an application-level error object or unexpected payload, Symfony AI should expose the raw provider error payload instead of collapsing it to Object not found.

Actual Behavior

• OpenAI HTTP request logs as 200
• Symfony AI then throws Object not found
• Production enrichment fails even though the underlying raw OpenAI call works

Why This Was Hard To Debug

The failure mode hides the most useful information:

• raw HTTP status appears successful
• the provider response body is not surfaced clearly
• the thrown message is too generic to distinguish:
  • provider-side application error
  • payload shape mismatch
  • bridge conversion issue

Workaround

We bypassed Symfony AI for the affected image-enrichment path and called the OpenAI Responses API directly via HttpClientInterface, then parsed the returned text/JSON ourselves.

That workaround succeeded against the same production image URL and same OpenAI key.

Happy to post that code if helpful.

Follow-Up Locally

Track internally:

• keep direct-call workaround documented
• keep package versions recorded per release
• retest when new symfony/ai releases land