feat(llmobs): capture inline base64 images for OpenAI chat completions and responses - #19690
feat(llmobs): capture inline base64 images for OpenAI chat completions and responses#19690joizddog wants to merge 1 commit into
Conversation
|
BenchmarksBenchmark execution time: 2026-08-13 23:08:12 Comparing candidate commit 1796508 in PR branch Found 0 performance improvements and 8 performance regressions! Performance is the same for 612 metrics, 10 unstable metrics. scenario:httppropagationextract-datadog_tracecontext_tracestate_not_propagated_on_trace_id_no_match
scenario:httppropagationinject-ids_only
scenario:iastaspects-lower_aspect
scenario:iastaspects-modulo_noaspect
scenario:iastaspectsospath-ospathbasename_aspect
scenario:span-start
scenario:telemetryaddmetric-1-count-metric-1-times
scenario:tracer-small
|
1796508 to
c358c4f
Compare
Codeowners resolved asResolved from the full PR diff against |
Circular import analysis
|
Dependency direction analysis
|
c358c4f to
47c2a73
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 47c2a7394b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if image_parts: | ||
| processed_message["image_parts"] = image_parts |
There was a problem hiding this comment.
Update the LLMObs integration skill alongside this new extraction pattern: .claude/skills/llmobs-integrations/SKILL.md currently demonstrates only AudioPart, while its references/implementation-guide.md message-extraction section documents only audio attachments. Leaving both references unchanged means future integration work will miss the new ImagePart capture and size-guard conventions.
AGENTS.md reference: AGENTS.md:L19-L19
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Skill file update remains blocked by classifier restrictions
|
|
||
|
|
||
| def test_extract_content_parts_captures_inline_image_bare_string(): | ||
| """The URL may arrive as a bare string rather than the nested ``image_url.url`` object.""" |
There was a problem hiding this comment.
Convert the double-backtick literals in this and the other newly added test docstrings to plain text. Test docstrings are explicitly non-rendered documentation in this repository, so adding rST markup such as image_url.url violates the documented editor-facing prose convention.
AGENTS.md reference: AGENTS.md:L42-L50
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Removed 15 remaining rST backticks from docstrings
| def _is_data_url(value: Any) -> bool: | ||
| """Whether value carries its payload inline as a data: URL rather than by reference.""" | ||
| return isinstance(value, str) and value[:32].lstrip().lower().startswith("data:") |
There was a problem hiding this comment.
When a Responses API image URL has 28 or more leading whitespace characters, slicing to 32 characters removes part of the data: scheme, so this predicate returns false. _capture_inline_image() then treats the value as a remote reference, and _extract_image_reference() returns the complete data URL into message text, reintroducing the multi-megabyte base64 leak this change is intended to prevent. Inspect the scheme after stripping whitespace rather than stripping only a fixed prefix.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed _is_data_url whitespace leak
| # Measure the payload's span before materializing it. Stripping wrapped-line whitespace only | ||
| # shrinks it, so a span past double the budget cannot fit however it was wrapped, and bailing |
There was a problem hiding this comment.
When a valid base64 image is heavily line-wrapped, its raw payload span can exceed twice the 4 MiB budget even though removing permitted whitespace leaves less than 4 MiB of encoded content. This shortcut returns the too-large marker before performing that normalization, contradicting the regex's stated support for wrapped base64 and dropping an image that actually fits the guard. Count non-whitespace payload characters before deciding that the image is oversized.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Raised pre-filter limit from 2x to 8x to prevent false rejections of wrapped base64 payloads.
7f91360 to
71a2d6c
Compare
Inline `data:image/...;base64,...` content is captured as typed `image_parts` on LLM span messages for both Chat Completions and the Responses API. Three paths previously wrote a raw base64 payload into span text instead: - the Responses API concatenated the whole data URL into the message content, which also dropped an image-only message because no text remained; - a reusable-prompt variable holding a data URL recorded it verbatim; - image-generation results and computer-use screenshots fell through to a `str(item)` fallback that renders every field of the SDK model, base64 included. Each now degrades to a marker, or to the remote reference where one exists. The catch-all is bounded so a future output item carrying binary cannot leak. The size guard mirrors the audio helper and shares its base64 length calculation, and derives its budget from the configured event-size limit so a lowered `DD_LLMOBS_EVENT_SIZE_BYTES` cannot admit an oversize image. An oversize image is left as a marker so the surrounding text and the model response survive. Remote URLs and file_ids are not fetched. MLOB-6408
71a2d6c to
a528b61
Compare
1 participant
Description
The OpenAI integration now captures inline base64 image inputs as typed
image_partson LLM spanmessages, for both Chat Completions and the Responses API, so image inputs render in LLM
Observability. Builds on the merged
image_partsfoundation (#18809) and emits the same wire shapedd-trace-js accepts as of DataDog/dd-trace-js#9684.
data:image/…;base64,…URLs intoImageParts ({mime_type, content}). Textand other content parts are untouched. The backend offloads the inline bytes to an attachment.
file_idare not fetched and keep theirexisting reference text.
[image omitted: too large]. Without it, one oversized image pushes the event past the 5 MB limitand
_writer._truncate_span_eventblanks the span's entire input and output — losing the prompttext and the model response along with the image. The budget is derived from
config._llmobs_event_size_limit, so a loweredDD_LLMOBS_EVENT_SIZE_BYTEScannot admit an imagelarger than the active event allowance.
Because the two capture sites are shared helpers, the OpenAI Agents SDK, LiteLLM, Azure OpenAI and
streaming inherit this with no integration-specific code.
Three base64-into-span-text leaks fixed
The two APIs failed differently, and neither was benign:
image_generation_call(raw base64result) andcomputer_call_output(screenshotimage_url) fell through to a genericstr(item), which onthe SDK's pydantic models renders every field value. That catch-all is now bounded, so a future
output item carrying binary cannot leak the same way.
Each degrades to a marker, or to the remote reference where one exists.
Scoped to OpenAI; the Anthropic half is #19148.
Testing
Deterministic, no credentials:
Unit coverage for the guard (at cap, over cap, encoded sizing for
bytes, string measured directly,empty/no-mime), capture (single, multiple, bare-string URL,
;charset=, uppercase scheme,svg+xml, line-wrapped payload), non-capture (remote URL,file_id, empty payload, whitespace-onlypayload, non-base64 payload, non-image mime, missing URL), and both output-item leaks. Two
invariants are pinned explicitly: an unparseable data URL never reaches the caller's reference text,
and leading whitespace of any length cannot hide the scheme. Verified on openai 2.46.0 and 1.66.0.
Precise scope of the oversize E2E test: the openai conftest replaces the span writer with a mock, so
it does not exercise
_truncate_span_event. It proves the payload never entersmeta.input.messagesand that the text and response survive.Not covered: a Responses-path VCR cassette; live Agents SDK / LiteLLM / Azure / streaming with
images — that inheritance is established by call graph, not by test.
Before / After
Verified live on staging (Org 2) via the AI Gateway — the same script and the same image run twice,
swapping only the ddtrace build. BEFORE is released
ddtrace 4.13.0; AFTER carriesgit.commit.shaof the commit under review.1. Chat Completions — BEFORE
trace
6a7f6e53…0d2729cemeta.input.messages[0].content="What is shown in this image? One short sentence.\n[image]"—56 chars, image bytes discarded.
2. Chat Completions — AFTER
trace
6a7f6f65…7557fb2fimage_parts=[{attachment_key: "input_message_image_0_0", mime_type: "image/png"}], contentback to the 48-char question — the image renders inline in the span panel.
3. Responses API — BEFORE
Same trace as (1), sibling span.
meta.input.messages[0].content= 15,902 chars — the questionfollowed by the entire base64 payload (
…sentence.data:image/png;base64,iVBORw0KGgo…).4. Responses API — AFTER
trace
6a7f6f65…7557fb2f48 chars plus
image_parts— a 331× reduction in recorded message text for the same request.A remote-URL image was run as a control and keeps its reference text in both, confirming the change
is scoped to inline bytes.
Risks
Low / additive. Text-only and non-image content are unchanged; capture is bytes-only with no network
fetch.
One risk worth stating plainly: for Chat Completions this introduces a size-drop mode that did not
exist before. An image was always
[image], so no chat span could be dropped for image size. Nowseveral in-budget inline images can exceed the per-event limit and cost the span its whole input and
output.
Known limitations:
has its own independent budget — can still collectively exceed the event limit. Pinned by
test_multiple_in_budget_images_can_still_exceed_the_event_size_limit.function_call_outputtool results remain uncaptured.images.generate/edit/create_variationare patchedfor APM but never call
llmobs_set_tags, so no LLMObs span exists;image_generation_callnowemits a marker rather than an
image_part.event also rides the shared 20 MB trace-writer buffer; that interaction is unmeasured.
Additional Notes
A
DD_-prefixed kill-switch was considered and declined. Unlike the realtime audio case, whichintroduced a whole streaming state machine, this is a few dozen lines inside already-patched code
paths; a permanent public config surface is not warranted. To suppress image bytes today, strip
image_partsin aLLMObs.register_processorhook. Straightforward to add if reviewers disagree.Motivating datapoint: the AI Gateway's own LLMObs spans, on released ddtrace, currently carry the
full base64 payload of every image routed through it — this is not a synthetic concern.
MLOB-6408