Conversation
820b376 to
411618a
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #508 +/- ##
==========================================
+ Coverage 75.21% 76.65% +1.44%
==========================================
Files 75 81 +6
Lines 6741 7252 +511
Branches 1860 2125 +265
==========================================
+ Hits 5070 5559 +489
- Misses 1660 1682 +22
Partials 11 11
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
fernandocorreia-galileo
left a comment
There was a problem hiding this comment.
@Murike please check if these auto-generated notes are valid.
Issues Found
-
[Major] Agent spans misclassified as workflow spans
- File:
src/handlers/openai-agents/span-mapping.ts(line 36-39) andsrc/handlers/openai-agents/index.ts(lines 370-379) mapSpanType()maps OpenAI's'agent'type to'workflow', and_logNodeTree()catches all non-llm/non-tool nodes in theelsebranch callingaddWorkflowSpan(). GalileoLogger has a distinctaddAgentSpan()method (withagentType,stepNumberparameters) that should be used for agent spans to preserve the Trace→Agent hierarchy in the Galileo platform.- Suggestion: In
mapSpanType(), map'agent'to'agent'instead of'workflow'. In_logNodeTree(), add a dedicated branch fornode.nodeType === 'agent'that callsthis._galileoLogger.addAgentSpan(). Update the test attracing-processor.test.ts:142accordingly.
- File:
-
[Major]
_firstInputis never populated- File:
src/handlers/openai-agents/index.ts(lines 79, 304-305) _firstInputis declared asnull, reset tonullinonTraceEnd, and checked in_logNodeTree— but never assigned a value anywhere. The trace-level input always falls back to the root node's input, which for agent traces is typically empty (since agent spans haveinput: '').- Suggestion: In
onSpanEnd(), after extracting span data, add:if (this._firstInput === null && node.spanParams.input) { this._firstInput = node.spanParams.input; }. This captures the first non-empty input (typically from the first LLM or tool span).
- File:
-
[Major]
parseUsagemisses standard OpenAI token detail paths- File:
src/handlers/openai-agents/data-extraction.ts(lines 44-53) - The function looks for reasoning/cached tokens under
usage.details.reasoning_tokensandusage.details.cached_tokens. However, the standard OpenAI API (and Agents SDK) usesusage.output_tokens_details.reasoning_tokensandusage.input_tokens_details.cached_tokens. The currentdetailsfallback is not the correct path, so reasoning and cached token counts will be zero for all spans. - Suggestion: Add lookups for
output_tokens_details.reasoning_tokensandinput_tokens_details.cached_tokensas the primary paths, falling back to the existingdetailsand top-level fields.
- File:
-
[Minor] GalileoCustomSpanData's
galileoSpanreference is never used- File:
src/handlers/openai-agents/index.ts(lines 158-159) andsrc/handlers/openai-agents/custom-span.ts - The
createGalileoCustomSpanData()function andGalileoCustomSpanDatainterface provide API to embed a Galileo span reference, butonSpanStart()mapsGALILEO_CUSTOM_TYPEto'workflow'and discards thegalileoSpanreference. The custom span becomes a generic workflow span with data fromextractWorkflowData(). Either wire thegalileoSpaninto the logging path or simplify the API (removegalileoSpanif it's not needed yet). - Suggestion: If this is intended for future use, add a TODO comment. If it should work now, extract
spanData.data.galileoSpaninonSpanStartand use it when logging the node.
- File:
-
[Minor] Orphan spans silently dropped
- File:
src/handlers/openai-agents/index.ts(lines 187-192) - If
parentIdpoints to a node not yet in_nodes, the new span is stored but never linked as a child. When_commitTracewalks the tree from root, orphaned spans are never visited and never logged. In normal SDK operation spans arrive in order, but out-of-order delivery (e.g., async agent flows) could cause silent data loss. - Suggestion: Add a fallback — if
parentNodeis not found, try linking to the trace root node. Add asdkLogger.warn()when this happens.
- File:
-
[Nit]
void _removedpattern for unused destructured variable- File:
src/handlers/openai-agents/index.ts(lines 225-226) const { _responseObject: _removed, ...rest } = finalData; void _removed;— the underscore-prefixed variable plusvoidis unusual. A cleaner pattern is justconst { _responseObject: _, ...rest } = finalData;or delete the key from the object.
- File:
-
[Nit] Dynamic import warning fires eagerly at module load
- File:
src/handlers/openai-agents/index.ts(lines 25-29) - The top-level
import('@openai/agents-core' as string).catch(...)runs as soon as the module is loaded, even if the consumer never intends to useGalileoTracingProcessor. This causes a spurious warning for users who import other Galileo features but don't use OpenAI Agents. - Suggestion: Move the availability check into the constructor or
registerGalileoTraceProcessor()so it only fires when actually used.
- File:
-
[Nit] Trace metadata — consider defensive filtering
- File:
src/handlers/openai-agents/index.ts(lines 101-108) - While trace metadata is user-provided (not SDK-injected credentials), it would be a good defensive practice to at least warn or document that all metadata is forwarded verbatim to Galileo.
- File:
Questions
- Is the GalileoCustomSpanData /
galileoSpanreference intended to be functional in this PR, or is it scaffolding for a follow-up? The current API is exposed publicly but does nothing with the embedded span. - Should the
agenttype inmapSpanTypemap to'agent'(usingaddAgentSpan()) or is the'workflow'mapping intentional for some Galileo backend reason? - The
_firstInputfield — was there a plan to populate it that got lost during development, or is the fallback-to-root-input behavior intentional?
… of token information.
…ng every SDK user for OpenAI Agents dependency.
FIXED - [Major] Agent spans misclassified as workflow spans FIXED - [Major] _firstInput is never populated FIXED - [Major] parseUsage misses standard OpenAI token detail paths FIXED - [Minor] GalileoCustomSpanData's galileoSpan reference is never used FIXED - [Minor] Orphan spans silently dropped ANSWERED - [Nit] void _removed pattern for unused destructured variable FIXED - [Nit] Dynamic import warning fires eagerly at module load ANSWERED - [Nit] Trace metadata — consider defensive filtering Questions FIXED - Is the GalileoCustomSpanData / galileoSpan reference intended to be functional in this PR, or is it scaffolding for a follow-up? The current API is exposed publicly but does nothing with the embedded span. |
User description
(TS SDK Parity Effort) Missing - OpenAI Agent support - sc-34519
Description
Generated description
Below is a concise technical summary of the changes proposed in this PR:
graph LR registerGalileoTraceProcessor_("registerGalileoTraceProcessor"):::added GalileoTracingProcessor_("GalileoTracingProcessor"):::added OPENAI_AGENTS_CORE_("OPENAI_AGENTS_CORE"):::added mapSpanType_("mapSpanType"):::added mapSpanName_("mapSpanName"):::added createNode_("createNode"):::added extractLlmData_("extractLlmData"):::added createGalileoCustomSpanData_("createGalileoCustomSpanData"):::added registerGalileoTraceProcessor_ -- "Registers and returns a new tracing processor instance." --> GalileoTracingProcessor_ registerGalileoTraceProcessor_ -- "Calls addTraceProcessor from @openai/agents-core to register processor." --> OPENAI_AGENTS_CORE_ GalileoTracingProcessor_ -- "Determines node type including galileo_custom sentinel handling." --> mapSpanType_ GalileoTracingProcessor_ -- "Derives human-readable span names for logging." --> mapSpanName_ GalileoTracingProcessor_ -- "Creates node entries with empty children stored in _nodes." --> createNode_ GalileoTracingProcessor_ -- "Normalizes LLM span payloads, tokens, responses for logging." --> extractLlmData_ GalileoTracingProcessor_ -- "Wraps Galileo span for injection into agent tracing." --> createGalileoCustomSpanData_ classDef added stroke:#15AA7A classDef removed stroke:#CD5270 classDef modified stroke:#EDAC4C linkStyle default stroke:#CBD5E1,font-size:13pxEnable OpenAI Agents support by adding the
GalileoTracingProcessor/GalileoCustomSpansuite plus node/span utilities so agent, LLM, tool, workflow, and Galileo custom spans are enriched, mapped, and emitted throughGalileoLoggerwith status, metadata, embedded tool, and input/output handling. Harden the workflow by exporting the new processors, updating logger typings, and wiring dependency/test coverage so the agent tracing flow works when the optional @openai/agents peer dependencies are present.@openai/agents/openaipeer deps are advertised while verifying the new tracing helpers across unit and integration scenarios.Modified files (9)
Latest Contributors(2)
GalileoTracingProcessor, node/span utilities, span mappers, Galileo custom span helpers, embedded tool extraction, and logger hooks so OpenAI Agent traces normalize inputs/outputs/statuses and flow throughGalileoLoggerwith metadata and embedded tool awareness.Modified files (9)
Latest Contributors(0)