tool call tracing #40518
tool call tracing #40518
Conversation
#40023) * fixing a bug with agent tracing causing event hander return values to no be returned * code checker tool related changes --------- Co-authored-by: Marko Hietala <[email protected]>
* Show the reference in sample * Fix * Fixes * Fix
* Regeerate python code to use extensible connection types * Regenerate from upstream
* fixing tool call tracing * review related changes and code checker tool related changes --------- Co-authored-by: Marko Hietala <[email protected]>
Co-authored-by: Darren Cohen <[email protected]>
Co-authored-by: Marko Hietala <[email protected]>
M-Hietala
requested review from
luigiw,
needuv,
paulshealy1,
singankit,
dargilco and
jhakulin
as code owners
|
API change check APIView has identified API level changes in this PR and created following API reviews. |
| @@ -42,6 +42,13 @@ | |||
| GEN_AI_USAGE_OUTPUT_TOKENS = "gen_ai.usage.output_tokens" | |||
| GEN_AI_SYSTEM_MESSAGE = "gen_ai.system.message" | |||
| GEN_AI_EVENT_CONTENT = "gen_ai.event.content" | |||
| GEN_AI_CREATED_AT = "gen_ai.created_at" | |||
There was a problem hiding this comment.
this attribute is not defined in the conventions. If you think it's important to record, we'll need to find a descriptive name - is it a message creation time?
There was a problem hiding this comment.
it is the tool call and message creation creation time, and then there is corresponding complted/cancelled/failed at
| @@ -42,6 +42,13 @@ | |||
| GEN_AI_USAGE_OUTPUT_TOKENS = "gen_ai.usage.output_tokens" | |||
| GEN_AI_SYSTEM_MESSAGE = "gen_ai.system.message" | |||
| GEN_AI_EVENT_CONTENT = "gen_ai.event.content" | |||
| GEN_AI_CREATED_AT = "gen_ai.created_at" | |||
| GEN_AI_COMPLETED_AT = "gen_ai.completed_at" | |||
There was a problem hiding this comment.
is it about run ?
if you think we should capture it, let's call it gen_ai.run | run_step.end.timestamp
There was a problem hiding this comment.
these are reported for the tool calls and message creation
There was a problem hiding this comment.
changed these to
gen_ai.run_step.start.timestamp
gen_ai.run_step.end.timestamp
gen_ai.run_step.cancel.timestamp
gen_ai.run_step.fail.timestamp
sdk/ai/azure-ai-projects/azure/ai/projects/telemetry/agents/_ai_agents_instrumentor.py
Outdated
Show resolved
Hide resolved
| completed_at=step.completed_at, | ||
| cancelled_at=step.cancelled_at, | ||
| failed_at=step.failed_at, | ||
| run_step_last_error=step.last_error, |
There was a problem hiding this comment.
I think it's not a string in OpenAI - https://platform.openai.com/docs/api-reference/run-steps/step-object#run-steps/step-object-last_error
There was a problem hiding this comment.
the code should go to error.type and the message should go to error.string
| } | ||
| else: | ||
| try: | ||
| step_details = str(step["step_details"]["tool_calls"]) |
There was a problem hiding this comment.
should we do json dumps instead?
There was a problem hiding this comment.
that causes an error
| tool_call = { | ||
| "id": t.id, | ||
| "type": t.type, | ||
| "tool_call_details": step_details, |
There was a problem hiding this comment.
| "tool_call_details": step_details, | |
| t.type: step_details, |
I believe the patterns is
"type" : {tool name}
{tool name}: {details}
so while we don't know the structure for arbitrary tool, we know the name, let's use it instead of generic ``tool_call_details`
| GEN_AI_FAILED_AT, | ||
| GEN_AI_RUN_STEP_STATUS, | ||
| GEN_AI_RUN_STEP_LAST_ERROR, | ||
| GEN_AI_RUN_STEP_DETAILS, |
There was a problem hiding this comment.
it's not used, can you please remove it along with the definition?
|
|
||
| if created_at: | ||
| created_at_string = created_at.strftime("%Y-%m-%d %H:%M:%S.%f") | ||
| attrs[GEN_AI_CREATED_AT] = created_at_string |
There was a problem hiding this comment.
the conventional way to record timestamps in otel is ISO 8601, if we end up recording this attribute, let's use it
(and similar for other attributes)
Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines