-
Notifications
You must be signed in to change notification settings - Fork 419
Open
Labels
Milestone
Description
Description
“Time To First token” is an important and required attribute to measure the model performance. We need to add this data to AIM via LlmChatCompletionSummary.first_time_to_token.
- This attribute should only be captured for streaming requests
- The attribute name should be captured as
LlmChatCompletionSummary.time_to_first_token - The attribute value should be stored as the number of milliseconds between the time when the request was issued and the first token is received
Pending Questions to Clarify
- Is this for all LLMs or just a select few?
- What is the default/error value?
0,undefined?
Acceptance Criteria
-
first_time_to_tokenis added to theLlmChatCompletionSummaryclass viathis['response.first_time_to_token'] - this attribute can be seen in the UI
- Appropriate tests are created
Additional context
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
Triage Needed