-
Notifications
You must be signed in to change notification settings - Fork 421
Open
Labels
Milestone
Description
Description
“Time To First token” is an important and required metric to measure the model performance. We need to add this metric to AIM via LlmChatCompletionSummary.first_time_to_token.
- This metric should only be captured for streaming requests
- The metric name should be captured as
LlmChatCompletionSummary.time_to_first_token - The metric value should be stored as the number of milliseconds between the time when the request was issued and the first token is received
Pending Questions to Clarify
- Is this for all LLMs or just a select few?
- What is the default/error value?
0,undefined?
Acceptance Criteria
-
first_time_to_tokenis added to theLlmChatCompletionSummaryclass viathis['response.first_time_to_token'] - this attribute can be seen in the UI
- Appropriate tests are created
Additional context
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
Reviewed