Skip to content

Commit a2e6fa4

Browse files
B-Step62CopilotBenWilson2
authored
Fix contents in tracing doc pages (mlflow#18750)
Signed-off-by: B-Step62 <[email protected]> Signed-off-by: Yuki Watanabe <[email protected]> Signed-off-by: Ben Wilson <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Ben Wilson <[email protected]>
1 parent e596512 commit a2e6fa4

File tree

10 files changed

+247
-1169
lines changed

10 files changed

+247
-1169
lines changed

‎docs/docs/genai/tracing/attach-tags/index.mdx‎

Lines changed: 17 additions & 344 deletions
Large diffs are not rendered by default.

‎docs/docs/genai/tracing/collect-user-feedback/index.mdx‎

Lines changed: 67 additions & 173 deletions
Original file line numberDiff line numberDiff line change
@@ -1,202 +1,87 @@
1-
import FeatureHighlights from "@site/src/components/FeatureHighlights";
2-
import ConceptOverview from "@site/src/components/ConceptOverview";
1+
import { APILink } from "@site/src/components/APILink";
32
import TilesGrid from "@site/src/components/TilesGrid";
43
import TileCard from "@site/src/components/TileCard";
54
import ImageBox from "@site/src/components/ImageBox";
6-
import { ThumbsUp, BarChart3, Shield, Target, MessageSquare, FileText, Search, TrendingUp, Clock, Users, Zap, Database, Lock } from "lucide-react";
7-
import AssessmentsTraceDetailImageUrl from '@site/static/images/assessments/assessments_trace_detail_ui.png';
8-
import AddFeedbackImageUrl from '@site/static/images/assessments/add_feedback_ui.png';
9-
import AdditionalFeedbackImageUrl from '@site/static/images/assessments/additional_feedback_ui.png';
5+
import { BarChart3, Target, FileText, Search } from "lucide-react";
106

117
# Collect User Feedback
128

13-
Capturing user feedback is critical for understanding the real-world quality of your GenAI application. MLflow's **Feedback API** provides a structured, standardized approach to collecting, storing, and analyzing user feedback directly within your traces.
14-
15-
## Why Use MLflow Feedback for User Feedback?
16-
17-
<FeatureHighlights features={[
18-
{
19-
icon: Target,
20-
title: "Direct Trace Integration",
21-
description: "Feedback is linked directly to specific application executions, creating an immediate connection between user reactions and system performance."
22-
},
23-
{
24-
icon: Shield,
25-
title: "Structured Data Model",
26-
description: "Standardized format with clear attribution and rationale ensures consistent feedback collection across your entire application."
27-
},
28-
{
29-
icon: BarChart3,
30-
title: "Production Ready",
31-
description: "Available in OSS MLflow 3.2.0+ with no external dependencies, designed for high-throughput production environments."
32-
},
33-
{
34-
icon: ThumbsUp,
35-
title: "Complete Audit Trail",
36-
description: "Track every feedback change with timestamps and user attribution, enabling comprehensive quality analysis over time."
37-
}
38-
]} />
39-
40-
## Step-by-Step Guide: Collecting User Feedback
41-
42-
### 1. Set Up Your GenAI Application with Tracing
43-
44-
First, create a simple application that automatically generates traces using MLflow's OpenAI autologging:
9+
Capturing user feedback is critical for understanding the real-world quality of your GenAI application. MLflow's Feedback API provides a structured, standardized approach to collecting, storing, and analyzing user feedback directly within your traces.
10+
11+
## Adding Feedback with MLflow UI
12+
13+
<ImageBox src="/images/llms/tracing/logging-feedback.gif" alt="Add Feedback" />
14+
15+
## Adding Feedback with API
16+
17+
To annotate traces with feedback programmatically, use the <APILink fn="mlflow.log_feedback" /> API.
4518

4619
```python
4720
import mlflow
4821
from mlflow.entities import AssessmentSource, AssessmentSourceType
49-
import openai
50-
51-
# Enable automatic tracing for OpenAI calls
52-
mlflow.openai.autolog()
53-
54-
# Initialize your LLM client
55-
client = openai.OpenAI()
56-
57-
58-
def ask_question(question):
59-
"""Simple Q&A application with automatic tracing."""
60-
# This call is automatically traced by MLflow
61-
response = client.chat.completions.create(
62-
model="gpt-4o-mini",
63-
messages=[
64-
{
65-
"role": "system",
66-
"content": "You are a helpful assistant. Answer questions clearly and concisely.",
67-
},
68-
{"role": "user", "content": question},
69-
],
70-
temperature=0.7,
71-
)
72-
73-
answer = response.choices[0].message.content
74-
return answer
75-
76-
77-
# Generate some traces - each call creates a trace automatically
78-
question = "What is machine learning?"
79-
answer = ask_question(question)
80-
print(f"Question: {question}")
81-
print(f"Answer: {answer}")
82-
83-
# You can get the trace ID from the MLflow UI or search API
84-
# For this example, we'll show how to collect feedback programmatically
22+
23+
mlflow.log_feedback(
24+
trace_id="<your trace id>",
25+
name="user_satisfaction",
26+
value=True,
27+
rationale="User indicated response was helpful",
28+
source=AssessmentSource(
29+
source_type=AssessmentSourceType.HUMAN, source_id="user_123"
30+
),
31+
)
8532
```
8633

87-
### 2. Collect Simple Thumbs Up/Down Feedback
34+
If you have a `Feedback` object already (e.g., a response from LLM-as-a-Judge), you can log it
35+
directly using the <APILink fn="mlflow.log_assessment" /> API. This is equivalent to using the
8836

89-
Implement basic boolean feedback collection. In a real application, you'd get the trace_id from your tracing system:
37+
<APILink fn="mlflow.log_feedback" /> API with unpacked fields.
9038

9139
```python
92-
def collect_thumbs_feedback(trace_id, is_helpful, user_id):
93-
"""Collect simple thumbs up/down feedback from users."""
94-
mlflow.log_feedback(
95-
trace_id=trace_id,
96-
name="user_satisfaction",
97-
value=is_helpful,
98-
rationale="User indicated response was helpful"
99-
if is_helpful
100-
else "User indicated response was not helpful",
101-
source=AssessmentSource(
102-
source_type=AssessmentSourceType.HUMAN, source_id=user_id
103-
),
104-
)
105-
print(f"✓ Feedback recorded: {'👍' if is_helpful else '👎'}")
106-
107-
108-
# Example: Collect feedback on a trace
109-
trace_id = mlflow.get_last_active_trace_id()
110-
collect_thumbs_feedback(trace_id, True, "user_123")
40+
import mlflow
41+
from mlflow.genai.judges import make_judge
42+
from typing import Literal
43+
44+
coherence_judge = make_judge(
45+
name="coherence",
46+
instructions=(
47+
"Evaluate if the response is coherent, maintaining a constant tone "
48+
"and following a clear flow of thoughts/concepts"
49+
"Trace: {{ trace }}\n"
50+
),
51+
feedback_value_type=Literal["coherent", "somewhat coherent", "incoherent"],
52+
model="anthropic:/claude-opus-4-1-20250805",
53+
)
54+
55+
trace = mlflow.get_trace("<your trace id>")
56+
feedback = coherence_judge(trace=trace)
57+
58+
mlflow.log_assessment(trace_id="<your trace id>", assessment=feedback)
59+
# Equivalent to log_feedback(trace_id="<trace_id>", name=feedback.name, value=feedback.value, ...)"
11160
```
11261

113-
### 3. View Feedback in MLflow UI
114-
115-
After collecting feedback, you can view it in the MLflow UI:
116-
117-
<ImageBox
118-
src={AssessmentsTraceDetailImageUrl}
119-
alt="Feedback in MLflow UI"
120-
width="90%"
121-
/>
122-
123-
The trace detail page shows all feedback attached to your traces, making it easy to analyze user satisfaction and identify patterns in your application's performance.
124-
125-
### 4. Adding and Updating Feedback via UI
126-
127-
Users can also provide feedback directly through the MLflow UI:
128-
129-
**Creating New Feedback:**
62+
## Supported Value Types
13063

131-
<ImageBox
132-
src={AddFeedbackImageUrl}
133-
alt="Create Feedback"
134-
width="90%"
135-
/>
136-
137-
**Adding Additional Feedback:**
138-
139-
<ImageBox
140-
src={AdditionalFeedbackImageUrl}
141-
alt="Additional Feedback"
142-
width="90%"
143-
/>
64+
MLflow feedback supports various formats to match your application's needs:
14465

145-
This collaborative approach enables both programmatic feedback collection and manual review workflows.
66+
| Feedback Type | Description | Example Use Cases |
67+
| ------------- | ------------------------------ | ----------------------------------- |
68+
| **Boolean** | Simple `True`/`False` feedback | Thumbs up/down, correct/incorrect |
69+
| **Numeric** | Integer or float ratings | 1-5 star ratings, confidence scores |
70+
| **Text** | Free-form text feedback | Detailed quality breakdowns |
14671

147-
## Feedback Value Types
72+
## Supported Feedback Sources
14873

149-
MLflow feedback supports various formats to match your application's needs:
74+
The `source` field of the feedback provides information about where the feedback came from.
15075

151-
| Feedback Type | Description | Example Use Cases |
152-
| --------------- | ---------------------------------------- | ----------------------------------- |
153-
| **Boolean** | Simple `True`/`False` feedback | Thumbs up/down, correct/incorrect |
154-
| **Numeric** | Integer or float ratings | 1-5 star ratings, confidence scores |
155-
| **Categorical** | String classifications | "Helpful", "Neutral", "Unhelpful" |
156-
| **Structured** | Complex objects with multiple dimensions | Detailed quality breakdowns |
157-
158-
## MLflow Feedback Collection Best Practices
159-
160-
<ConceptOverview concepts={[
161-
{
162-
icon: TrendingUp,
163-
title: "Start with Boolean Feedback",
164-
description: "Use MLflow's boolean feedback type for simple thumbs up/down collection. Once you analyze patterns with MLflow's search APIs, expand to numeric ratings or structured feedback types."
165-
},
166-
{
167-
icon: Clock,
168-
title: "Link Feedback to Fresh Traces",
169-
description: "Collect feedback immediately after trace generation when the interaction context is available. MLflow's direct trace-feedback linkage ensures you always have the full execution context."
170-
},
171-
{
172-
icon: Database,
173-
title: "Use Consistent Naming Conventions",
174-
description: "Standardize feedback names like 'user_satisfaction' or 'quality_rating' across traces. This enables MLflow's search and aggregation features to provide meaningful insights across your application."
175-
},
176-
{
177-
icon: Lock,
178-
title: "Use Source Attribution Properly",
179-
description: "Set meaningful source_id values in AssessmentSource objects for tracking feedback providers. MLflow preserves complete audit trails with timestamps and source attribution."
180-
},
181-
{
182-
icon: Users,
183-
title: "Combine Programmatic and UI Collection",
184-
description: "Use MLflow's API for automated collection and the UI for manual review. Both methods integrate seamlessly, allowing different teams to contribute feedback through their preferred interface."
185-
}
186-
]} />
76+
| Source Type | Description | Example Use Cases |
77+
| ------------- | --------------------- | ---------------------------------------- |
78+
| **HUMAN** | Human feedback | User thumbs up/down, correct/incorrect |
79+
| **LLM_JUDGE** | LLM-based feedback | Score traces with an LLM-based judge |
80+
| **CODE** | Programmatic feedback | Score traces with a programmatic checker |
18781

18882
## Next Steps
18983

19084
<TilesGrid>
191-
<TileCard
192-
icon={MessageSquare}
193-
iconSize={48}
194-
title="Feedback API Guide"
195-
description="Comprehensive guide to all feedback APIs with advanced patterns and examples"
196-
href="/genai/assessments/feedback"
197-
linkText="View API reference →"
198-
containerHeight={64}
199-
/>
20085
<TileCard
20186
icon={Target}
20287
iconSize={48}
@@ -215,4 +100,13 @@ MLflow feedback supports various formats to match your application's needs:
215100
linkText="Start analyzing →"
216101
containerHeight={64}
217102
/>
103+
<TileCard
104+
icon={BarChart3}
105+
iconSize={48}
106+
title="Evaluate Traces"
107+
description="Learn how to evaluate traces with feedback data and analyze patterns for quality insights"
108+
href="/genai/eval-monitor"
109+
linkText="Start evaluating →"
110+
containerHeight={64}
111+
/>
218112
</TilesGrid>

0 commit comments

Comments
 (0)