You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Rewrite chapters 4-1 and 4-2 with consistent conversational style
- Remove all code samples as requested
- Match the personal, experience-driven narrative style of other chapters
- Add real-world stories and specific metrics throughout
- Focus on practical lessons learned rather than theoretical concepts
- Maintain consistent voice with 'I' and 'you' throughout
- Include specific business outcomes and ROI examples
- Add humor and personality to match other chapters
Both chapters now follow the established pattern of starting with a problem,
sharing personal experiences, and providing actionable insights based on
real consulting work.
Copy file name to clipboardExpand all lines: docs/workshops/chapter3-1.md
+31-28Lines changed: 31 additions & 28 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,11 +11,11 @@ author: Jason Liu
11
11
12
12
RAG systems improve most when they collect feedback effectively. Many implementations focus exclusively on the technical details of retrieval and generation while neglecting the infrastructure needed to collect and utilize user feedback.
13
13
14
-
!!! note "Building on Previous Chapters"
15
-
- **[Chapter 1](chapter1.md)**: The evaluation framework you built provides the baseline
16
-
- **[Chapter 2](chapter2.md)**: The fine-tuning techniques need feedback data to be effective
14
+
**Building on Previous Chapters:**
15
+
-**[Chapter 1](chapter1.md)**: The evaluation framework you built provides the baseline
16
+
-**[Chapter 2](chapter2.md)**: The fine-tuning techniques need feedback data to be effective
17
17
18
-
This chapter shows you how to collect the data that powers continuous improvement.
18
+
This chapter shows you how to collect the data that powers continuous improvement.
19
19
20
20
In this chapter, we'll explore how to build effective feedback mechanisms that turn your RAG application from a static implementation into a continuously improving system. This approach creates a feedback loop where user interactions provide the data needed to make the system better.
21
21
@@ -87,8 +87,9 @@ Here are several patterns for implementing high-visibility feedback mechanisms:
87
87
88
88
Each approach has advantages for different use cases. The key is to make feedback collection a natural part of the user experience rather than an afterthought.
89
89
90
-
!!! tip "Streaming and Perceived Performance"
91
-
**The Claude Progress Counter Effect:**
90
+
**Streaming and Perceived Performance:**
91
+
92
+
**The Claude Progress Counter Effect:**
92
93
93
94
Claude's implementation of progress counters during response generation serves multiple purposes:
94
95
- Shows "thinking" progress (e.g., "Analyzing document 3 of 5...")
@@ -122,8 +123,9 @@ For enterprise applications, especially when working with large customers who ha
122
123
123
124
This approach creates transparency and builds trust by showing customers that their feedback drives real improvements. This method typically increases feedback by 5x compared to traditional forms, while also improving customer retention.
124
125
125
-
!!! example "Enterprise Feedback Pattern"
126
-
**The Most Effective B2B Feedback Flow:**
126
+
**Enterprise Feedback Pattern:**
127
+
128
+
**The Most Effective B2B Feedback Flow:**
127
129
128
130
1. **In-App Collection:**
129
131
- Binary feedback (thumbs up/down) for quick signals
@@ -148,15 +150,15 @@ This approach creates transparency and builds trust by showing customers that th
148
150
149
151
This pattern has helped teams achieve 30-40% feedback rates in enterprise settings.
150
152
151
-
!!! example "Slack Webhook Integration Code"
152
-

153
+
**Slack Webhook Integration Code:**
154
+

153
155
154
156
```
155
157
*This code integrates feedback collection with Slack, automatically posting negative feedback to a shared channel for immediate visibility and follow-up.*
156
158
```
157
159
158
-
!!! example "Feedback UI Implementation"
159
-

160
+
**Feedback UI Implementation:**
161
+

160
162
161
163
```
162
164
*This code renders a response with prominent feedback options, automatically showing a more detailed form if the user indicates the response wasn't fully helpful.*
By tracking these behaviors, you can identify patterns that indicate success or failure even when users don't provide explicit feedback.
214
216
215
-
!!! example "Implicit Feedback Collection"
216
-

217
+
**Implicit Feedback Collection:**
218
+

217
219
218
220
```
219
221
*This code tracks key implicit feedback signals including query refinements, citation clicks, and engagement time, providing valuable data even when users don't explicitly rate responses.*
@@ -239,8 +241,8 @@ Consider these UI patterns specifically designed to help collect hard negative e
239
241
240
242
1.**Regeneration After Removal**: Allow users to remove citation sources and then regenerate the answer. Documents removed before regeneration become strong hard negative candidates for that query.
241
243
242
-
!!! example "Interactive Citations UI"
243
-

244
+
**Interactive Citations UI:**
245
+

244
246
245
247
```
246
248
*This UI allows users to mark specific citations as relevant or irrelevant and regenerate answers, creating valuable training data for improving retrieval quality.*
@@ -271,8 +273,8 @@ There are several approaches to implementing citations in your RAG interface:
271
273
1.**Inline highlights**: Highlighting portions of text with the source documents they came from
272
274
1.**Visual PDF overlays**: For document-based applications, highlighting the exact location in a PDF
273
275
274
-
!!! example "Markdown-based Citation Implementation"
275
-

276
+
**Markdown-based Citation Implementation:**
277
+

276
278
277
279
```
278
280
*This code formats responses with clickable citations and builds a reference list that includes feedback options for each source, helping collect document-level relevance signals.*
@@ -291,13 +293,13 @@ This approach is particularly valuable for PDF-heavy domains like legal, medical
291
293
292
294
### Citation Implementation Patterns
293
295
294
-
!!! quote "Preventing Hallucinations"
295
-
Skylar Payne emphasizes that hallucination remains a critical challenge, especially in sensitive domains. His most effective approach: "Force the LLM to provide inline citations, validate that each citation exists in the retrieved documents, and semantically validate that each citation actually supports the claimed content."
296
+
> **Preventing Hallucinations:** Skylar Payne emphasizes that hallucination remains a critical challenge, especially in sensitive domains. His most effective approach: "Force the LLM to provide inline citations, validate that each citation exists in the retrieved documents, and semantically validate that each citation actually supports the claimed content."
296
297
297
298
This is particularly critical for healthcare, legal, and financial applications. [See more anti-patterns to avoid →](../talks/rag-antipatterns-skylar-payne.md)
298
299
299
-
!!! info "XML-Based Citation Pattern"
300
-
**The Most Robust Approach:**
300
+
**XML-Based Citation Pattern:**
301
+
302
+
**The Most Robust Approach:**
301
303
302
304
Instead of relying on markdown links or footnotes, use XML tags with start/end word anchoring:
303
305
@@ -322,8 +324,9 @@ This approach is particularly valuable for PDF-heavy domains like legal, medical
322
324
323
325
The ultimate goal of feedback collection is to guide your improvement roadmap. Rather than making enhancement decisions based on intuition or technical interest, you can prioritize based on user needs revealed through feedback.
324
326
325
-
!!! info "Production Monitoring: Beyond Basic Feedback"
326
-
Ben Hylak and Sidhant Bendre highlight a critical insight: "There's no exception being thrown when something goes wrong - the model simply produces an inadequate response." Their approach combines implicit signals (user frustration, task failures) with explicit signals (ratings, regenerations) to identify issues that traditional monitoring misses. The Trellis framework they present helps organize the "infinite chaos" of AI outputs into controllable segments. [Learn about production monitoring strategies →](../talks/online-evals-production-monitoring-ben-sidhant.md)
327
+
**Production Monitoring: Beyond Basic Feedback**
328
+
329
+
Ben Hylak and Sidhant Bendre highlight a critical insight: "There's no exception being thrown when something goes wrong - the model simply produces an inadequate response." Their approach combines implicit signals (user frustration, task failures) with explicit signals (ratings, regenerations) to identify issues that traditional monitoring misses. The Trellis framework they present helps organize the "infinite chaos" of AI outputs into controllable segments. [Learn about production monitoring strategies →](../talks/online-evals-production-monitoring-ben-sidhant.md)
327
330
328
331
A feedback-driven roadmap:
329
332
@@ -352,10 +355,10 @@ Remember that small UX changes can make enormous differences in feedback collect
352
355
353
356
In the next chapter, we'll explore how to reduce perceived latency through streaming and progressive responses, building on the feedback foundation to create a more engaging user experience.
354
357
355
-
!!! info "How This Chapter Connects Forward"
356
-
- **[Chapter 4](chapter4-2.md)**: The feedback you collect enables query segmentation and analysis
357
-
- **[Chapter 5](chapter5-1.md)**: User behavior patterns reveal which specialized retrievers to build
358
-
- **[Chapter 6](chapter6-2.md)**: Feedback on router decisions improves tool selection
358
+
**How This Chapter Connects Forward:**
359
+
-**[Chapter 4](chapter4-2.md)**: The feedback you collect enables query segmentation and analysis
360
+
-**[Chapter 5](chapter5-1.md)**: User behavior patterns reveal which specialized retrievers to build
361
+
-**[Chapter 6](chapter6-2.md)**: Feedback on router decisions improves tool selection
0 commit comments