You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
*A systematic approach to building self-improving AI systems*
14
14
15
15
!!! abstract "About This Book"
16
-
This book provides a structured approach to evolving Retrieval-Augmented Generation (RAG) from a technical implementation into a continuously improving product. You'll learn to combine product thinking with data science principles to create AI systems that deliver increasing value over time.
17
-
18
-
Most teams focus on the latest models and algorithms while missing the fundamentals: understanding their data, measuring performance, and systematically improving based on user feedback. This resource shows you the proven approach used by companies like Zapier, Glean, and Exa.
16
+
This book provides a structured approach to evolving Retrieval-Augmented Generation (RAG) from a technical implementation into a continuously improving product. You'll learn to combine product thinking with data science principles to create AI systems that deliver increasing value over time.
19
17
20
18
## The RAG Improvement Flywheel
21
19
22
-
At the core of this book is the RAG improvement flywheel - a continuous cycle that transforms user interactions into product enhancements:
20
+
At the core of this book is the RAG improvement flywheel - a continuous cycle that transforms user interactions into product enhancements.
23
21
24
22
```mermaid
25
23
graph TD
@@ -36,159 +34,64 @@ graph TD
36
34
```
37
35
38
36
!!! tip "Beyond Technical Implementation"
39
-
This book goes beyond teaching you how to implement RAG. It shows you how to think about RAG as a product that continuously evolves to meet user needs and deliver business value.
40
-
41
-
## Table of Contents
42
-
43
-
### Workshop Series
44
-
45
-
| Chapter | Title | Focus Area | Key Outcomes |
46
-
|---------|-------|------------|-------------|
47
-
|[Introduction](workshops/chapter0.md)| Beyond Implementation to Improvement | Product Mindset | Shift from technical to product thinking |
48
-
|[Chapter 1](workshops/chapter1.md)| Starting the Flywheel | Evaluation & Metrics | Build synthetic data and evaluation frameworks |
49
-
|[Chapter 2](workshops/chapter2.md)| From Evaluation to Enhancement | Fine-tuning & Training | Convert evaluations into training data |
50
-
|[Chapter 3.1](workshops/chapter3-1.md)| Feedback Collection | User Experience | Design feedback mechanisms that users actually use |
51
-
|[Chapter 3.2](workshops/chapter3-2.md)| Streaming & Performance | User Experience | Implement streaming and reduce perceived latency |
52
-
|[Chapter 3.3](workshops/chapter3-3.md)| Quality Improvements | User Experience | Citations, chain-of-thought, and validation |
53
-
|[Chapter 4.1](workshops/chapter4-1.md)| Topic Modeling | User Analysis | Find patterns in user feedback and queries |
54
-
|[Chapter 4.2](workshops/chapter4-2.md)| Prioritization | User Analysis | Turn insights into strategic action plans |
55
-
|[Chapter 5.1](workshops/chapter5-1.md)| Specialized Retrieval | Architecture | Build specialized capabilities for different content |
|[Kelly Hong](talks/embedding-performance-generative-evals-kelly-hong.md)| Independent | Evaluation | Custom benchmarks often contradict public ones |
69
-
|[Ben & Sidhant](talks/online-evals-production-monitoring-ben-sidhant.md)| Independent | Production Monitoring | Traditional error monitoring doesn't work for AI |
|[Anton](talks/query-routing-anton.md)| ChromaDB | Query Routing | Separate indexes often outperform filtered large ones |
79
-
80
-
## Quick Wins: High-Impact RAG Improvements
81
-
82
-
Based on real-world implementations, here are proven improvements you can implement quickly:
83
-
84
-
!!! success "Top 5 Quick Wins"
85
-
1. **Change Feedback Copy** : Replace "How did we do?" with "Did we answer your question?"
86
-
2. **Use Markdown Tables** : Format structured data as markdown tables instead of JSON/CSV or XML when tables are complex and multiple columns / headers are needed.
87
-
3. **Add Streaming Progress** : Show "Searching... Analyzing... Generating..." with progress
88
-
4. **Implement Page-Level Chunking** : For documentation, respect page boundaries, and use page-level chunking. Humans tend to create semantically coherent chunks at the page level.
89
-
90
-
!!! tip "Medium-Term Improvements (2-4 weeks)"
91
-
- **Fine-tune embeddings**: $1.50 and 40 minutes for 6-10% improvement
92
-
- **Add re-ranker**: 15-20% retrieval improvement
93
-
- **Build specialized tools**: 10x better for specific use cases
- **Create Slack feedback integration**: 5x more enterprise feedback
96
-
97
-
## Workshop Series
98
-
99
-
### Foundation: Metrics & Evaluation
100
-
101
-
**[Introduction: Beyond Implementation to Improvement](workshops/chapter0.md)**
102
-
Shifting from technical implementation to product-focused continuous improvement. Understanding RAG as a recommendation engine wrapped around language models and the improvement flywheel.
103
-
104
-
**[Chapter 1: Kickstarting the Data Flywheel with Synthetic Data](workshops/chapter1.md)**
105
-
Common pitfalls in AI development, leading vs. lagging metrics, understanding precision and recall for retrieval evaluation, and synthetic data generation techniques.
106
-
107
-
**[Chapter 2: Converting Evaluations into Training Data for Fine-Tuning](workshops/chapter2.md)**
108
-
Why generic embeddings fall short, converting evaluation examples into effective few-shot prompts, contrastive learning, and re-rankers as cost-effective enhancement strategies.
109
-
110
-
### User Experience & Feedback
111
-
112
-
**[Chapter 3.1: Feedback Collection - Building Your Improvement Flywheel](workshops/chapter3-1.md)**
113
-
Making feedback visible and engaging (increasing rates from <1% to >30%), proven copy patterns, segmented feedback, and enterprise feedback collection.
37
+
This book goes beyond teaching you how to implement RAG. It shows you how to think about RAG as a product that continuously evolves to meet user needs and deliver business value.
114
38
115
-
**[Chapter 3.2: Overcoming Latency - Streaming and Interstitials](workshops/chapter3-2.md)**
116
-
Psychology of waiting, implementing streaming responses for 30-40% higher feedback collection, skeleton screens and meaningful interstitials.
39
+
## Chapters
117
40
118
-
**[Chapter 3.3: Quality of Life Improvements](workshops/chapter3-3.md)**
119
-
Interactive citations, chain of thought reasoning for 8-15% accuracy improvements, validation patterns as safety nets, and strategic rejection.
41
+
### [Introduction: Beyond Implementation to Improvement](workshops/chapter0.md)
120
42
121
-
### Analysis & Specialization
43
+
Understand why systematic improvement matters and how to approach RAG as a product rather than just a technical implementation.
122
44
123
-
**[Chapter 4.1: Topic Modeling and Analysis](workshops/chapter4-1.md)**
124
-
Moving from individual feedback to systematic pattern identification, topics vs. capabilities, and transforming "make the AI better" into specific priorities.
45
+
### [Chapter 1: Starting the Flywheel](workshops/chapter1.md)
125
46
126
-
**[Chapter 4.2: Prioritization and Roadmapping](workshops/chapter4-2.md)**
127
-
Impact/effort prioritization using 2x2 frameworks, failure mode analysis, and building strategic roadmaps based on user behavior patterns.
47
+
Learn how to overcome the cold-start problem, establish meaningful metrics, and create a data foundation that drives product decisions.
128
48
129
-
### Advanced Architecture
49
+
### [Chapter 2: From Evaluation to Enhancement](workshops/chapter2.md)
The API mindset, organizational structure, evolution from monolithic to modular architecture, and performance formula.
55
+
Design interfaces that both delight users and gather valuable feedback, creating a virtuous cycle of improvement.
139
56
140
-
**[Chapter 6.2: Tool Interfaces and Implementation](workshops/chapter6-2.md)**
141
-
Designing tool interfaces, router implementation using structured outputs, dynamic example selection, and tool portfolio design.
57
+
### [Chapter 4: Understanding Your Users](workshops/chapter4-1.md)
142
58
143
-
**[Chapter 6.3: Performance Measurement and Improvement](workshops/chapter6-3.md)**
144
-
Measuring tool selection effectiveness, dual-mode UI, user feedback as training data, and creating improvement flywheel.
59
+
Segment users and queries to identify high-value opportunities and create targeted improvement strategies.
145
60
146
-
##How to Use This Resource
61
+
### [Chapter 5: Building Specialized Capabilities](workshops/chapter5-1.md)
147
62
148
-
**For Beginners**: Start with the [Introduction](workshops/chapter0.md) to understand the product mindset, then work through the chapters sequentially.
63
+
Develop purpose-built solutions for different user needs spanning documents, images, tables, and structured data.
149
64
150
-
**For Quick Wins**: Jump to the [Quick Wins section](#quick-wins-high-impact-rag-improvements) above for immediate improvements you can implement today.
**For Specific Problems**: Check the [FAQ](office-hours/faq.md) for answers to common questions, or browse talks by topic in the table above.
67
+
Create a cohesive product experience that intelligently routes to specialized components while maintaining a seamless user experience.
153
68
154
-
**For Complete Implementation**: Follow the full workshop series from Chapter 1 through 6.3 to build a comprehensive self-improving RAG system.
69
+
### [Key Takeaways: Product Principles for AI Applications](misc/what-i-want-you-to-takeaway.md)
155
70
156
-
## Key Insights Across All Content
71
+
Core principles that will guide your approach to building AI products regardless of how the technology evolves.
157
72
158
-
**Most Important Finding**: Teams that iterate fastest on data examination consistently outperform those focused on algorithmic sophistication.
73
+
## Talks and Presentations
159
74
160
-
**Most Underutilized Techniques**: Fine-tuning embeddings and re-rankers are more accessible and impactful than most teams realize.
75
+
Explore insights from industry experts and practitioners through our collection of talks, lightning lessons, and presentations:
161
76
162
-
**Biggest Mistake**: 90% of teams add complexity that makes their RAG systems worse. Start simple, measure everything, improve systematically.
77
+
### [Featured Talks](talks/index.md)
163
78
164
-
**Critical Success Factors**:
165
-
- Establish evaluation frameworks before building
166
-
- Design feedback collection into your UX from day one
167
-
- Understand your users and their query patterns
168
-
- Build specialized tools for different content types
169
-
- Create unified routing that feels seamless to users
79
+
-**[Fine-tuning Re-rankers and Embedding Models for Better RAG Performance](talks/fine-tuning-rerankers-embeddings-ayush-lancedb.md)** - Practical approaches to enhancing retrieval quality (Ayush from LanceDB)
80
+
-**[RAG Anti-patterns in the Wild](talks/rag-antipatterns-skylar-payne.md)** - Common mistakes and how to fix them (Skylar Payne)
81
+
-**[Semantic Search Over the Web with Exa](talks/semantic-search-exa-will-bryk.md)** - Building AI-first search engines (Will Bryk)
-**[Online Evals and Production Monitoring](talks/online-evals-production-monitoring-ben-sidhant.md)** - Monitoring AI systems at scale (Ben Hylak & Sidhant Bendre)
-**"How do we handle time-based queries?"** Use PostgreSQL with pgvector-scale. [Learn more →](office-hours/faq.md#how-do-we-introduce-a-concept-of-time-and-vector-search-to-answer-questions-like-whats-the-latest-news-without-needing-to-move-to-a-graph-database)
177
-
-**"Should we use DSPy for prompt optimization?"** It depends. [See when →](office-hours/faq.md#what-is-your-take-on-dspy-should-we-use-it)
178
-
-**"Would you recommend ColBERT models?"** Test against your baseline first. [See approach →](office-hours/faq.md#would-you-recommend-using-colbert-models-or-other-specialized-retrieval-approaches)
179
-
180
-
[Browse All FAQ](office-hours/faq.md){ .md-button } [View Office Hours](office-hours/){ .md-button }
85
+
[View all talks →](talks/index.md)
181
86
182
87
## For Product Leaders, Engineers, and Data Scientists
183
88
184
89
!!! info "What You'll Learn"
185
-
186
90
**For Product Leaders**
187
91
- How to establish metrics that align with business outcomes
188
92
- Frameworks for prioritizing AI product improvements
189
93
- Approaches to building product roadmaps for RAG applications
190
94
- Methods for communicating AI improvements to stakeholders
191
-
192
95
**For Engineers**
193
96
- Implementation patterns that facilitate rapid iteration
194
97
- Architectural decisions that enable continuous improvement
@@ -201,32 +104,56 @@ Top questions from office hours:
201
104
- Frameworks for measuring retrieval effectiveness
202
105
- Approaches to continuous learning from user interactions
- **Create Slack feedback integration**: 5x more enterprise feedback
215
140
141
+
!!! info "Learn from the Experts"
142
+
Before implementing, learn from these practical talks:
143
+
- [**RAG Anti-patterns in the Wild**](talks/rag-antipatterns-skylar-payne.md) - Common mistakes across industries and how to fix them
144
+
- [**Document Ingestion Best Practices**](talks/reducto-docs-adit.md) - Production-ready parsing for tables, PDFs, and complex documents
216
145
217
146
## About the Author
218
147
219
-
Jason Liu brings practical experience from his work at Facebook, Stitch Fix, and as a consultant for companies like HubSpot, Zapier, and many others. His background spans computer vision, recommendation systems, and RAG applications across diverse domains.
148
+
Jason Liu brings practical experience from Facebook, Stitch Fix, and as a consultant for companies like HubSpot, Zapier, and many others. His background spans computer vision, recommendation systems, and RAG applications across diverse domains.
220
149
221
150
!!! quote "Author's Philosophy"
222
-
"The most successful AI products aren't the ones with the most sophisticated models, but those built on disciplined processes for understanding users, measuring performance, and systematically improving. This resource will show you how to create not just a RAG application, but a product that becomes more valuable with every interaction."
223
-
224
-
---
151
+
"The most successful AI products aren't the ones with the most sophisticated models, but those built on disciplined processes for understanding users, measuring performance, and systematically improving. This book will show you how to create not just a RAG application, but a product that becomes more valuable with every interaction."
225
152
226
153
## Getting Started
227
154
228
155
Begin your journey by reading the [Introduction](workshops/chapter0.md) or jump directly to [Chapter 1](workshops/chapter1.md) to start building your evaluation framework and data foundation.
229
156
230
157
If you want to get discounts and 6 day email source on the topic make sure to subscribe to
0 commit comments