You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
## Data-Driven Product Development for AI Applications
12
12
13
-
_A systematic approach to building self-improving AI systems_
13
+
*A systematic approach to building self-improving AI systems*
14
14
15
15
!!! abstract "About This Book"
16
-
This book provides a structured approach to evolving Retrieval-Augmented Generation (RAG) from a technical implementation into a continuously improving product. You'll learn to combine product thinking with data science principles to create AI systems that deliver increasing value over time.
16
+
This book provides a structured approach to evolving Retrieval-Augmented Generation (RAG) from a technical implementation into a continuously improving product. You'll learn to combine product thinking with data science principles to create AI systems that deliver increasing value over time.
17
+
18
+
Most teams focus on the latest models and algorithms while missing the fundamentals: understanding their data, measuring performance, and systematically improving based on user feedback.
17
19
18
20
## The RAG Improvement Flywheel
19
21
20
-
At the core of this book is the RAG improvement flywheel - a continuous cycle that transforms user interactions into product enhancements.
21
-
22
22
```mermaid
23
23
graph TD
24
24
A[Synthetic Data & Evaluation] --> B[Learning from Evaluations]
@@ -33,129 +33,171 @@ graph TD
33
33
style E fill:#dfd,stroke:#333,stroke-width:2px
34
34
```
35
35
36
-
!!! tip "Beyond Technical Implementation"
37
-
This book goes beyond teaching you how to implement RAG. It shows you how to think about RAG as a product that continuously evolves to meet user needs and deliver business value.
38
-
39
-
## Chapters
36
+
## Workshop Series
40
37
41
38
### [Introduction: Beyond Implementation to Improvement](workshops/chapter0.md)
39
+
Shifting from technical implementation to product-focused continuous improvement. Understanding RAG as a recommendation engine and the improvement flywheel.
42
40
43
-
Understand why systematic improvement matters and how to approach RAG as a product rather than just a technical implementation.
41
+
### [Chapter 1: Kickstarting the Data Flywheel with Synthetic Data](workshops/chapter1.md)
42
+
Common pitfalls in AI development, leading vs. lagging metrics, precision and recall for retrieval evaluation, and synthetic data generation techniques.
44
43
45
-
### [Chapter 1: Starting the Flywheel](workshops/chapter1.md)
44
+
### [Chapter 2: Converting Evaluations into Training Data for Fine-Tuning](workshops/chapter2.md)
45
+
Why generic embeddings fall short, converting evaluation examples into few-shot prompts, contrastive learning, and re-rankers as cost-effective strategies.
46
46
47
-
Learn how to overcome the cold-start problem, establish meaningful metrics, and create a data foundation that drives product decisions.
47
+
### Chapter 3: User Experience and Feedback Collection
48
48
49
-
### [Chapter 2: From Evaluation to Enhancement](workshops/chapter2.md)
Making feedback visible and engaging (increasing rates from <1% to >30%), proven copy patterns, and enterprise feedback collection through Slack integrations.
50
51
51
-
Transform evaluation insights into concrete product improvements through fine-tuning, re-ranking, and targeted enhancements.
52
+
#### [Chapter 3.2: Streaming and Interstitials](workshops/chapter3-2.md)
53
+
Psychology of waiting, implementing streaming responses for 30-40% higher feedback collection, and meaningful interstitials.
52
54
53
-
### [Chapter 3: The User Experience of AI](workshops/chapter3-1.md)
Measuring tool selection effectiveness, dual-mode UI, user feedback as training data, and creating improvement flywheel.
74
84
75
-
Explore insights from industry experts and practitioners through our collection of talks, lightning lessons, and presentations:
85
+
## Expert Talks
76
86
77
-
### [Featured Talks](talks/index.md)
87
+
### Foundation and Evaluation
78
88
79
-
-**[Fine-tuning Re-rankers and Embedding Models for Better RAG Performance](talks/fine-tuning-rerankers-embeddings-ayush-lancedb.md)** - Practical approaches to enhancing retrieval quality (Ayush from LanceDB)
80
-
-**[RAG Anti-patterns in the Wild](talks/rag-antipatterns-skylar-payne.md)** - Common mistakes and how to fix them (Skylar Payne)
81
-
-**[Semantic Search Over the Web with Exa](talks/semantic-search-exa-will-bryk.md)** - Building AI-first search engines (Will Bryk)
-**[Online Evals and Production Monitoring](talks/online-evals-production-monitoring-ben-sidhant.md)** - Monitoring AI systems at scale (Ben Hylak & Sidhant Bendre)
Simple UX changes increased feedback collection 4x. Key insight: specific questions like "Did this run do what you expected?" dramatically outperform generic prompts.
84
91
85
-
[View all talks →](talks/index.md)
92
+
**[Text Chunking Strategies](talks/chromadb-anton-chunking.md)** - Anton (ChromaDB)
93
+
Why chunking remains critical even with infinite context windows. Default chunking strategies in popular libraries often produce terrible results.
86
94
87
-
## For Product Leaders, Engineers, and Data Scientists
95
+
**[Embedding Performance Evaluation](talks/embedding-performance-generative-evals-kelly-hong.md)** - Kelly Hong
96
+
Model rankings on custom benchmarks often contradict MTEB rankings - public benchmark performance doesn't guarantee real-world success.
88
97
89
-
!!! info "What You'll Learn"
90
-
**For Product Leaders**
91
-
- How to establish metrics that align with business outcomes
92
-
- Frameworks for prioritizing AI product improvements
93
-
- Approaches to building product roadmaps for RAG applications
94
-
- Methods for communicating AI improvements to stakeholders
95
-
**For Engineers**
96
-
- Implementation patterns that facilitate rapid iteration
97
-
- Architectural decisions that enable continuous improvement
98
-
- Techniques for building modular, specialized capabilities
99
-
- Approaches to technical debt management in AI systems
100
-
101
-
**For Data Scientists**
102
-
- Methods for creating synthetic evaluation datasets
103
-
- Techniques for segmenting and analyzing user queries
104
-
- Frameworks for measuring retrieval effectiveness
105
-
- Approaches to continuous learning from user interactions
-**"How do we handle time-based queries?"** Use PostgreSQL with pgvector-scale. [Learn more →](office-hours/faq.md#how-do-we-introduce-a-concept-of-time-and-vector-search-to-answer-questions-like-whats-the-latest-news-without-needing-to-move-to-a-graph-database)
164
+
-**"Should we use DSPy for prompt optimization?"** It depends. [See when →](office-hours/faq.md#what-is-your-take-on-dspy-should-we-use-it)
165
+
-**"Would you recommend ColBERT models?"** Test against your baseline first. [See approach →](office-hours/faq.md#would-you-recommend-using-colbert-models-or-other-specialized-retrieval-approaches)
166
+
167
+
[Browse All FAQ](office-hours/faq.md){ .md-button } [View Office Hours](office-hours/index.md){ .md-button }
168
+
169
+
## For Product Leaders, Engineers, and Data Scientists
170
+
171
+
!!! info "What You'll Learn"
172
+
**For Product Leaders**: Establish metrics that align with business outcomes, prioritization frameworks, and roadmapping approaches
173
+
174
+
**For Engineers**: Implementation patterns for rapid iteration, architectural decisions, and modular capabilities
175
+
176
+
**For Data Scientists**: Synthetic evaluation datasets, query segmentation techniques, and continuous learning approaches
Jason Liu brings practical experience from his work at Facebook, Stitch Fix, and as a consultant for companies like HubSpot, Zapier, and many others. His background spans computer vision, recommendation systems, and RAG applications across diverse domains.
190
+
Jason Liu brings practical experience from Facebook, Stitch Fix, and as a consultant for companies like HubSpot, Zapier, and many others. His background spans computer vision, recommendation systems, and RAG applications across diverse domains.
149
191
150
192
!!! quote "Author's Philosophy"
151
-
"The most successful AI products aren't the ones with the most sophisticated models, but those built on disciplined processes for understanding users, measuring performance, and systematically improving. This book will show you how to create not just a RAG application, but a product that becomes more valuable with every interaction."
193
+
"The most successful AI products aren't the ones with the most sophisticated models, but those built on disciplined processes for understanding users, measuring performance, and systematically improving."
194
+
195
+
---
152
196
153
197
## Getting Started
154
198
155
199
Begin your journey by reading the [Introduction](workshops/chapter0.md) or jump directly to [Chapter 1](workshops/chapter1.md) to start building your evaluation framework and data foundation.
156
200
157
-
---
158
-
159
-
IF you want to get discounts and 6 day email source on the topic make sure to subscribe to
201
+
If you want to get discounts and 6 day email source on the topic make sure to subscribe to
0 commit comments