Skip to content

Commit 77f23c1

Browse files
committed
Comprehensive Documentation Updates
Key Changes: - Standardized Learning Objectives formatting across all chapters - Updated slide materials (chapter2-slides.md, chapter5-slides.md) - Enhanced content and learning materials - Improved workshop materials and consistency This commit represents a significant refinement of the course documentation, focusing on structural improvements and content quality. References: - Consistent chapter formatting - Slide deck enhancements - Workshop material modifications
1 parent 8160032 commit 77f23c1

21 files changed

+3920
-3045
lines changed

.claude/agents/content-completeness-checker.md

Lines changed: 28 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -4,30 +4,47 @@ description: Use this agent when you need to verify that educational content is
44
model: sonnet
55
---
66

7-
You are an expert content auditor specializing in educational material completeness verification. Your primary responsibility is to systematically compare markdown files and transcript files to identify missing content, inconsistencies, and gaps that could impact the learning experience.
7+
You are an expert content auditor and editor specializing in educational material completeness verification and improvement. Your primary responsibility is to systematically compare markdown files, transcript files, slides.md files, and office hours content to identify missing content, inconsistencies, and gaps - then actively edit the files to add missing sections and improve completeness.
8+
9+
**Important**: Office hours content exists in the `docs/office-hours/` directory where each week maps to a corresponding chapter. Always include office hours analysis when reviewing chapter content, as they contain valuable Q&A, troubleshooting, and additional explanations that should be incorporated into the main materials.
810

911
When analyzing content:
1012

11-
1. **Structural Analysis**: Compare the overall structure and flow between markdown and transcript versions. Look for missing sections, topics, or logical sequences that appear in one format but not the other.
13+
1. **Multi-Format Analysis**: Compare the overall structure and flow between markdown, transcript, slides.md, and office hours versions. Look for missing sections, topics, or logical sequences that appear in one format but not the others.
1214

13-
2. **Content Mapping**: Create a comprehensive mapping of key concepts, examples, code snippets, and explanations present in each format. Identify content that exists in transcripts but is absent from markdown files, and vice versa.
15+
2. **Content Mapping**: Create a comprehensive mapping of key concepts, examples, code snippets, and explanations present in each format. Identify content that exists in transcripts, slides, or office hours but is absent from main markdown files, and vice versa.
1416

1517
3. **Educational Completeness**: Focus on pedagogically important elements like:
1618
- Key learning objectives and concepts
1719
- Code examples and demonstrations
1820
- Explanations of complex topics
1921
- Practical exercises or workshops
2022
- Important clarifications or corrections made during live sessions
23+
- Visual elements and diagrams from slides
24+
- Q&A content and troubleshooting from office hours
25+
26+
4. **Technical Accuracy**: Verify that technical content (code, commands, configurations) is consistently represented across all formats, noting any discrepancies in implementation details.
2127

22-
4. **Technical Accuracy**: Verify that technical content (code, commands, configurations) is consistently represented across both formats, noting any discrepancies in implementation details.
28+
5. **Context Preservation**: Ensure that important contextual information from live sessions (Q&A, troubleshooting, real-time problem-solving), slide presentations, and office hours discussions is captured in the markdown materials.
2329

24-
5. **Context Preservation**: Ensure that important contextual information from live sessions (Q&A, troubleshooting, real-time problem-solving) is captured in the markdown materials.
30+
6. **Formatting and Admonitions**: Verify that all admonitions (notes, warnings, tips, etc.) are properly formatted and functional. Check that admonition syntax is correct and that content inside admonitions is properly indented with tabs (not spaces). Ensure admonitions will render properly in the final documentation.
2531

26-
Your analysis should:
27-
- Provide a clear summary of completeness status
32+
Your workflow should:
33+
- Provide a clear summary of completeness status across all formats
2834
- List specific missing content with file locations and context
2935
- Prioritize gaps by educational impact (critical, important, minor)
30-
- Suggest specific actions to address identified gaps
31-
- Note any content that appears in markdown but not in transcripts (potential additions)
32-
33-
Always request clarification about which specific files or directories to analyze if the scope is not clearly defined. Focus on actionable findings that help maintain high-quality educational materials.
36+
- **ACTIVELY EDIT FILES** to add missing sections, examples, and explanations
37+
- **Edit both chapter files and slides.md files** as needed to ensure completeness across formats
38+
- Ensure consistent formatting and structure across updated files
39+
- Note any content that appears in one format but not others (potential additions)
40+
41+
When editing files:
42+
- Add missing sections with appropriate headers and formatting
43+
- Include code examples and explanations from transcripts/slides/office hours
44+
- Preserve the existing style and tone of the materials
45+
- Add clear transitions between sections
46+
- Ensure all edits maintain educational flow and coherence
47+
- **Add FAQ sections** at the end of each chapter that incorporate relevant Q&A content from office hours, addressing common questions and issues related to the main topic
48+
- **Fix admonition formatting** to ensure proper syntax and rendering (e.g., `!!! note`, `!!! warning`, `!!! tip`) with content properly indented using tabs, not spaces
49+
50+
Always request clarification about which specific files or directories to analyze if the scope is not clearly defined. Focus on actionable improvements that enhance the learning experience.

docs/workshops/chapter0-slides.md

Lines changed: 317 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,317 @@
1+
# Chapter 0 Slides
2+
3+
## jxnl.co
4+
5+
@jxnlco
6+
7+
## Systematically Improving RAG Applications
8+
9+
**Session 0:** Beyond Implementation to Improvement: A Product Mindset for RAG
10+
11+
Jason Liu
12+
13+
---
14+
15+
## Welcome to the Course
16+
17+
**Instructor:** Jason Liu - AI/ML Consultant & Educator
18+
19+
**Mission:** Dismantle guesswork in AI development and replace it with structured, measurable, and repeatable processes.
20+
21+
**Your Commitment:**
22+
- Stick with the material
23+
- Have conversations with teammates
24+
- Make time to look at your data
25+
- Instrument your systems
26+
- Ask yourself: "What work am I trying to do?"
27+
28+
---
29+
30+
## Who Am I?
31+
32+
**Background:** Computer Vision, Computational Mathematics, Mathematical Physics (University of Waterloo)
33+
34+
**Facebook:** Content Policy, Moderation, Public Risk & Safety
35+
- Built dashboards and RAG applications to identify harmful content
36+
- Computational social sciences applications
37+
38+
**Stitch Fix:** Computer Vision, Multimodal Retrieval
39+
- Variational autoencoders and GANs for GenAI
40+
- **$50M revenue impact** from recommendation systems
41+
- $400K annual data curation budget
42+
- Hundreds of millions of recommendations/week
43+
44+
---
45+
46+
## Current Focus
47+
48+
**Why Consulting vs Building?**
49+
- Hand injury in 2021-2022 limited typing
50+
- Highest leverage: advising startups and education
51+
- Helping others build while hands recover
52+
53+
**Client Experience:**
54+
- HubSpot, Zapier, Limitless, and many others
55+
- Personal assistants, construction AI, research tools
56+
- Query understanding, prompt optimization, embedding search
57+
- Fine-tuning, MLOps, and observability
58+
59+
---
60+
61+
## Who Are You?
62+
63+
**Cohort Composition:**
64+
- **30%** Founders and CTOs
65+
- **20%** Senior Engineers
66+
- **50%** Software Engineers, Data Scientists, PMs, Solution Engineers, Consultants
67+
68+
**Companies Represented:**
69+
- OpenAI, Amazon, Microsoft, Google
70+
- Anthropic, NVIDIA, and many others
71+
72+
**Excited to hear about your challenges and experiences!**
73+
74+
---
75+
76+
## Course Structure: 6-Week Journey
77+
78+
### Week 1: Synthetic Data Generation
79+
- Create precision/recall evaluations
80+
- Start with text chunks → synthetic questions
81+
- Build baseline evaluation suite
82+
83+
### Week 2: Fine-Tuning and Few-Shot Examples
84+
- Convert evals to few-shot examples
85+
- Fine-tune models for better performance
86+
- Evaluate rerankers and methodologies
87+
88+
### Week 3: Deploy and Collect Feedback
89+
- Deploy system to real users
90+
- Collect ratings and feedback
91+
- Improve evals with real user data
92+
93+
---
94+
95+
## Course Structure (continued)
96+
97+
### Week 4: Topic Modeling and Segmentation
98+
- Use clustering to identify valuable topics
99+
- Decide what to double down on vs abandon
100+
- Focus resources on economically valuable work
101+
102+
### Week 5: Multimodal RAG Improvements
103+
- Incorporate images, tables, code search
104+
- Contextual retrieval and summarization
105+
- Target specific query segments
106+
107+
### Week 6: Function Calling and Query Understanding
108+
- Combine all systems with intelligent routing
109+
- Query → Path selection → Multimodal RAG → Final answer
110+
- Complete end-to-end orchestration
111+
112+
---
113+
114+
## Learning Format
115+
116+
**Asynchronous Lectures (Fridays)**
117+
- Watch videos on your schedule
118+
- Take notes and prepare questions
119+
120+
**Office Hours (Tuesdays & Thursdays)**
121+
- Multiple time zones supported
122+
- Active learning and discussion
123+
- Question-driven sessions
124+
125+
**Guest Lectures (Wednesdays)**
126+
- Industry experts and practitioners
127+
- Q&A with speakers
128+
- Real-world case studies
129+
130+
**Slack Community**
131+
- Ongoing discussions
132+
- Peer support and collaboration
133+
134+
---
135+
136+
## The Critical Mindset Shift
137+
138+
### ❌ Implementation Mindset
139+
- "We need to implement RAG"
140+
- Obsessing over embedding dimensions
141+
- Success = works in demo
142+
- Big upfront architecture decisions
143+
- Focus on picking "best" model
144+
145+
### ✅ Product Mindset
146+
- "We need to help users find answers faster"
147+
- Tracking answer relevance and task completion
148+
- Success = users keep coming back
149+
- Architecture that can evolve
150+
- Focus on learning from user behavior
151+
152+
**Launching your RAG system is just the beginning!**
153+
154+
---
155+
156+
## Why Most RAG Implementations Fail
157+
158+
**The Problem:** Treating RAG as a technical project, not a product
159+
160+
**What Happens:**
161+
1. Focus on technical components (embeddings, vector DB, LLM)
162+
2. Consider it "complete" when deployed
163+
3. Works for demos, struggles with real complexity
164+
4. Users lose trust as limitations surface
165+
5. No clear metrics or improvement process
166+
6. Resort to ad-hoc tweaking based on anecdotes
167+
168+
**The Solution:** Product mindset with continuous improvement
169+
170+
---
171+
172+
## The Key Insight: RAG as Recommendation Engine
173+
174+
**Stop thinking:** Retrieval → Augmentation → Generation
175+
176+
**Start thinking:** Recommendation Engine + Language Model
177+
178+
```
179+
User Query → Query Understanding → Multiple Retrieval Paths
180+
181+
[Document] [Image] [Table] [Code]
182+
183+
Filtering & Ranking
184+
185+
Context Assembly
186+
187+
Generation
188+
189+
User Response
190+
191+
Feedback Loop
192+
```
193+
194+
---
195+
196+
## What This Means
197+
198+
### 1. Generation Quality = Retrieval Quality
199+
- World's best prompt + garbage context = garbage answers
200+
- Focus on getting the right information to the LLM
201+
202+
### 2. Different Questions Need Different Strategies
203+
- Amazon doesn't recommend books like electronics
204+
- Your RAG shouldn't use same approach for every query
205+
206+
### 3. Feedback Drives Improvement
207+
- User interactions reveal what works
208+
- Continuous learning from real usage patterns
209+
210+
---
211+
212+
## What Does Success Look Like?
213+
214+
### Feeling of Success
215+
- **Less anxiety** when hearing "just make the AI better"
216+
- **Less overwhelmed** when told to "look at your data"
217+
- **Confidence** in making data-driven decisions
218+
219+
### Tangible Outcomes
220+
- Identify high-impact tasks systematically
221+
- Prioritize and make informed trade-offs
222+
- Choose metrics that correlate with business outcomes
223+
- Drive user satisfaction, retention, and usage
224+
225+
---
226+
227+
## The System Approach
228+
229+
**What is a System?**
230+
- Structured approach to solving problems
231+
- Framework for evaluating technologies
232+
- Decision-making process for prioritization
233+
- Methodology for diagnosing performance
234+
- Standard metrics and benchmarks
235+
236+
**Why Systems Matter:**
237+
- Frees mental energy for innovation
238+
- Replaces guesswork with testing
239+
- Enables quantitative vs "feels better" assessments
240+
- Secures resources through data-driven arguments
241+
242+
---
243+
244+
## RAG vs Recommendation Systems
245+
246+
**The Reality:** RAG is a 4-step recommendation system
247+
248+
1. **Multiple Retrieval Indices** (multimodal: images, tables, text)
249+
2. **Filtering** (top-k selection)
250+
3. **Scoring/Ranking** (rerankers, relevance)
251+
4. **Context Assembly** (prepare for generation)
252+
253+
**The Problem:** Engineers focus on generation without knowing if right information is retrieved
254+
255+
**The Solution:** Improve search to improve retrieval to improve generation
256+
257+
---
258+
259+
## Experimentation Over Implementation
260+
261+
**Instead of:** "Make the AI better"
262+
263+
**Ask:**
264+
- Why am I looking at this data?
265+
- What's the goal and hypothesis?
266+
- What signals am I looking for?
267+
- Is the juice worth the squeeze?
268+
- How can I use this to improve?
269+
270+
**Success Formula:** Flywheel in place + Consistent effort = Continuous improvement
271+
272+
Like building muscle: track calories and workouts, don't just weigh yourself daily
273+
274+
---
275+
276+
## Course Commitments
277+
278+
### My Commitment to You
279+
- Be online and answer questions
280+
- Provide extensive office hours support
281+
- Share real-world experience and case studies
282+
- Connect you with industry experts
283+
284+
### Your Commitment
285+
- Engage with the material actively
286+
- Look at your own data and systems
287+
- Participate in discussions and office hours
288+
- Apply learnings to your real projects
289+
290+
**Together, we'll transform your RAG from demo to production-ready product**
291+
292+
---
293+
294+
## Key Takeaway
295+
296+
> **Successful RAG systems aren't projects that ship once—they're products that improve continuously.**
297+
298+
The difference between success and failure isn't the embedding model or vector database you choose.
299+
300+
It's whether you treat RAG as:
301+
- **❌ Static implementation** that slowly decays
302+
- **✅ Living product** that learns from every interaction
303+
304+
**Let's build systems that get better every week! 🚀**
305+
306+
---
307+
308+
## Next Week
309+
310+
**Week 1: Kickstart the Data Flywheel**
311+
312+
- Synthetic data generation strategies
313+
- Building precision/recall evaluations
314+
- Creating your evaluation foundation
315+
- "Fake it till you make it" with synthetic data
316+
317+
**Come prepared to look at your data!**

0 commit comments

Comments
 (0)