The complete guide to choosing and implementing the right AI framework for your project
Quick Decision Tree:
Need RAG + chat? → LangChain
Just search/retrieval? → LlamaIndex
Maximum control? → Custom (OpenAI + Pinecone)
Production scale? → LangChain + Custom hybrid
Prototyping fast? → LlamaIndex
| Feature | LangChain | LlamaIndex | Custom (OpenAI + Vector DB) |
|---|---|---|---|
| Learning Curve | Steep | Moderate | Low (if you know Python) |
| RAG Performance | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ (depends on your skill) |
| Customization | ⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| Production Ready | ⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐ (you control everything) |
| Documentation | ⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ (official APIs) |
| Community | 🔥 Huge | 🔥 Growing | ✅ Stable |
| Breaking Changes | ✅ Rare | ||
| Cost | Free (OSS) | Free (OSS) | Pay per API call |
- Building conversational agents with memory
- Need multi-step reasoning chains
- Want to connect multiple tools (SQL, APIs, search)
- Building complex workflows
- Just need simple retrieval
- Can't handle breaking changes
- Need minimal dependencies
from langchain.chat_models import ChatOpenAI
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Pinecone
from langchain.chains import RetrievalQA
from langchain.document_loaders import TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
# 1. Load documents
loader = TextLoader('your_docs.txt')
documents = loader.load()
# 2. Chunk them
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200
)
chunks = text_splitter.split_documents(documents)
# 3. Embed and store
embeddings = OpenAIEmbeddings()
vectorstore = Pinecone.from_documents(
chunks,
embeddings,
index_name='your-index'
)
# 4. Create RAG chain
llm = ChatOpenAI(model='gpt-4', temperature=0)
qa_chain = RetrievalQA.from_chain_type(
llm=llm,
chain_type='stuff',
retriever=vectorstore.as_retriever(search_kwargs={'k': 3})
)
# 5. Ask questions
result = qa_chain.run("What is RAG?")
print(result)from langchain.memory import ConversationBufferMemory
from langchain.chains import ConversationalRetrievalChain
# Add memory to remember chat history
memory = ConversationBufferMemory(
memory_key='chat_history',
return_messages=True,
output_key='answer'
)
# Create conversational chain
conv_chain = ConversationalRetrievalChain.from_llm(
llm=llm,
retriever=vectorstore.as_retriever(),
memory=memory,
return_source_documents=True
)
# Multi-turn conversation
response1 = conv_chain({"question": "What is RAG?"})
response2 = conv_chain({"question": "How do I implement it?"})
# ^ Remembers context from first question!- Use
RecursiveCharacterTextSplitterfor most cases - Set
chunk_overlap=200to avoid context loss - Use
stuffchain type for < 4k tokens,map_reducefor larger - Always set
temperature=0for RAG (deterministic) - Use
return_source_documents=Truefor citations
- Primary use case is search/retrieval
- Want best-in-class indexing strategies
- Need to query structured data (SQL, CSV, JSON)
- Building knowledge bases
- Need complex multi-step reasoning
- Want to build conversational agents
- Need extensive tool integration
from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
from llama_index.llms import OpenAI
from llama_index.embeddings import OpenAIEmbedding
# 1. Load documents (auto-detects file types!)
documents = SimpleDirectoryReader('data/').load_data()
# 2. Configure LLM and embeddings
llm = OpenAI(model='gpt-4', temperature=0)
embed_model = OpenAIEmbedding()
service_context = ServiceContext.from_defaults(
llm=llm,
embed_model=embed_model,
chunk_size=512
)
# 3. Create index (auto-chunks and embeds!)
index = VectorStoreIndex.from_documents(
documents,
service_context=service_context
)
# 4. Query
query_engine = index.as_query_engine()
response = query_engine.query("What is RAG?")
print(response)That's it! Way simpler than LangChain for basic RAG.
from llama_index.tools import QueryEngineTool, ToolMetadata
from llama_index.agent import OpenAIAgent
# Create separate indexes for different doc types
doc_index = VectorStoreIndex.from_documents(documentation_docs)
api_index = VectorStoreIndex.from_documents(api_docs)
tutorial_index = VectorStoreIndex.from_documents(tutorial_docs)
# Wrap as tools
doc_tool = QueryEngineTool(
query_engine=doc_index.as_query_engine(),
metadata=ToolMetadata(
name="documentation",
description="Search official documentation"
)
)
api_tool = QueryEngineTool(
query_engine=api_index.as_query_engine(),
metadata=ToolMetadata(
name="api_reference",
description="Search API reference"
)
)
# Create agent that picks the right index
agent = OpenAIAgent.from_tools(
[doc_tool, api_tool, tutorial_tool],
verbose=True
)
# Agent automatically chooses which index to query
response = agent.chat("How do I authenticate API requests?")
# ^ Will use api_tool automatically- Use
SimpleDirectoryReader- it's magical (auto-detects PDFs, CSVs, etc.) - Default chunk size (512) works great for most cases
- Use
response_mode='tree_summarize'for long documents - Check
response.source_nodesfor citations - Use
VectorStoreIndex.from_vector_store()to connect to Pinecone/Weaviate
- You need 100% control
- Can't tolerate breaking changes
- Want minimal dependencies
- Building production systems
- Framework overhead is too much
- Prototyping quickly
- Need built-in features (memory, agents, tools)
- Small team without ML expertise
import openai
import pinecone
from typing import List, Dict
class CustomRAG:
"""Production-ready RAG with full control"""
def __init__(self, pinecone_index: str, openai_key: str):
openai.api_key = openai_key
pinecone.init(api_key='your-key', environment='your-env')
self.index = pinecone.Index(pinecone_index)
def chunk_text(self, text: str, chunk_size: int = 1000) -> List[str]:
"""Simple recursive chunking"""
if len(text) <= chunk_size:
return [text]
# Try to split on paragraph
mid = len(text) // 2
split_chars = ['\n\n', '\n', '. ', ' ']
for char in split_chars:
split_idx = text.rfind(char, mid - 200, mid + 200)
if split_idx != -1:
return (
self.chunk_text(text[:split_idx]) +
self.chunk_text(text[split_idx:])
)
# Fallback: hard split
return [text[:chunk_size]] + self.chunk_text(text[chunk_size:])
def embed(self, texts: List[str]) -> List[List[float]]:
"""Batch embedding with retry logic"""
response = openai.Embedding.create(
input=texts,
model='text-embedding-ada-002'
)
return [item['embedding'] for item in response['data']]
def index_documents(self, documents: List[str], metadata: List[Dict] = None):
"""Chunk, embed, and store documents"""
all_chunks = []
all_metadata = []
for i, doc in enumerate(documents):
chunks = self.chunk_text(doc)
for j, chunk in enumerate(chunks):
all_chunks.append(chunk)
meta = metadata[i] if metadata else {}
meta.update({'chunk_id': j, 'doc_id': i})
all_metadata.append(meta)
# Batch embed (OpenAI supports up to 2048 texts per request)
embeddings = self.embed(all_chunks)
# Upsert to Pinecone
vectors = [
(f'doc_{i}', emb, meta)
for i, (emb, meta) in enumerate(zip(embeddings, all_metadata))
]
self.index.upsert(vectors=vectors)
print(f"✅ Indexed {len(all_chunks)} chunks from {len(documents)} documents")
def retrieve(self, query: str, top_k: int = 3) -> List[Dict]:
"""Semantic search"""
query_embedding = self.embed([query])[0]
results = self.index.query(
vector=query_embedding,
top_k=top_k,
include_metadata=True
)
return [
{
'text': match['metadata'].get('text'),
'score': match['score'],
'metadata': match['metadata']
}
for match in results['matches']
]
def generate(self, query: str, context: List[str]) -> str:
"""LLM generation with retrieved context"""
context_str = '\n\n'.join([f"[{i+1}] {ctx}" for i, ctx in enumerate(context)])
prompt = f"""Use the following context to answer the question.
If the answer isn't in the context, say "I don't have enough information."
Context:
{context_str}
Question: {query}
Answer:"""
response = openai.ChatCompletion.create(
model='gpt-4',
messages=[
{'role': 'system', 'content': 'You are a helpful assistant that answers questions based on provided context.'},
{'role': 'user', 'content': prompt}
],
temperature=0
)
return response['choices'][0]['message']['content']
def query(self, question: str, top_k: int = 3) -> Dict:
"""Full RAG pipeline"""
# Retrieve
retrieved = self.retrieve(question, top_k=top_k)
contexts = [item['text'] for item in retrieved]
# Generate
answer = self.generate(question, contexts)
return {
'answer': answer,
'sources': retrieved
}
# Usage
rag = CustomRAG(pinecone_index='my-index', openai_key='sk-...')
# Index
rag.index_documents([
"RAG stands for Retrieval Augmented Generation...",
"Vector databases store embeddings..."
])
# Query
result = rag.query("What is RAG?")
print(result['answer'])
print(f"\nSources: {len(result['sources'])}")- ✅ 100 lines, zero framework dependencies
- ✅ Full control over chunking, retrieval, generation
- ✅ Easy to debug (no abstraction layers)
- ✅ Production-ready with error handling
- ✅ No breaking changes from framework updates
→ Use LlamaIndex
- Fastest to prototype
- Best documentation for beginners
- Built-in support for multiple file types
→ Use LangChain + Custom hybrid
- LangChain for conversation management
- Custom retrieval for performance optimization
- Flexibility to replace components as needed
→ Custom Solution
- No dependency risk
- Complete control over costs
- Easy to maintain long-term
→ LangChain
- Most features out-of-the-box
- Agent capabilities for exploration
- Largest community for weird use cases
# Before (LangChain)
from langchain.chains import RetrievalQA
qa_chain = RetrievalQA.from_chain_type(llm, retriever=retriever)
result = qa_chain.run("question")
# After (Custom)
retrieved = rag.retrieve("question", top_k=3)
result = rag.generate("question", [r['text'] for r in retrieved])# Convert LlamaIndex retriever to LangChain
from langchain.retrievers import LlamaIndexRetriever
llama_index = VectorStoreIndex.from_documents(docs)
retriever = LlamaIndexRetriever(index=llama_index)
# Now use with LangChain
from langchain.chains import RetrievalQA
qa_chain = RetrievalQA.from_chain_type(llm, retriever=retriever)Example: 10,000 document chunks, 1,000 queries/day
| Component | LangChain | LlamaIndex | Custom |
|---|---|---|---|
| Embedding (one-time) | $0.40 | $0.40 | $0.40 |
| Vector DB storage | $70/mo | $70/mo | $70/mo |
| LLM calls (1k/day) | $120/mo | $120/mo | $120/mo |
| Framework overhead | ~5-10% slower | ~3-5% slower | Fastest |
| Total | ~$190/mo | ~$190/mo | ~$190/mo |
Winner: Cost is the same! Choose based on features.
pip install llama-index
# 10 lines of code → working RAGpip install openai pinecone-client
# 100 lines of code → full controlpip install langchain
# Multi-step reasoning, tools, memoryWeek 1: Pick a framework, build basic RAG Week 2: Optimize chunking and retrieval Week 3: Add evaluation metrics Week 4: Deploy to production
- Start with custom, add framework later - Easier to understand what's happening
- Don't over-optimize early - Get it working first
- Log everything - You'll need it for debugging
- Version your embeddings - Model updates = re-embed everything
- Test with real users ASAP - Synthetic tests lie
Questions? Issues?
Check official docs:
- LangChain: https://python.langchain.com
- LlamaIndex: https://docs.llamaindex.ai
- OpenAI: https://platform.openai.com/docs
Made with ❤️ for AI builders
Share this guide with your team!