Notion Chatbot

• Deep-crawling dynamic JS-support web scraper to find all subpages of a given URL: src/scraping/scrape.py.
• Content chunking, NLP embedding, and vector storing pipeline: src/processing/
• An RAG chatbot that can answer questions based on context retrieved from a chroma database: src/rag/rag_chat.py.
• Gradio demo app: gradio_app/app.py.
• Development journal: Journal

Introduction

As an organization grows large, the huge Notion workplace can be incredibly difficult to navigate, with possibly thousands of pages. A context-aware chatbot can help answer organization- or mission- specific questions, thus born this project.

The project consists of three stages: scraping all Notion subpages of a URL recursively, chunking documents and creating/storing NLP vector embeddings, and orchestrating retrieval-augmented agent.

Demo

This demo has been made private due to privacy concern: notion-chatbot

Primary Libraries & Frameworks

Python Version: 3.13.9

• Dynamic Recursive Web Scraping: Playwright, BeautifulSoup
• Embedding & Vector Storage: LangChain, OpenAI API, Chroma
• RAG Agent Workflow: LangChain, LangGraph, OpenAI API
• Demo Application: Gradio, Hugging Face

Quick Start: make your own RAG chatbot

In your desired directory, clone the repo:

git clone https://github.com/Preston-Cai/notion-chatbot

Install dependencies:

pip install -r requirements.txt

If failed, try loosening the dependencies.

0. Config

Since this project uses OpenAI API for embedding and LLM inference, it does not support other API (as of now).
To config your OpenAI API key, run the following commands in your project root:

echo OPENAI_API_KEY = '#put your api key here in this pair of quotes' > .env

1. Scraping

To scrap an URL recursively (dynamic JS supported):

1. In src/scraping/scrape.py, scroll down to the main block. Adjust the parameters of the function.
2. Run in terminal:

python -m src.scraping.scrape

3. For a URL with around 1000 children links, for cap=8 it should take around 30 minutes.
4. Find the data in data/scraping.

2. Chunking/Embedding/Vector Storing

Chunk, embed, and vector store the content in data/scraping that has been generated in step 1.

For data/scraping/json_docs

• Recommended approach. Generated JSON docs contain the sources that the RAG agent needs to generate the correct response format.

Run in terminal:

python -m src.processing.embed_with_source

For data/scraping/text_docs

• Not recommended for later RAG agent workflow.

Run in terminal:

python -m src.processing.embed_no_source

3. Launching RAG Chatbot

To config the context and prompt for your chatbot:

1. Open up src/rag/rag_chat.py and edit SYSTEM_PROMPT and the interface (docstring and decorator description) of the _retrieve_context tool with the @tool decorator.
2. Create a file data/context/big_context.json and enter your big-picture context for your chatbot. Ideally the content should be in JSON dict format, but the app won't break if it isn't.

To simulate a command line chat loop, run in terminal:

python -m src.rag.rag_chat

To open up a gradio demo app in browser, run in terminal:

python -m gradio_app.app

Project Tree

📦 
├─ .gitattributes
├─ .gitignore
├─ README.md
├─ development-journal.md
├─ experimental_legacy/     # experimental/exploratory files
├─ experimental_requirements.txt    # requirements for experimental files
├─ gradio_app
│  └─ app.py
├─ requirements.txt
└─ src
   ├─ __init__.py
   ├─ file_config.py
   ├─ processing/
   ├─ rag/
   ├─ scraping/
   ├─ tests/
   └─ utils/

General Next Steps / Possible Expansions

1. Isolate the dynamic recursive scraper and make it generic and pip installable.
2. Command line interface for scraping, embedding, and launching agent.
3. Modern web app development: dynamically fetching changes from Notion page, multi-user support & session managemenet, persistent storage & databases, asynchronous backend processing, scalable architecture.

For more details, view the "Next Steps" section in Development Journal.