Skip to content

Commit ade7048

Browse files
decipher3114decipher3114
committed
feat: add Notion connector
1 parent 30834bb commit ade7048

10 files changed

Lines changed: 1399 additions & 7 deletions

File tree

‎packages/moss-data-connector/README.md‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ moss-data-connector/
1010
├── moss-connector-sqlite/ # SQLite source (stdlib, no driver)
1111
├── moss-connector-mongodb/ # MongoDB source (requires pymongo)
1212
├── moss-connector-mysql/ # MySQL / MariaDB source (requires pymysql)
13+
├── moss-connector-notion/ # Notion source (requires notion-client)
1314
├── moss-connector-supabase/ # Supabase source (requires supabase)
1415
└── moss-connector-dynamodb/ # Amazon DynamoDB source (requires boto3)
1516
```
@@ -35,13 +36,14 @@ Use `auto_id=True` when your mapper does not have a stable primary key and you w
3536

3637
## Available connectors
3738

38-
| Package | Source | Extra driver |
39-
| ---------------------------------------------------------- | ------------- | ------------ |
40-
| [`moss-connector-sqlite`](moss-connector-sqlite) | SQLite | — |
41-
| [`moss-connector-mongodb`](moss-connector-mongodb) | MongoDB | `pymongo` |
42-
| [`moss-connector-mysql`](moss-connector-mysql) | MySQL | `pymysql` |
43-
| [`moss-connector-supabase`](moss-connector-supabase) | Supabase | `supabase` |
44-
| [`moss-connector-dynamodb`](moss-connector-dynamodb) | Amazon DynamoDB | `boto3` |
39+
| Package | Source | Extra driver |
40+
| --- | --- | --- |
41+
| [`moss-connector-sqlite`](moss-connector-sqlite) | SQLite | — |
42+
| [`moss-connector-mongodb`](moss-connector-mongodb) | MongoDB | `pymongo` |
43+
| [`moss-connector-mysql`](moss-connector-mysql) | MySQL | `pymysql` |
44+
| [`moss-connector-notion`](moss-connector-notion) | Notion | `notion-client` |
45+
| [`moss-connector-supabase`](moss-connector-supabase) | Supabase | `supabase` |
46+
| [`moss-connector-dynamodb`](moss-connector-dynamodb) | Amazon DynamoDB | `boto3` |
4547

4648
## Adding a new connector
4749

‎packages/moss-data-connector/moss-connector-notion/.gitignore‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
build/
2+
dist/
3+
*.egg-info/
4+
__pycache__/
5+
*.py[cod]
6+
.venv/
7+
.pytest_cache/
8+
.ruff_cache/
9+
.mypy_cache/
10+
.env

‎packages/moss-data-connector/moss-connector-notion/README.md‎

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
# moss-connector-notion
2+
3+
Notion source connector for Moss. Uses the official [notion-client](https://github.com/ramnes/notion-sdk-py) to read pages from a Notion workspace and ingest them into a Moss index.
4+
5+
## Install
6+
7+
```bash
8+
pip install moss-connector-notion
9+
```
10+
11+
This installs `notion-client` automatically.
12+
13+
## Usage
14+
15+
```python
16+
import asyncio
17+
18+
from moss import DocumentInfo
19+
from moss_connector_notion import NotionConnector, ingest
20+
21+
22+
def flatten_blocks(blocks: list[dict]) -> str:
23+
lines = []
24+
25+
def visit(block):
26+
block_type = block.get("type")
27+
28+
if block_type:
29+
payload = block.get(block_type, {})
30+
rich_text = payload.get("rich_text", [])
31+
32+
if rich_text:
33+
lines.append(
34+
"".join(part.get("plain_text", "") for part in rich_text)
35+
)
36+
37+
for child in block.get("children", []):
38+
visit(child)
39+
40+
for block in blocks:
41+
visit(block)
42+
43+
return "\n".join(lines)
44+
45+
46+
async def main():
47+
source = NotionConnector(
48+
token="your_notion_token",
49+
query="Rust",
50+
mapper=lambda page: DocumentInfo(
51+
id=page["id"],
52+
text=flatten_blocks(page["content"]),
53+
metadata={
54+
"url": page["url"],
55+
},
56+
),
57+
)
58+
59+
result = await ingest(
60+
source,
61+
project_id="your_project_id",
62+
project_key="your_project_key",
63+
index_name="notion-pages",
64+
)
65+
66+
print(f"Copied {result.doc_count} pages")
67+
68+
69+
asyncio.run(main())
70+
```
71+
72+
Use `auto_id=True` when your mapper does not have a stable document ID and you want Moss to generate UUID document IDs.
73+
74+
## Data requirements
75+
76+
The connector doesn't enforce a schema. Every Notion page is returned as a Python dictionary with one additional field:
77+
78+
```python
79+
page["content"]
80+
```
81+
82+
`content` contains the complete block tree for the page, including nested child blocks.
83+
84+
The connector passes this page dictionary directly to your mapper. The mapper is responsible for converting it into a `DocumentInfo`.
85+
86+
`DocumentInfo` fields:
87+
88+
| Field | Type | Required? | Typical Notion value |
89+
| --- | --- | --- | --- |
90+
| `id` | `str` | yes | `page["id"]` |
91+
| `text` | `str` | yes | extracted text from `page["content"]` |
92+
| `metadata` | `Optional[Dict[str, str]]` | no | page URL, title, author, etc. |
93+
| `embedding` | `Optional[Sequence[float]]` | no | only when using `model_id="custom"` |
94+
95+
A typical mapper looks like:
96+
97+
```python
98+
mapper=lambda page: DocumentInfo(
99+
id=page["id"],
100+
text=flatten_blocks(page["content"]),
101+
metadata={
102+
"url": page["url"],
103+
},
104+
)
105+
```
106+
107+
## One gotcha: metadata values must be strings
108+
109+
`DocumentInfo.metadata` expects `Dict[str, str]`.
110+
111+
If you include values that are numbers, booleans or lists, convert them first.
112+
113+
```python
114+
# Incorrect
115+
metadata={
116+
"views": 42,
117+
"published": True,
118+
}
119+
120+
# Correct
121+
metadata={
122+
"views": str(42),
123+
"published": str(True),
124+
}
125+
```
126+
127+
## Authentication
128+
129+
Create a Notion integration and copy its internal integration token.
130+
131+
The integration only has access to pages that have been explicitly shared with it.
132+
133+
If a page doesn't appear during ingestion:
134+
135+
1. Open the page in Notion.
136+
2. Click **Share**.
137+
3. Invite your integration.
138+
139+
## Searching pages
140+
141+
The connector optionally accepts a `query` argument.
142+
143+
```python
144+
NotionConnector(
145+
token=TOKEN,
146+
query="Python",
147+
...
148+
)
149+
```
150+
151+
This uses Notion's built-in search API and only returns matching pages.
152+
153+
If `query` is omitted, every page visible to the integration is scanned.
154+
155+
## Filtering
156+
157+
The connector supports client-side filtering through `filter_fn`.
158+
159+
```python
160+
source = NotionConnector(
161+
token=TOKEN,
162+
filter_fn=lambda page: page["url"].startswith("https://"),
163+
mapper=...,
164+
)
165+
```
166+
167+
The filter runs after the page and all of its blocks have been fetched.
168+
169+
It receives the complete page dictionary, including the `content` field.
170+
171+
## Nested blocks
172+
173+
Notion pages can contain nested blocks (toggles, lists, callouts, etc.).
174+
175+
The connector recursively fetches all child blocks before yielding the page, so `page["content"]` always contains the complete block tree.
176+
177+
The connector does **not** flatten or modify the content. This is left to the mapper so applications can decide how much structure to preserve.
178+
179+
## Pagination
180+
181+
The connector handles pagination automatically.
182+
183+
Both page search results and block children are fetched across multiple requests until all results have been retrieved.
184+
185+
No additional configuration is required.
186+
187+
## Layout
188+
189+
```
190+
src/
191+
├── __init__.py # re-exports NotionConnector and ingest
192+
├── connector.py # NotionConnector class
193+
└── ingest.py # ingest() - shared across connector packages
194+
```
195+
196+
## Tests
197+
198+
```bash
199+
pip install -e ".[dev]"
200+
201+
pytest tests/test_notion.py -v
202+
pytest tests/test_notion_integration.py -v -s
203+
```
204+
205+
The integration test requires:
206+
207+
- `NOTION_TOKEN`
208+
- `NOTION_PARENT_PAGE_ID`
209+
- `MOSS_PROJECT_ID`
210+
- `MOSS_PROJECT_KEY`
211+
212+
The test creates a temporary child page under `NOTION_PARENT_PAGE_ID`, ingests it into Moss, verifies semantic search, and archives the page after the test completes.

‎packages/moss-data-connector/moss-connector-notion/pyproject.toml‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
[project]
2+
name = "moss-connector-notion"
3+
version = "0.0.1"
4+
description = "Notion data connector for Moss"
5+
readme = "README.md"
6+
requires-python = ">=3.10,<3.15"
7+
license = { text = "BSD-2-Clause" }
8+
authors = [{ name = "InferEdge Inc.", email = "contact@moss.dev" }]
9+
keywords = ["connectors", "ingest", "moss", "notion"]
10+
classifiers = [
11+
"Development Status :: 3 - Alpha",
12+
"Intended Audience :: Developers",
13+
"License :: OSI Approved :: BSD License",
14+
"Programming Language :: Python :: 3",
15+
"Programming Language :: Python :: 3.10",
16+
"Programming Language :: Python :: 3.11",
17+
"Programming Language :: Python :: 3.12",
18+
"Programming Language :: Python :: 3.13",
19+
"Topic :: Database",
20+
]
21+
dependencies = [
22+
"moss>=1.6",
23+
"notion-client>=3.1",
24+
]
25+
26+
[project.urls]
27+
Homepage = "https://github.com/usemoss/moss"
28+
Repository = "https://github.com/usemoss/moss"
29+
Source = "https://github.com/usemoss/moss/tree/main/packages/moss-data-connector/moss-connector-notion"
30+
31+
[project.optional-dependencies]
32+
dev = [
33+
"pytest>=9.0",
34+
"pytest-asyncio>=1.0",
35+
"python-dotenv>=1.0",
36+
"ruff>=0.15",
37+
]
38+
39+
[build-system]
40+
requires = ["setuptools>=61.0"]
41+
build-backend = "setuptools.build_meta"
42+
43+
[tool.pytest.ini_options]
44+
asyncio_mode = "auto"
45+
46+
[tool.ruff]
47+
line-length = 100
48+
target-version = "py310"
49+
50+
[tool.ruff.lint]
51+
select = ["E", "W", "F", "I", "B", "UP"]
52+
53+
[tool.setuptools]
54+
packages = ["moss_connector_notion"]
55+
package-dir = { "moss_connector_notion" = "src" }

‎packages/moss-data-connector/moss-connector-notion/src/__init__.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
from .connector import NotionConnector
2+
from .ingest import ingest
3+
4+
__all__ = ["NotionConnector", "ingest"]

0 commit comments

Comments
 (0)