Skip to content

Commit 0f72971

Browse files
bryan-unstructuredmr-unstructured
bryan-unstructured
and
mr-unstructured
authored
feat: Migrate Notion source to Connector V2 structure (#162)
* Mantain the current v1 file * finished * Black format and changelog add * Fix Makefile issue with lint * more fixes * restore main files * More restored files from main * removed useless variable * Optimized imports * few ruff fixes * last ruff fix * version updated * added notion-client to base.in * remove unused error * changed reference * ruff fix * Fixed and saving files now * addressed * Roman Access Config request addressed * Library type_check done * black fix * version file matching * More libraries that needed to be capsulated * Remove leftover comment * Multiple PR changes assigned * fixes * tries * More Client * most done * missed this * trying * black * version change * async client * connector.py updates * autopep8 updates * Roman comments addressed * params issue * stop ignoring Notion * my bad, versions dont match * migrate notion source connector to V2 * add integration tests for downloading notion database * fix expected output files in notion e2e test * make sure the recursive child block getter to point at the next page / cursor * fix syntax * fix block retrieval logic * remove unnecessary e2e test for notion connector * Add more complex integration test --------- Co-authored-by: mr-unstructured <[email protected]> Co-authored-by: Bryan Chen <[email protected]>
1 parent 2491652 commit 0f72971

File tree

91 files changed

+4832
-1077
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

91 files changed

+4832
-1077
lines changed

Diff for: ‎CHANGELOG.md

+1
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99

1010
### Enhancements
1111

12+
* **Migrate Notion Source Connector to V2**
1213
* **Migrate Vectara Destination Connector to v2**
1314
* **Added Redis destination connector**
1415
* **Improved Milvus error handling**

Diff for: ‎test/integration/connectors/expected_results/notion_database/directory_structure.json

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"directory_structure": [
3+
"1722c3765a0a8082b382ebc2c62d3f4c.html"
4+
]
5+
}

Diff for: ‎test/integration/connectors/expected_results/notion_database/downloads/1722c3765a0a8082b382ebc2c62d3f4c.html

+330
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,330 @@
1+
<table>
2+
<tr>
3+
<th>
4+
Author
5+
</th>
6+
<th>
7+
Email
8+
</th>
9+
<th>
10+
Formula
11+
</th>
12+
<th>
13+
ID
14+
</th>
15+
<th>
16+
Item
17+
</th>
18+
<th>
19+
Phone
20+
</th>
21+
<th>
22+
Priority Level
23+
</th>
24+
<th>
25+
Publication Date
26+
</th>
27+
<th>
28+
Status
29+
</th>
30+
<th>
31+
Tag
32+
</th>
33+
<th>
34+
URL
35+
</th>
36+
<th>
37+
Views
38+
</th>
39+
</tr>
40+
<tr>
41+
<td>
42+
<div>
43+
<span>
44+
<a href='https://lh3.googleusercontent.com/a/AATXAJyU_JkTO5JUkbJ8l_RuIHq4K3KdPghBV0r9U_GU=s100'>
45+
Brian Raymond
46+
</a>
47+
</span>
48+
</div>
49+
</td>
50+
<td>
51+
<div>
52+
[email protected]
53+
</div>
54+
</td>
55+
<td>
56+
<div>
57+
12
58+
</div>
59+
</td>
60+
<td>
61+
<div>
62+
None-4
63+
</div>
64+
</td>
65+
<td>
66+
<div>
67+
test-page4-in-database
68+
</div>
69+
</td>
70+
<td>
71+
<div>
72+
1234567890
73+
</div>
74+
</td>
75+
<td>
76+
<div style='color: gray'>
77+
High
78+
</div>
79+
</td>
80+
<td>
81+
<div>
82+
2025-01-31
83+
</div>
84+
</td>
85+
<td>
86+
<div>
87+
Not started
88+
</div>
89+
</td>
90+
<td>
91+
<div>
92+
<span style='color: purple'>
93+
V1
94+
</span>
95+
<span style='color: yellow'>
96+
V5
97+
</span>
98+
<span style='color: gray'>
99+
V7
100+
</span>
101+
</div>
102+
</td>
103+
<td>
104+
<a href='https://abcde.com'>
105+
https://abcde.com
106+
</a>
107+
</td>
108+
<td>
109+
<div>
110+
6
111+
</div>
112+
</td>
113+
</tr>
114+
<tr>
115+
<td>
116+
<div>
117+
<span>
118+
<a href='https://lh3.googleusercontent.com/a/AATXAJyU_JkTO5JUkbJ8l_RuIHq4K3KdPghBV0r9U_GU=s100'>
119+
Brian Raymond
120+
</a>
121+
</span>
122+
</div>
123+
</td>
124+
<td>
125+
<div>
126+
[email protected]
127+
</div>
128+
</td>
129+
<td>
130+
<div>
131+
90
132+
</div>
133+
</td>
134+
<td>
135+
<div>
136+
None-3
137+
</div>
138+
</td>
139+
<td>
140+
<div>
141+
test-page3-in-database
142+
</div>
143+
</td>
144+
<td>
145+
<div>
146+
1234567890
147+
</div>
148+
</td>
149+
<td>
150+
<div style='color: pink'>
151+
Medium
152+
</div>
153+
</td>
154+
<td>
155+
<div>
156+
2025-01-06
157+
</div>
158+
</td>
159+
<td>
160+
<div style='color: orange'>
161+
In Review
162+
</div>
163+
</td>
164+
<td>
165+
<div>
166+
<span style='color: yellow'>
167+
V5
168+
</span>
169+
<span>
170+
V6
171+
</span>
172+
</div>
173+
</td>
174+
<td>
175+
<a href='https://abcde.com'>
176+
https://abcde.com
177+
</a>
178+
</td>
179+
<td>
180+
<div>
181+
45
182+
</div>
183+
</td>
184+
</tr>
185+
<tr>
186+
<td>
187+
<div>
188+
<span>
189+
<a href='https://lh3.googleusercontent.com/a/AATXAJyU_JkTO5JUkbJ8l_RuIHq4K3KdPghBV0r9U_GU=s100'>
190+
Brian Raymond
191+
</a>
192+
</span>
193+
</div>
194+
</td>
195+
<td>
196+
<div>
197+
[email protected]
198+
</div>
199+
</td>
200+
<td>
201+
<div>
202+
46
203+
</div>
204+
</td>
205+
<td>
206+
<div>
207+
None-2
208+
</div>
209+
</td>
210+
<td>
211+
<div>
212+
test-page2-in-database
213+
</div>
214+
</td>
215+
<td>
216+
<div>
217+
1234567890
218+
</div>
219+
</td>
220+
<td>
221+
<div style='color: blue'>
222+
Low
223+
</div>
224+
</td>
225+
<td>
226+
<div>
227+
2025-01-04
228+
</div>
229+
</td>
230+
<td>
231+
<div style='color: green'>
232+
Done
233+
</div>
234+
</td>
235+
<td>
236+
<div>
237+
<span style='color: purple'>
238+
V1
239+
</span>
240+
<span style='color: orange'>
241+
V2
242+
</span>
243+
<span style='color: brown'>
244+
V4
245+
</span>
246+
</div>
247+
</td>
248+
<td>
249+
<a href='https://abcde.com'>
250+
https://abcde.com
251+
</a>
252+
</td>
253+
<td>
254+
<div>
255+
23
256+
</div>
257+
</td>
258+
</tr>
259+
<tr>
260+
<td>
261+
<div>
262+
<span>
263+
<a href='https://lh3.googleusercontent.com/a/AATXAJyU_JkTO5JUkbJ8l_RuIHq4K3KdPghBV0r9U_GU=s100'>
264+
Brian Raymond
265+
</a>
266+
</span>
267+
</div>
268+
</td>
269+
<td>
270+
<div>
271+
[email protected]
272+
</div>
273+
</td>
274+
<td>
275+
<div>
276+
4
277+
</div>
278+
</td>
279+
<td>
280+
<div>
281+
None-1
282+
</div>
283+
</td>
284+
<td>
285+
<div>
286+
test-page1-in-datab
287+
</div>
288+
</td>
289+
<td>
290+
<div>
291+
1234567890
292+
</div>
293+
</td>
294+
<td>
295+
<div style='color: gray'>
296+
High
297+
</div>
298+
</td>
299+
<td>
300+
<div>
301+
2024-12-01
302+
</div>
303+
</td>
304+
<td>
305+
<div style='color: blue'>
306+
In progress
307+
</div>
308+
</td>
309+
<td>
310+
<div>
311+
<span style='color: purple'>
312+
V1
313+
</span>
314+
<span style='color: green'>
315+
V3
316+
</span>
317+
</div>
318+
</td>
319+
<td>
320+
<a href='https://abcde.com'>
321+
https://abcde.com
322+
</a>
323+
</td>
324+
<td>
325+
<div>
326+
2
327+
</div>
328+
</td>
329+
</tr>
330+
</table>

Diff for: ‎test/integration/connectors/expected_results/notion_database/file_data/1722c3765a0a8082b382ebc2c62d3f4c.json

+39
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
{
2+
"identifier": "1722c3765a0a8082b382ebc2c62d3f4c",
3+
"connector_type": "notion",
4+
"source_identifiers": {
5+
"filename": "1722c3765a0a8082b382ebc2c62d3f4c.html",
6+
"fullpath": "1722c3765a0a8082b382ebc2c62d3f4c.html",
7+
"rel_path": "1722c3765a0a8082b382ebc2c62d3f4c.html"
8+
},
9+
"metadata": {
10+
"url": null,
11+
"version": null,
12+
"record_locator": {
13+
"database_id": "1722c3765a0a8082b382ebc2c62d3f4c"
14+
},
15+
"date_created": "2025-01-05T18:34:00.000Z",
16+
"date_modified": "2025-01-07T19:15:00.000Z",
17+
"date_processed": "1736277913.3980532",
18+
"permissions_data": null,
19+
"filesize_bytes": null
20+
},
21+
"additional_metadata": {
22+
"created_by": {
23+
"id": "118d872b-594c-8171-b46f-00020d10d8b2",
24+
"object": "user"
25+
},
26+
"last_edited_by": {
27+
"id": "118d872b-594c-8171-b46f-00020d10d8b2",
28+
"object": "user"
29+
},
30+
"parent": {
31+
"type": "workspace",
32+
"workspace": true
33+
},
34+
"url": "https://www.notion.so/1722c3765a0a8082b382ebc2c62d3f4c"
35+
},
36+
"reprocess": false,
37+
"local_download_path": "/private/var/folders/h7/n848df9s5yn7ml8rxb61vhyc0000gp/T/tmpxu906ary/1722c3765a0a8082b382ebc2c62d3f4c.html",
38+
"display_name": null
39+
}

Diff for: ‎test/integration/connectors/expected_results/notion_page/directory_structure.json

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"directory_structure": [
3+
"1572c3765a0a806299f0dd6999f9e4c7.html"
4+
]
5+
}

0 commit comments

Comments
 (0)