Skip to content

Commit e3f818d

Browse files
authored
increase max concurrency to 50 (#203)
### Notes Tested concurrency settings with a 2500 page PDF. Notes on various settings: 15 threads: 29 minutes 50 threads: 11 minutes 150 threads: 5 minutes Memory usage: 15 threads: Partition of a set of 530464 objects. Total size = 150187404 bytes. (~150MB) 50 threads: Partition of a set of 530606 objects. Total size = 150206646 bytes. (~150MB) 150 threads: Partition of a set of 530654 objects. Total size = 150221029 bytes. (~150MB) For now proposing we increase this to a max of 50. We can always keep increasing later.
1 parent 99c6385 commit e3f818d

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

Diff for: src/unstructured_client/_hooks/custom/split_pdf_hook.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,8 @@
4040

4141
DEFAULT_STARTING_PAGE_NUMBER = 1
4242
DEFAULT_ALLOW_FAILED = False
43-
DEFAULT_CONCURRENCY_LEVEL = 8
44-
MAX_CONCURRENCY_LEVEL = 15
43+
DEFAULT_CONCURRENCY_LEVEL = 10
44+
MAX_CONCURRENCY_LEVEL = 50
4545
MIN_PAGES_PER_SPLIT = 2
4646
MAX_PAGES_PER_SPLIT = 20
4747

0 commit comments

Comments
 (0)