Skip to content

Commit 2009db6

Browse files
pmannion2pmannion2
committed
feat: added chunking
1 parent c192aea commit 2009db6

File tree

6 files changed

+499
-58
lines changed

6 files changed

+499
-58
lines changed

‎.env.example‎

Lines changed: 0 additions & 12 deletions
This file was deleted.

‎.gitmodules‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[submodule "whk-ignition-scada"]
2+
path = whk-ignition-scada
3+
url = https://github.com/WhiskeyHouse/whk-ignition-scada.git

‎indexer.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@
4242

4343
# Initialize tokenizer for GPT models
4444
enc = tiktoken.get_encoding("cl100k_base")
45-
MAX_TOKENS = 400 # Target max tokens per chunk
45+
MAX_TOKENS = 6000 # Increased from 400 to reduce API calls while staying under limits
4646

4747
# Initialize Chroma client
4848
PERSIST_DIRECTORY = "chroma_index"

0 commit comments

Comments
 (0)