Merge pull request coleam00#213 from coleam00/fix/consolidate-concurrency-settings

Fix crawler concurrency configuration to prevent memory crashes

Author: Wirasm

.env.example

@@ -33,4 +33,9 @@ EMBEDDING_DIMENSIONS=1536
 # - OPENAI_API_KEY (encrypted)
 # - MODEL_CHOICE 
 # - TRANSPORT settings
-# - RAG strategy flags (USE_CONTEXTUAL_EMBEDDINGS, USE_HYBRID_SEARCH, etc.)
+# - RAG strategy flags (USE_CONTEXTUAL_EMBEDDINGS, USE_HYBRID_SEARCH, etc.)
+# - Crawler settings:
+#   * CRAWL_MAX_CONCURRENT (default: 10) - Max concurrent pages per crawl operation
+#   * CRAWL_BATCH_SIZE (default: 50) - URLs processed per batch
+#   * MEMORY_THRESHOLD_PERCENT (default: 80) - Memory % before throttling
+#   * DISPATCHER_CHECK_INTERVAL (default: 0.5) - Memory check interval in seconds