Basic Internal Search Tools gives error : Failed to process chat message due to list index out of range #5526
Description
.env:
GEN_AI_MAX_TOKENS=8192
Model: any GPT model
I set 8192 because i want to try later with smaller models like llama
Re-produce:
Create file connector, upload single file which is beyond GEN_AI_MAX_TOKENS like my file is ( Successfully embedded 3 texts with 6982 )
api_server-1 | ERROR: 09/28/2025 05:06:00 PM process_message.py 709: [API:mV9aBXmT] Failed to process chat message due to list index out of range
api_server-1 | Traceback (most recent call last):
api_server-1 | File "/app/onyx/chat/process_message.py", line 692, in stream_chat_message_objects
api_server-1 | yield from process_streamed_packets(
api_server-1 | File "/app/onyx/chat/packet_proccessing/process_streamed_packets.py", line 20, in process_streamed_packets
api_server-1 | for packet in answer_processed_output:
api_server-1 | File "/app/onyx/chat/answer.py", line 141, in processed_streamed_output
api_server-1 | for packet in stream:
api_server-1 | File "/app/onyx/agents/agent_search/run_graph.py", line 73, in run_dr_graph
api_server-1 | yield from run_graph(compiled_graph, config, input)
api_server-1 | File "/app/onyx/agents/agent_search/run_graph.py", line 47, in run_graph
api_server-1 | for event in manage_sync_streaming(
api_server-1 | File "/app/onyx/agents/agent_search/run_graph.py", line 33, in manage_sync_streaming
api_server-1 | for event in compiled_graph.stream(
api_server-1 | File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 1724, in stream
api_server-1 | for _ in runner.tick(
api_server-1 | File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/runner.py", line 302, in tick
api_server-1 | _panic_or_proceed(
api_server-1 | File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/runner.py", line 619, in _panic_or_proceed
api_server-1 | raise exc
api_server-1 | File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/executor.py", line 83, in done
api_server-1 | task.result()
api_server-1 | File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
api_server-1 | return self.__get_result()
api_server-1 | ^^^^^^^^^^^^^^^^^^^
api_server-1 | File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
api_server-1 | raise self._exception
api_server-1 | File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
api_server-1 | result = self.fn(*self.args, **self.kwargs)
api_server-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api_server-1 | File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/retry.py", line 40, in run_with_retry
api_server-1 | return task.proc.invoke(task.input, config)
api_server-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api_server-1 | File "/usr/local/lib/python3.11/site-packages/langgraph/utils/runnable.py", line 506, in invoke
api_server-1 | input = step.invoke(input, config, **kwargs)
api_server-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api_server-1 | File "/usr/local/lib/python3.11/site-packages/langgraph/utils/runnable.py", line 270, in invoke
api_server-1 | ret = context.run(self.func, *args, **kwargs)
api_server-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api_server-1 | File "/app/onyx/agents/agent_search/dr/nodes/dr_a3_logger.py", line 217, in logging
api_server-1 | save_iteration(
api_server-1 | File "/app/onyx/agents/agent_search/dr/nodes/dr_a3_logger.py", line 124, in save_iteration
api_server-1 | citation_dict[cited_doc_nr] = search_docs[cited_doc_nr - 1].id
api_server-1 | ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
api_server-1 | IndexError: list index out of range
api_server-1 | During task with name 'DRPath.LOGGER' and id '7c548685-7ee8-a97a-1c17-6bd6ff0b3b07'
api_server-1 | INFO: 09/28/2025 05:06:00 PM timing.py 76: [API:mV9aBXmT] stream_chat_message took 16.564602851867676 seconds
Indexing log:
indexing_model_server-1 | INFO: 09/28/2025 05:02:14 PM encoders.py 143: [DIP:A2Cbvzmm] Successfully embedded 8 texts with 21835 total characters with local model nomic-ai/nomic-embed-text-v1 in 3.05
indexing_model_server-1 | INFO: 09/28/2025 05:02:14 PM encoders.py 147: [DIP:A2Cbvzmm] event=embedding_model texts=8 chars=21835 model=nomic-ai/nomic-embed-text-v1 gpu=none elapsed=3.05
indexing_model_server-1 | NOTICE: 09/28/2025 05:02:14 PM utils.py 41: [DIP:A2Cbvzmm] embed_text took 3.0512733459472656 seconds
indexing_model_server-1 | INFO: 09/28/2025 05:02:14 PM h11_impl.py 499: [DIP:A2Cbvzmm] 172.18.0.9:55458 - "POST /encoder/bi-encoder-embed HTTP/1.1" 200
indexing_model_server-1 | INFO: 09/28/2025 05:02:14 PM encoders.py 121: [DIP:A2Cbvzmm] Embedding 3 texts with 6982 total characters with local model: nomic-ai/nomic-embed-text-v1
indexing_model_server-1 | INFO: 09/28/2025 05:02:15 PM encoders.py 143: [DIP:A2Cbvzmm] Successfully embedded 3 texts with 6982 total characters with local model nomic-ai/nomic-embed-text-v1 in 1.06
indexing_model_server-1 | INFO: 09/28/2025 05:02:15 PM encoders.py 147: [DIP:A2Cbvzmm] event=embedding_model texts=3 chars=6982 model=nomic-ai/nomic-embed-text-v1 gpu=none elapsed=1.06
indexing_model_server-1 | NOTICE: 09/28/2025 05:02:15 PM utils.py 41: [DIP:A2Cbvzmm] embed_text took 1.0601780414581299 seconds
indexing_model_server-1 | INFO: 09/28/2025 05:02:15 PM h11_impl.py 499: [DIP:A2Cbvzmm] 172.18.0.9:54984 - "POST /encoder/bi-encoder-embed HTTP/1.1" 200
indexing_model_server-1 | INFO: 09/28/2025 05:02:15 PM encoders.py 121: [DIP:A2Cbvzmm] Embedding 1 texts with 14 total characters with local model: nomic-ai/nomic-embed-text-v1
indexing_model_server-1 | INFO: 09/28/2025 05:02:15 PM encoders.py 143: [DIP:A2Cbvzmm] Successfully embedded 1 texts with 14 total characters with local model nomic-ai/nomic-embed-text-v1 in 0.04
indexing_model_server-1 | INFO: 09/28/2025 05:02:15 PM encoders.py 147: [DIP:A2Cbvzmm] event=embedding_model texts=1 chars=14 model=nomic-ai/nomic-embed-text-v1 gpu=none elapsed=0.04
indexing_model_server-1 | NOTICE: 09/28/2025 05:02:15 PM utils.py 41: [DIP:A2Cbvzmm] embed_text took 0.04292440414428711 seconds
indexing_model_server-1 | INFO: 09/28/2025 05:02:15 PM h11_impl.py 499: [DIP:A2Cbvzmm] 172.18.0.9:54994 - "POST /encoder/bi-encoder-embed HTTP/1.1" 200
Search Logs:
api_server-1 | NOTICE: 09/28/2025 05:03:14 PM timing.py 41: [API:tgcm7dpj] retrieval_preprocessing took 0.028 seconds
api_server-1 | WARNING: 09/28/2025 05:03:14 PM federated_retrieval.py 40: [API:tgcm7dpj] No user ID provided, skipping federated retrieval. Federated retrieval not supported with AUTH_TYPE=disabled.
inference_model_server-1 | INFO: 09/28/2025 05:03:14 PM encoders.py 121: [INF:E4y3dykB] Embedding 1 texts with 19 total characters with local model: nomic-ai/nomic-embed-text-v1
inference_model_server-1 | INFO: 09/28/2025 05:03:14 PM encoders.py 143: [INF:E4y3dykB] Successfully embedded 1 texts with 19 total characters with local model nomic-ai/nomic-embed-text-v1 in 0.09
inference_model_server-1 | INFO: 09/28/2025 05:03:14 PM encoders.py 147: [INF:E4y3dykB] event=embedding_model texts=1 chars=19 model=nomic-ai/nomic-embed-text-v1 gpu=none elapsed=0.09
inference_model_server-1 | NOTICE: 09/28/2025 05:03:14 PM utils.py 41: [INF:E4y3dykB] embed_text took 0.08868813514709473 seconds
inference_model_server-1 | INFO: 09/28/2025 05:03:14 PM h11_impl.py 499: [INF:E4y3dykB] 172.18.0.8:35780 - "POST /encoder/bi-encoder-embed HTTP/1.1" 200
api_server-1 | INFO: 09/28/2025 05:03:14 PM index.py 968: [API:tgcm7dpj] Selected ranking profile: hybrid_search_semantic_base_768
api_server-1 | NOTICE: 09/28/2025 05:03:14 PM timing.py 41: [API:tgcm7dpj] retrieval_preprocessing took 0.002 seconds
api_server-1 | WARNING: 09/28/2025 05:03:14 PM federated_retrieval.py 40: [API:tgcm7dpj] No user ID provided, skipping federated retrieval. Federated retrieval not supported with AUTH_TYPE=disabled.
inference_model_server-1 | INFO: 09/28/2025 05:03:14 PM encoders.py 121: [INF:xWd8pSvH] Embedding 1 texts with 23 total characters with local model: nomic-ai/nomic-embed-text-v1
inference_model_server-1 | INFO: 09/28/2025 05:03:14 PM encoders.py 143: [INF:xWd8pSvH] Successfully embedded 1 texts with 23 total characters with local model nomic-ai/nomic-embed-text-v1 in 0.05
inference_model_server-1 | INFO: 09/28/2025 05:03:14 PM encoders.py 147: [INF:xWd8pSvH] event=embedding_model texts=1 chars=23 model=nomic-ai/nomic-embed-text-v1 gpu=none elapsed=0.05
inference_model_server-1 | NOTICE: 09/28/2025 05:03:14 PM utils.py 41: [INF:xWd8pSvH] embed_text took 0.05399823188781738 seconds
inference_model_server-1 | INFO: 09/28/2025 05:03:14 PM h11_impl.py 499: [INF:xWd8pSvH] 172.18.0.8:35794 - "POST /encoder/bi-encoder-embed HTTP/1.1" 200
api_server-1 | INFO: 09/28/2025 05:03:14 PM index.py 968: [API:tgcm7dpj] Selected ranking profile: hybrid_search_semantic_base_768
api_server-1 | INFO: 09/28/2025 05:03:14 PM chunk_retrieval.py 395: [API:tgcm7dpj] Retrieved 27 inference chunks for 1 documents
api_server-1 | INFO: 09/28/2025 05:03:14 PM chunk_retrieval.py 395: [API:tgcm7dpj] Retrieved 27 inference chunks for 1 documents
api_server-1 | INFO: 09/28/2025 05:03:14 PM search_runner.py 249: [API:tgcm7dpj] Overall number of top initial retrieval chunks: 27
api_server-1 | INFO: 09/28/2025 05:03:14 PM search_runner.py 249: [API:tgcm7dpj] Overall number of top initial retrieval chunks: 27
api_server-1 | NOTICE: 09/28/2025 05:03:14 PM timing.py 41: [API:tgcm7dpj] doc_index_retrieval took 0.375 seconds
api_server-1 | NOTICE: 09/28/2025 05:03:14 PM timing.py 41: [API:tgcm7dpj] doc_index_retrieval took 0.235 seconds
api_server-1 | NOTICE: 09/28/2025 05:03:14 PM timing.py 41: [API:tgcm7dpj] _get_sections took 0.383 seconds
api_server-1 | NOTICE: 09/28/2025 05:03:14 PM timing.py 41: [API:tgcm7dpj] _get_sections took 0.243 seconds
api_server-1 | NOTICE: 09/28/2025 05:03:15 PM timing.py 41: [API:tgcm7dpj] retrieval_preprocessing took 0.001 seconds
api_server-1 | WARNING: 09/28/2025 05:03:15 PM federated_retrieval.py 40: [API:tgcm7dpj] No user ID provided, skipping federated retrieval. Federated retrieval not supported with AUTH_TYPE=disabled.
inference_model_server-1 | INFO: 09/28/2025 05:03:15 PM encoders.py 121: [INF:8tCFFsRZ] Embedding 1 texts with 17 total characters with local model: nomic-ai/nomic-embed-text-v1
inference_model_server-1 | INFO: 09/28/2025 05:03:15 PM encoders.py 143: [INF:8tCFFsRZ] Successfully embedded 1 texts with 17 total characters with local model nomic-ai/nomic-embed-text-v1 in 0.03
inference_model_server-1 | INFO: 09/28/2025 05:03:15 PM encoders.py 147: [INF:8tCFFsRZ] event=embedding_model texts=1 chars=17 model=nomic-ai/nomic-embed-text-v1 gpu=none elapsed=0.03
inference_model_server-1 | NOTICE: 09/28/2025 05:03:15 PM utils.py 41: [INF:8tCFFsRZ] embed_text took 0.036435604095458984 seconds
inference_model_server-1 | INFO: 09/28/2025 05:03:15 PM h11_impl.py 499: [INF:8tCFFsRZ] 172.18.0.8:35810 - "POST /encoder/bi-encoder-embed HTTP/1.1" 200
api_server-1 | INFO: 09/28/2025 05:03:15 PM index.py 968: [API:tgcm7dpj] Selected ranking profile: hybrid_search_semantic_base_768
api_server-1 | INFO: 09/28/2025 05:03:15 PM chunk_retrieval.py 395: [API:tgcm7dpj] Retrieved 27 inference chunks for 1 documents
api_server-1 | INFO: 09/28/2025 05:03:15 PM search_runner.py 249: [API:tgcm7dpj] Overall number of top initial retrieval chunks: 27
api_server-1 | NOTICE: 09/28/2025 05:03:15 PM timing.py 41: [API:tgcm7dpj] doc_index_retrieval took 0.090 seconds
api_server-1 | NOTICE: 09/28/2025 05:03:15 PM timing.py 41: [API:tgcm7dpj] _get_sections took 0.094 seconds
api_server-1 | NOTICE: 09/28/2025 05:03:16 PM timing.py 41: [API:tgcm7dpj] retrieval_preprocessing took 0.002 seconds
api_server-1 | WARNING: 09/28/2025 05:03:16 PM federated_retrieval.py 40: [API:tgcm7dpj] No user ID provided, skipping federated retrieval. Federated retrieval not supported with AUTH_TYPE=disabled.
inference_model_server-1 | INFO: 09/28/2025 05:03:16 PM encoders.py 121: [INF:XzU9NRLl] Embedding 1 texts with 17 total characters with local model: nomic-ai/nomic-embed-text-v1
inference_model_server-1 | INFO: 09/28/2025 05:03:16 PM encoders.py 143: [INF:XzU9NRLl] Successfully embedded 1 texts with 17 total characters with local model nomic-ai/nomic-embed-text-v1 in 0.03
inference_model_server-1 | INFO: 09/28/2025 05:03:16 PM encoders.py 147: [INF:XzU9NRLl] event=embedding_model texts=1 chars=17 model=nomic-ai/nomic-embed-text-v1 gpu=none elapsed=0.03
inference_model_server-1 | NOTICE: 09/28/2025 05:03:16 PM utils.py 41: [INF:XzU9NRLl] embed_text took 0.03460288047790527 seconds
inference_model_server-1 | INFO: 09/28/2025 05:03:16 PM h11_impl.py 499: [INF:XzU9NRLl] 172.18.0.8:35822 - "POST /encoder/bi-encoder-embed HTTP/1.1" 200
api_server-1 | INFO: 09/28/2025 05:03:16 PM index.py 968: [API:tgcm7dpj] Selected ranking profile: hybrid_search_semantic_base_768
api_server-1 | INFO: 09/28/2025 05:03:16 PM chunk_retrieval.py 395: [API:tgcm7dpj] Retrieved 27 inference chunks for 1 documents
api_server-1 | INFO: 09/28/2025 05:03:16 PM search_runner.py 249: [API:tgcm7dpj] Overall number of top initial retrieval chunks: 27
api_server-1 | NOTICE: 09/28/2025 05:03:16 PM timing.py 41: [API:tgcm7dpj] doc_index_retrieval took 0.082 seconds
api_server-1 | NOTICE: 09/28/2025 05:03:16 PM timing.py 41: [API:tgcm7dpj] _get_sections took 0.086 seconds
background-1 | INFO: INFO 09/28/2025 05:03:22 PM tasks.py:912 : [monitor_celery_queues(7e20f28b-9656-4beb-85cd-70a92accaabb)] Queue lengths: celery=0 docfetching=0 docfetching_prefetched=0 docprocessing=0 docprocessing_prefetched=0 user_files_indexing=0 sync=0 deletion=0 pruning=0 permissions_sync=0 external_group_sync=0 permissions_upsert=0
api_server-1 | ERROR: 09/28/2025 05:03:25 PM process_message.py 709: [API:tgcm7dpj] Failed to process chat message due to list index out of range
api_server-1 | Traceback (most recent call last):
api_server-1 | File "/app/onyx/chat/process_message.py", line 692, in stream_chat_message_objects
api_server-1 | yield from process_streamed_packets(