How many text files are supported for importing, whenever I import files larger than 1w words, I can't import them

[cdxiaodong]

Use Case

How many text files are supported for importing, whenever I import files larger than 1w words, I can't import them

Problem Statement

whenever I import files larger than 1w words, I can't import them

How This Feature Would Help

Since Hindsight is usually meant for large-scale data processing, I'm testing its ability to analyze and perform 'reflect' operations on large amounts of data.

Proposed Solution

No response

Alternatives Considered

Perhaps this is related to the AI's native context capacity, so we could adopt alternative approaches, such as chunking.

Priority

Important - affects my workflow

Additional Context

No response

Checklist

• I would be willing to contribute this feature

[nicoloboschi]

@cdxiaodong there's no hard limit on that. hindsight auto split large files into batches and run them with a backpressure mechanism.
can you check the api logs?

[cdxiaodong]

Okay, the following is the log of the error.

2026-01-12 08:54:14,007 - INFO - hindsight_api.engine.llm_wrapper - slow llm call: model=openai/deepseek-chat, input_tokens=6034, output_tokens=3908, total_tokens=9942, cached_tokens=4672, time=133.292s, ratio out/in=0.65 2026-01-12 08:54:18,906 - WARNING - hindsight_api.engine.llm_wrapper - JSON parse error from LLM response (attempt 1/11): Unterminated string starting at: line 448 column 29 (char 18105) Model: openai/deepseek-chat Content length: 18106 chars Content preview: '{\n "facts": [\n {\n "what": "The financial system is considered a core asset, and obtaining financial data would make a report more substantial",\n "when": "N/A",\n "where": "N/A",\n "who": "The attackers (likely including 老K, 阿鬼, 米罗, 影子) as part of their penetration testing or hacking operation",\n "why": "To emphasize the importance of targeting financial systems for gaining significant evidence or leverage in their report",\n ...TRUNCATED...occurred_start": null,\n "occurred_end": null,\n "entities": [\n {\n "text": "老K"\n },\n {\n "text": "' Finish reason: length 2026-01-12 08:54:19,293 - WARNING - hindsight_api.engine.llm_wrapper - JSON parse error from LLM response (attempt 1/11): Expecting value: line 363 column 25 (char 17389) Model: openai/deepseek-chat Content length: 17389 chars Content preview: '{\n "facts": [\n {\n "what": "User (米罗) looked at the process list using \'tasklist /svc\' and identified processes including egui.exe (ESET GUI), ekrn.exe (ESET Service), and TaniumClient.exe, noting that TaniumClient.exe has real-time asset retrieval capabilities and is annoying",\n "when": "Thursday, May 20, 2026 at 13:45:20",\n "where": "On the target server (OA server Web-01) with System permissions",\n "who": "米罗 (user/operator), ESET (security...TRUNCATED...ion file to bypass authentication on Jenkins, demonstrating a common security weakness and gaining unauthorized access to a critical system",\n "fact_type": "world",\n "fact_kind":' Finish reason: length