Flesh out RAG system (#6197)

# Description of Changes
Flesh out the RAG system and connect it to the PDF Question Agent so it
can respond to questions about PDFs of an extremely large size.

I'd expect lots more work will need to be done to finish off the RAG
system to really be what we need, but this should be a reasonable start
which will let us connect it to tools and have the ingestion mostly
handled automatically. I'm leaving file deletion and proper file ID
management to be done in a future PR. We also need to consider whether
all tools should retrieve content exclusively via RAG, or whether it's
beneficial to have tools sometimes fetch the direct content and other
times fetch it from RAG.

A diagram of the expected interaction is as follows:

```mermaid
sequenceDiagram
    autonumber
    actor U as User
    participant FE as Frontend<br/>(ChatPanel)
    participant J as Java<br/>(AiWorkflowService)
    participant O as Engine:<br/>OrchestratorAgent
    participant QA as Engine:<br/>PdfQuestionAgent
    participant RAG as Engine:<br/>RagService + SqliteVecStore
    participant V as VoyageAI<br/>(embeddings)
    participant L as LLM<br/>(Claude / etc.)

    U->>FE: types "Summarise this PDF"<br/>(PDF already uploaded)
    FE->>J: POST /api/v1/ai/orchestrate/stream<br/>multipart: fileInputs[], userMessage
    Note over J: ByteHashFileIdStrategy<br/>id = sha256(bytes)[:16]
    J->>O: POST /api/v1/orchestrator<br/>{ files:[{id,name}], userMessage }

    O->>L: route via fast model
    L-->>O: delegate_pdf_question
    O->>QA: PdfQuestionRequest

    loop for each file
        QA->>RAG: has_collection(file.id)
        RAG-->>QA: false
    end
    QA-->>O: NeedIngestResponse(files_to_ingest)
    O-->>J: { outcome:"need_ingest", filesToIngest:[...] }

    Note over J: onNeedIngest
    loop per file
        J->>J: PDFBox: extract page text
        J->>O: POST /api/v1/rag/documents<br/>(long-running timeout)
        O->>RAG: chunk + stage documents
        O->>V: embed_documents (batches of 256)
        V-->>O: embeddings
        O->>RAG: add_documents
        O-->>J: { chunks_indexed: N }
    end

    Note over J: retry with resumeWith=pdf_question
    J->>O: POST /api/v1/orchestrator
    Note over O: fast-path to PdfQuestionAgent

    O->>QA: PdfQuestionRequest
    Note over QA: build RagCapability<br/>pinned to file IDs
    QA->>L: run(prompt) with search_knowledge tool

    loop up to max_searches
        L->>QA: search_knowledge(query)
        QA->>V: embed_query
        V-->>QA: query vector
        QA->>RAG: search(vector, collections=[file.id])
        RAG-->>QA: top-k chunks
        QA-->>L: formatted chunks
    end

    Note over QA: once budget spent,<br/>prepare() hides the tool
    L-->>QA: PdfQuestionAnswerResponse
    QA-->>O: answer
    O-->>J: { outcome:"answer", answer, evidence }
    J-->>FE: SSE "result"
    FE->>U: assistant bubble
```

Author: jbrunton96

app/common/src/main/java/stirling/software/common/model/ApplicationProperties.java

@@ -237,6 +237,13 @@ public static class AiEngine {
         private boolean enabled = false;
         private String url = "http://localhost:5001";
         private int timeoutSeconds = 120;
+
+        /**
+         * Longer timeout for heavy operations like RAG ingestion, which embeds the whole document
+         * and can take multiple minutes for large books. Applied per-call when the caller
+         * explicitly requests it via {@code AiEngineClient.postWithTimeout}.
+         */
+        private int longRunningTimeoutSeconds = 600;
     }
 
     @Data

app/proprietary/src/main/java/stirling/software/proprietary/model/api/ai/AiFile.java

@@ -0,0 +1,27 @@
+package stirling.software.proprietary.model.api.ai;
+
+import io.swagger.v3.oas.annotations.media.Schema;
+
+import lombok.AllArgsConstructor;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+
+/**
+ * A file supplied to the AI engine, identified by a stable opaque id plus a display name.
+ *
+ * <p>Values MUST match {@code AiFile} in {@code engine/src/stirling/contracts/common.py}.
+ */
+@Data
+@NoArgsConstructor
+@AllArgsConstructor
+@Schema(description = "File reference sent to the AI engine")
+public class AiFile {
+
+    @Schema(
+            description =
+                    "Opaque, stable identifier. Owned by Java; used as the RAG collection key.")
+    private String id;
+
+    @Schema(description = "Original filename, used by agents in user-facing prompts and responses.")
+    private String name;
+}

app/proprietary/src/main/java/stirling/software/proprietary/model/api/ai/AiRagIngestRequest.java

@@ -0,0 +1,24 @@
+package stirling.software.proprietary.model.api.ai;
+
+import java.util.List;
+
+import lombok.AllArgsConstructor;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+
+/**
+ * Body for {@code POST /api/v1/rag/documents} on the AI engine. Sent by Java when the engine
+ * reports {@code need_ingest} and the requested document's extracted content must be stored before
+ * the workflow can continue.
+ */
+@Data
+@NoArgsConstructor
+@AllArgsConstructor
+public class AiRagIngestRequest {
+
+    private String documentId;
+
+    private String source;
+
+    private List<AiRagPageText> pageText;
+}

app/proprietary/src/main/java/stirling/software/proprietary/model/api/ai/AiRagPageText.java

@@ -0,0 +1,16 @@
+package stirling.software.proprietary.model.api.ai;
+
+import lombok.AllArgsConstructor;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+
+/** A single page of extracted text for RAG ingest requests. */
+@Data
+@NoArgsConstructor
+@AllArgsConstructor
+public class AiRagPageText {
+
+    private int pageNumber;
+
+    private String text;
+}

app/proprietary/src/main/java/stirling/software/proprietary/model/api/ai/AiWorkflowEditPlan.java

@@ -11,8 +11,8 @@
 @Schema(description = "Per-file content extraction request from the AI engine")
 public class AiWorkflowFileRequest {
 
-    @Schema(description = "Original filename of the requested file", example = "contract.pdf")
-    private String fileName;
+    @Schema(description = "The file the engine wants content extracted for")
+    private AiFile file;
 
     @Schema(description = "Specific 1-based page numbers to extract from this file")
     private List<Integer> pageNumbers = new ArrayList<>();

app/proprietary/src/main/java/stirling/software/proprietary/model/api/ai/AiWorkflowFileRequest.java

@@ -12,6 +12,7 @@ public enum AiWorkflowOutcome {
     ANSWER("answer"),
     NOT_FOUND("not_found"),
     NEED_CONTENT("need_content"),
+    NEED_INGEST("need_ingest"),
     PLAN("plan"),
     NEED_CLARIFICATION("need_clarification"),
     CANNOT_DO("cannot_do"),

app/proprietary/src/main/java/stirling/software/proprietary/model/api/ai/AiWorkflowOutcome.java

@@ -73,6 +73,12 @@ public class AiWorkflowResponse {
     @Schema(description = "Per-file text extraction requests from the AI engine")
     private List<AiWorkflowFileRequest> files = new ArrayList<>();
 
+    @Schema(
+            description =
+                    "Files the AI engine requires to be ingested into RAG before it can continue"
+                            + " the workflow. Populated on need_ingest outcomes.")
+    private List<AiFile> filesToIngest = new ArrayList<>();
+
     @Schema(description = "Maximum number of pages the AI engine wants text extracted from")
     private Integer maxPages;
 
@@ -89,11 +95,4 @@ public class AiWorkflowResponse {
                             + " body or via the X-Stirling-Tool-Report header. May be null for tools"
                             + " that produce only a file.")
     private JsonNode report;
-
-    @Schema(
-            description =
-                    "Optional plan attached to an answer outcome. When non-null on outcome=ANSWER,"
-                            + " run the plan steps before delivering the answer; the resumed call"
-                            + " produces the real answer.")
-    private AiWorkflowEditPlan editPlan;
 }

app/proprietary/src/main/java/stirling/software/proprietary/model/api/ai/AiWorkflowResponse.java

@@ -43,20 +43,36 @@ public AiEngineClient(ApplicationProperties applicationProperties) {
 
     public String post(String path, String jsonBody) throws IOException {
         ApplicationProperties.AiEngine config = applicationProperties.getAiEngine();
+        return postWithTimeout(path, jsonBody, Duration.ofSeconds(config.getTimeoutSeconds()));
+    }
+
+    /**
+     * POST with an explicit per-call timeout, for heavy operations (e.g. RAG ingestion of a large
+     * document) that legitimately take longer than the default timeout.
+     */
+    public String postLongRunning(String path, String jsonBody) throws IOException {
+        ApplicationProperties.AiEngine config = applicationProperties.getAiEngine();
+        return postWithTimeout(
+                path, jsonBody, Duration.ofSeconds(config.getLongRunningTimeoutSeconds()));
+    }
+
+    private String postWithTimeout(String path, String jsonBody, Duration timeout)
+            throws IOException {
+        ApplicationProperties.AiEngine config = applicationProperties.getAiEngine();
         if (!config.isEnabled()) {
             throw new ResponseStatusException(
                     HttpStatus.SERVICE_UNAVAILABLE, "AI engine is not enabled");
         }
 
         String url = config.getUrl().stripTrailing() + path;
-        log.debug("Proxying AI engine request to {}", url);
+        log.debug("Proxying AI engine request to {} (timeout {}s)", url, timeout.toSeconds());
 
         HttpRequest request =
                 HttpRequest.newBuilder()
                         .uri(URI.create(url))
                         .header("Content-Type", "application/json")
                         .header("Accept", "application/json")
-                        .timeout(Duration.ofSeconds(config.getTimeoutSeconds()))
+                        .timeout(timeout)
                         .POST(HttpRequest.BodyPublishers.ofString(jsonBody))
                         .build();