Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
222 lines (165 loc) · 12.3 KB

ARCHITECTURE.md

File metadata and controls

222 lines (165 loc) · 12.3 KB
Raw

Talk2data orchestrator

This document describes the architecture and workflow of the Talk2data orchestrator.

Introduction

ReAct: The core approach follows ReAct: always require reasoning traces before and after actions (tool calls, sub tasks definition..), and keep them in context to guide future steps and limit hallucinations.

Patterns: The architecture combines the following patterns :

  • Orchestrator-worker to divide sub-tasks
  • Evaluator-optimizer to iteratively refine results

past_exchanges: At the start of the workflow, if this is not the beginning of the conversation with the user, provide the state with information on past exchanges (as a structured object) : message contents and what IDs were succesfully used for past queries.

Structured outputs: NEVER manually parse a raw text LLM response to extract structured data (eg booleans or IDs). If you need to do that, it means that the LLM calls should use structured outputs.

Architecture

The orchestrator's workflow is divided into 4 main components:

  • A. Planner
  • B. Sub-tasks
  • C. Aggregator
  • D. Final result

Part A: Planner (Query understanding, reformulation and decomposition)

  1. Intent extraction and reformulation: given past_exchanges and the user query, identify its main goal: give the LLM some context about DHIS2, its data structure (role, organisation units, data elements, indicators, periods, the data), the ICRC, and prompt it to rephrase the question, making it more clear to prepare for and ease disaggregation.

  2. Ambiguity detection: ensure that the intent matches the messages history and contains all needed info to proceed to decomposition, otherwise go to conversational refinement.

  3. Conversational refinement: Based on feedback from any of the following stages

  • Ambiguity detection
  • Subquery results aggregation
  • Parameter validation Evaluate whether the user query or found parameters are ambiguous (missing organisation unit/data element/indicator/period, invalid parameter combo, non-existent dimension search term). If they are, go to a conversational refinement node to ask clarifying questions before proceeding (the question should focus on what is missing/not working). Then go back to intent extraction.
  1. Decomposition: based on the intent and past_exchanges, break down the main query into smaller sub-queries that can be addressed individually. The sub-queries can be of the following types:
  • Find data elements tasks (find_dx)
  • Find organization units tasks (find_ou)
  • Find time periods tasks (find_pe)
  • (available but not always used) Find disaggregations tasks (per-dimension sub-queries) If past_exchanges contains info (e.g. IDs) that could speed up the work and we are sure that it is still the right fit for the current query, reuse it to avoid redundant subqueries. There can be several sub-queries of each type, for example if the user asks for a comparison: one sub-query for each component of the comparison (e.g., "Compare the number of malaria cases and tuberculosis cases in 2023 in Kenya"). On the first pass, do not do disaggregations. We first need to find the correct ou, dx and pe. Example prompt that we could use/adapt, from [0]:
 System: You are tasked with assisting users in generating structured plans for answering questions. Your
 goal is to deconstruct a query into manageable, simpler components. For each question, perform these
 tasks:
 *Analysis: Identify the core components of the question, emphasizing the key elements and context
 needed for a comprehensive understanding. Determine whether the question is straightforward or requires
 multiple steps to provide an accurate answer.
 *Plan Creation:- Break down the question into smaller, simpler questions by reasoning that lead to the final answer.
 Ensure those steps are non overlap.- Ensure each step is clear and logically sequenced.- Each step is a question to search, or to aggregate output from previous steps. Do not verify previous
 step.
 Notes:- Put your output in a list of string, each string describe a sub-task
 User: {Question}

Part B: Sub-tasks execution (Parameter discovery sub-queries)

This is where our MCP comes in : Sub-queries use LLMs provided with tools to discover the required parameters.

Using a principle derived from the Dynamic Capability Boundary, we filter tools based on the sub-query type, so that the subtask executor LLM only sees relevant ones for a given sub-query.

Using the MCP, the subtask executor finds the best matches for each sub-query: let it do multiple iterations (have a max_iter), using chain-of-thought prompting, to use the tools and their results to refine its search. It can stop early if the result is found, and if the iteration limit is reached, ask for an explanation on why it did not find results (it can also not know why, and should say it if so). The sub-queries are implemented via subgraph isolation, allowing each sub-query to be conducted independently. The aggregator receives the results of the current step's subqueries AND the past_exchanges elements that have been deemed reusable, aggregates them and sends them to part C (validation) or to disambiguation.

Disambiguation: Use an LLM to decide whether the right results have been identified for each sub-query or not (e.g. no results or many results when the sub-query expects few). They are then sent to the aggregator, and if disambiguation is needed, go to the conversational refinement node to ask the user to disambiguate the results before restarting the workflow.

Per-sub-query routing enables independent refinement loops: some sub-queries may loop for clarification while others proceed, without blocking each other. This is achieved through subgraph architecture where each sub-query executes in isolation with its own state machine.

Part C: Aggregator (Results aggregation and validation)

  1. Validation: run sample queries to check if the parameter combination return data. Evaluate query results to choose whether to go back to the conversational refinement node (with a feedback: add/modify/remove sub queries, to better direct decomposition) or to proceed to the disaggregation detection node. If the sample query succeeds, filter parameters to keep only those that returned data.

  2. Disaggregation detection: once a valid set of parameters ou/dx/pe has been found, proceed to disaggregation: check whether the intent of the user needs a disaggregation. If so, go back to the decomposition node, intructing it to generate disaggregation subqueries. If the disaggregation subqueries were already performed, or if there is no disaggregation needed, go to the final query node.

  3. Final query: Once all parameters are found, run the final query to get the DHIS2 data. If there are disaggregations and the validation fails, return the data without disaggregations (and tell the user that we could not disaggregate). Then, go to part D.

Part D: Final result

Using the data that results from the previous steps (passed as JSON, HTML or Markdown?), generate a final response to the user query using chain of thought prompting.

Use Self-Refine [1] on the response to improve the results.

References

Workflow graph

This part presents the complete workflow diagram, showing the four main parts of query processing and their relationships.

Visual Workflow (Mermaid)

graph TB

subgraph UserArea["USER"]
   User["USER"]
   UA_DOT["."]
end
style UserArea fill:#ffffff,stroke:#1e1e1e,stroke-width:2px
style User fill:#a5d8ff,stroke:#1e1e1e,stroke-width:3px,color:#000000
style UA_DOT fill:#ffffff,stroke:#ffffff,stroke-width:0px,color:#ffffff
classDef invisible fill:#ffffff,stroke-width:0px;
State["Add existing state:<br/>pertinent IDs found from previous<br/>queries, previous results..."]

subgraph PartA["PART A: Query Understanding & Decomposition"]
   A1_IntentExtraction[Intent Extraction &<br/>Reformulation]
   A2_MissingInfo{Missing Info?}
   A3_ConversationalRefinement[Conversational<br/>Refinement]
   A4_Decomposition[Decomposition]

   A1_IntentExtraction --> A2_MissingInfo
   A2_MissingInfo -->|Yes| A3_ConversationalRefinement
   A3_ConversationalRefinement -->|Clarified| A1_IntentExtraction
   A2_MissingInfo -->|No| A4_Decomposition
end

subgraph PartB["PART B: Parameter Discovery"]
   B1_ExecuteSubquery[Execute Subquery<br/>FindOU/FindPE/FindDX/<br/>Disaggregation]
   B2_EvaluationOutput[Subquery Evaluation<br/>and Output Generation]
   B1_Dispatch[Dispatch Sub-queries]
   B2_Collect[Aggregate Results]
   
   B1_Dispatch --> B1_ExecuteSubquery
   B1_ExecuteSubquery --> B2_EvaluationOutput
   B2_EvaluationOutput --> B2_Collect
   B2_Collect -->|Incomplete| A3_ConversationalRefinement
end

subgraph PartC["PART C: Results Aggregation & Validation"]
   C1_Validate[Validate Parameters<br/>with Sample Queries]
   C2_Disaggregation{Need<br/>Disaggregations?}
   C3_FinalQuery[Final Query<br/>Fetch Data from DHIS2]

   C1_Validate -->|Valid| C2_Disaggregation
   C2_Disaggregation -->|No| C3_FinalQuery
   C2_Disaggregation -->|Yes| A4_Decomposition
end

subgraph PartD["PART D: Analysis"]
  D1_AnswerQuery[Answer User Query]
  D2_EvaluateAnswer[Evaluate Answer]
  D3_RefineAnswer[Refine Answer]
     
  D1_AnswerQuery --> D2_EvaluateAnswer
  D2_EvaluateAnswer --> D3_RefineAnswer
  D3_RefineAnswer --> D2_EvaluateAnswer
  D3_RefineAnswer --> User
end

User --> A1_IntentExtraction
State --> A1_IntentExtraction
A4_Decomposition --> B1_Dispatch
B2_Collect -->|Valid| C1_Validate
C1_Validate -->|Invalid| A3_ConversationalRefinement
C3_FinalQuery --> D1_AnswerQuery

%% Part containers - white background, colored borders
style PartA fill:#ffffff,stroke:#0066cc,stroke-width:3px
style PartB fill:#ffffff,stroke:#cc8800,stroke-width:3px
style PartC fill:#ffffff,stroke:#00aa00,stroke-width:3px
style PartD fill:#ffffff,stroke:#cc0066,stroke-width:3px

%% Part A - Understanding & Decomposition (Blue theme)
%% Processing nodes
style A1_IntentExtraction fill:#ffec99,stroke:#0066cc,stroke-width:2px,color:#000000
style A4_Decomposition fill:#ffec99,stroke:#0066cc,stroke-width:2px,color:#000000
%% Decision nodes
style A2_MissingInfo fill:#ffc9c9,stroke:#0066cc,stroke-width:2px,color:#000000
%% User interaction nodes
style A3_ConversationalRefinement fill:#a5d8ff,stroke:#0066cc,stroke-width:2px,color:#000000

%% Part B - Parameter Discovery (Orange theme)
%% Processing nodes
style B1_Dispatch fill:#b2f2bb,stroke:#cc8800,stroke-width:2px,color:#000000
style B1_ExecuteSubquery fill:#b2f2bb,stroke:#cc8800,stroke-width:2px,color:#000000
style B2_EvaluationOutput fill:#b2f2bb,stroke:#cc8800,stroke-width:2px,color:#000000
style B2_Collect fill:#b2f2bb,stroke:#cc8800,stroke-width:2px,color:#000000

%% Part C - Aggregation & Validation (Green theme)
%% Processing nodes
style C1_Validate fill:#ffec99,stroke:#00aa00,stroke-width:2px,color:#000000
style C3_FinalQuery fill:#ffec99,stroke:#00aa00,stroke-width:2px,color:#000000
%% Decision nodes
style C2_Disaggregation fill:#ffc9c9,stroke:#00aa00,stroke-width:2px,color:#000000

%% Part D - Analysis (Pink theme)
%% Processing nodes
style D1_AnswerQuery fill:#ffec99,stroke:#cc0066,stroke-width:2px,color:#000000
style D2_EvaluateAnswer fill:#ffec99,stroke:#cc0066,stroke-width:2px,color:#000000
style D3_RefineAnswer fill:#ffec99,stroke:#cc0066,stroke-width:2px,color:#000000

%% State box - orange with dashed border
style State fill:#ffa94d,stroke:#1e1e1e,stroke-width:2px,stroke-dasharray: 5 5,color:#000000
Loading

🎨 Color & Shape Legend

By Part (stroke color):

  • 🔵 Part A (Blue): Query Understanding & Decomposition
  • 🟠 Part B (Orange): Parameter Discovery
  • 🟢 Part C (Green): Results Aggregation & Validation
  • 🔴 Part D (Red): Analysis & Refinement

By Node Type (fill color):

  • 🟡 Yellow (#ffec99): Processing/Action nodes
  • 🟢 Green (#b2f2bb): Parameter discovery processing nodes
  • 🔴 Red (#ffc9c9): Decision points
  • 🔵 Light Blue (#a5d8ff): User interaction nodes
  • 🟠 Orange (#ffa94d): State management

Note: For best results, use the "Markdown Preview Mermaid Support" extension in VS Code to view the diagram interactively.