DataAgent

DataAgent is an open-source, web-based intelligent assistant for data exploration. It enables users to interact with their business data through natural language conversations powered by LLMs. With built-in tools for SQL execution, schema exploration, and file browsing, DataAgent bridges the gap between domain users and their databases.

Features

• Natural language data exploration -- Ask questions about your data and get answers with real-time SQL execution
• Three workflow modes -- Choose between simple chat, autonomous ReAct agent loops with tool-calling, or predefined YAML workflows
• Real-time streaming -- Responses appear character-by-character via Server-Sent Events (SSE)
• Rich markdown rendering -- Code blocks, tables, and structured responses rendered with GitHub Flavored Markdown
• Context injection -- Select files from object storage or reference database tables; schemas are automatically injected into the LLM prompt
• Full debugging transparency -- Inspect every LLM request/response, including prompts, parameters, token usage, and timing
• Business ontology system -- Model your business domain (activities, objects, rules, metrics) and let the LLM understand how concepts map to database tables
• Multi-tenant support -- Isolated datasources, ontologies, and users per tenant
• Skill system -- Keyword-triggered YAML skills provide domain-specific guidance to the LLM
• Vector embeddings -- LanceDB integration for semantic search over business knowledge
• Multi-language UI -- Frontend internationalization via i18next
• Conversation memory -- Long conversations are automatically summarized to stay within context limits

Architecture

┌─────────────────────────────────────────────────────────────┐
│  Browser                                                     │
│  ┌──────────┐  ┌──────────────────┐  ┌───────────────────┐  │
│  │ Sidebar  │  │    Chat Area     │  │  Resource Panel   │  │
│  │(history) │  │  (SSE streaming) │  │(Storage/Data/WF)  │  │
│  └──────────┘  └──────────────────┘  └───────────────────┘  │
│                       │                                       │
│  ┌────────────────────────────────────────────────────────┐  │
│  │              Debug Drawer (LLM trace)                  │  │
│  └────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
                          │ HTTP/SSE
┌─────────────────────────▼─────────────────────────────────┐
│  FastAPI Backend (port 8010)                               │
│  ┌────────────┐  ┌─────────────┐  ┌───────────────────┐   │
│  │   Router   │→ │   Service   │→ │       DB          │   │
│  │  (HTTP)    │  │ (business)  │  │  (SQLAlchemy)     │   │
│  └────────────┘  └─────────────┘  └───────────────────┘   │
│                         │                                   │
│  ┌──────────────────────┼──────────────────────────────┐   │
│  │  Workflow Dispatcher                                  │   │
│  │  ┌─────────┐  ┌────────────┐  ┌─────────────────┐   │   │
│  │  │  none   │  │  dynamic   │  │     yaml        │   │   │
│  │  │(stream) │  │(agent loop)│  │(step templates) │   │   │
│  │  └─────────┘  └────────────┘  └─────────────────┘   │   │
│  └──────────────────────────────────────────────────────┘   │
│                         │                                   │
│  ┌──────────────────────┼──────────────────────────────┐   │
│  │              Tool Registry                           │   │
│  │  execute_sql │ list_tables │ get_table_schema      │   │
│  │  read_file   │ list_files                            │   │
│  └──────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘
                          │
          ┌───────────────┼───────────────┐
          ▼               ▼               ▼
   ┌───────────┐  ┌────────────┐  ┌───────────────┐
   │ PostgreSQL│  │  OpenAI-   │  │   Object      │
   │  / SQLite │  │  compatible│  │   Storage     │
   │           │  │  LLM API   │  │   (TOS/S3)    │
   └───────────┘  └────────────┘  └───────────────┘

Tech Stack

Layer	Technology
Frontend	React 19, TypeScript, Vite, TailwindCSS, Zustand
Backend	Python 3.12, FastAPI, SQLAlchemy (async)
Database	PostgreSQL (production) / SQLite (development)
LLM	Any OpenAI-compatible API (DeepSeek, GPT-4o, MiniMax, etc.)
Vector DB	LanceDB
Object Storage	Volcengine TOS (S3-compatible)

Quick Start

Prerequisites

• Docker and Docker Compose
• An API key from an OpenAI-compatible LLM provider

Docker Compose (Recommended)

# 1. Configure environment
cp backend/.env.example backend/.env
# Edit backend/.env with your LLM_API_KEY and other settings

# 2. Start services
bash scripts/build.sh

# 3. Access the application
# Frontend: http://localhost:9280
# Backend API: http://localhost:8010

This starts PostgreSQL 16, the FastAPI backend, and the React frontend behind Nginx.

For a clean rebuild (stop and remove all containers, remove old images, rebuild and restart):

bash scripts/build.sh

Local Development

Backend:

cd backend
uv sync
cp .env.example .env   # Configure your LLM_API_KEY
uv run uvicorn app.main:app --reload --port 8010

Frontend:

cd frontend
npm install
npm run dev   # Proxies /api to localhost:8010

The dev server runs at http://localhost:3000 and proxies API requests to the backend via Vite.

Testing

cd backend
uv run pytest tests/ -v

Configuration

All settings are in backend/conf.yml with environment variable overrides available. Key settings:

Setting	Env Variable	Description
llm.api_base	LLM_API_BASE	OpenAI-compatible API endpoint
llm.api_key	LLM_API_KEY	API key for the LLM provider
llm.model	LLM_MODEL	Model name (e.g., gpt-4o-mini)
auth.jwt_secret	JWT_SECRET	Secret for signing JWT tokens
workflow.mode	WORKFLOW_MODE	none, dynamic, or yaml
workflow.max_iterations	WORKFLOW_MAX_ITERATIONS	Max tool-calling loop iterations

Workflow Modes

Mode	Description
none	Simple chat with streaming responses, no tool calls
dynamic	ReAct agent loop -- the LLM autonomously decides which tools to call, up to max_iterations
yaml	Predefined workflows with keyword matching and template-based step resolution

You can also override the mode per request via the workflow_mode field in the chat API.

Built-in Tools

Tool	Description
execute_sql	Execute read-only SELECT queries on configured datasources
list_tables	List all tables across configured datasources
get_table_schema	Get column names and types for a specific table
read_file	Read file content from object storage
list_files	List files and directories in object storage

Project Structure

MaoDataAgent/
├── backend/
│   ├── app/
│   │   ├── main.py              # FastAPI entry point, lifespan events
│   │   ├── config.py            # Configuration (YAML + env vars)
│   │   ├── auth.py              # JWT + bcrypt authentication
│   │   ├── models/schemas.py    # Pydantic request/response models
│   │   ├── db/                  # SQLAlchemy ORM models + CRUD operations
│   │   ├── router/              # HTTP route handlers (thin layer)
│   │   ├── service/             # Business logic
│   │   │   ├── chat.py          # Chat streaming + mode dispatcher
│   │   │   ├── agent.py         # Dynamic ReAct agent loop
│   │   │   ├── engine.py        # YAML workflow engine
│   │   │   ├── tools.py         # Tool registry + implementations
│   │   │   └── ...
│   │   ├── ontology/            # Business ontology management
│   │   └── embedding/           # LanceDB vector embeddings
│   ├── skills/                  # YAML skill definitions
│   ├── workflows/               # YAML workflow definitions
│   ├── conf.yml                 # Configuration file
│   ├── pyproject.toml
│   └── tests/                   # pytest test suite
├── frontend/
│   ├── src/
│   │   ├── App.tsx               # Root component (auth gating)
│   │   ├── store.ts              # Zustand state management
│   │   ├── api.ts                # API client + SSE streaming
│   │   └── components/           # React components
│   │       ├── Layout.tsx        # Three-column layout
│   │       ├── ChatArea.tsx      # Chat + SSE streaming
│   │       ├── SidebarHistory.tsx
│   │       ├── ResourcePanel.tsx
│   │       ├── DebugDrawer.tsx
│   │       ├── WorkflowPanel.tsx
│   │       └── ...
│   ├── package.json
│   └── vite.config.ts
├── docs/                         # Documentation
│   ├── DESIGN.md                 # Technical design document
│   ├── Business_schema.md        # Procurement domain schema
│   └── Ontology_schema.md        # Ontology system schema
├── docker-compose.yml
└── LICENSE

API Overview

All endpoints return JSON except POST /api/chat/stream which returns SSE.

Method	Endpoint	Description
POST	/api/auth/login	Login, returns JWT + user info
GET	/api/auth/me	Current user profile
GET/POST/DELETE	/api/conversations[/{id}]	Conversation CRUD
POST	/api/chat/stream	Streaming chat (SSE)
GET	/api/storage/list	List files from object storage
GET	/api/storage/read	Read file from object storage
GET	/api/datasource/tables	List tables from datasources
GET	/api/datasource/table/{table}/columns	Get table column definitions
GET	/api/debug/{message_id}	Get LLM request/response debug info
GET	/api/workflow/executions/{id}	Get workflow execution details

See docs/DESIGN.md for full API specifications.

Security

• JWT-based authentication on all endpoints except login
• Password hashing via bcrypt
• Resource ownership validation on all user-specific endpoints
• SQL injection prevention through table name regex validation and parameterized queries
• execute_sql tool restricted to SELECT statements (read-only)

License

This project is licensed under the Apache License 2.0. See LICENSE for details.

Documentation

• Technical Design -- Full architecture, API specs, component tree
• Business Schema -- Procurement/warehousing domain model
• Ontology Schema -- Business ontology and metadata tables