DataAgent

DataAgent is an open-source, web-based intelligent assistant for data exploration. It enables users to interact with their business data through natural language conversations powered by LLMs. With built-in tools for SQL execution, schema exploration, and file browsing, DataAgent bridges the gap between domain users and their databases.

Features

Natural language data exploration -- Ask questions about your data and get answers with real-time SQL execution
Three workflow modes -- Choose between simple chat, autonomous ReAct agent loops with tool-calling, or predefined YAML workflows
Real-time streaming -- Responses appear character-by-character via Server-Sent Events (SSE)
Rich markdown rendering -- Code blocks, tables, and structured responses rendered with GitHub Flavored Markdown
Context injection -- Select files from object storage or reference database tables; schemas are automatically injected into the LLM prompt
Full debugging transparency -- Inspect every LLM request/response, including prompts, parameters, token usage, and timing
Business ontology system -- Model your business domain (activities, objects, rules, metrics) and let the LLM understand how concepts map to database tables
Multi-tenant support -- Isolated datasources, ontologies, and users per tenant
Skill system -- Keyword-triggered YAML skills provide domain-specific guidance to the LLM
Vector embeddings -- LanceDB integration for semantic search over business knowledge
Multi-language UI -- Frontend internationalization via i18next
Conversation memory -- Long conversations are automatically summarized to stay within context limits

Architecture

┌─────────────────────────────────────────────────────────────┐
│  Browser                                                     │
│  ┌──────────┐  ┌──────────────────┐  ┌───────────────────┐  │
│  │ Sidebar  │  │    Chat Area     │  │  Resource Panel   │  │
│  │(history) │  │  (SSE streaming) │  │(Storage/Data/WF)  │  │
│  └──────────┘  └──────────────────┘  └───────────────────┘  │
│                       │                                       │
│  ┌────────────────────────────────────────────────────────┐  │
│  │              Debug Drawer (LLM trace)                  │  │
│  └────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
                          │ HTTP/SSE
┌─────────────────────────▼─────────────────────────────────┐
│  FastAPI Backend (port 8010)                               │
│  ┌────────────┐  ┌─────────────┐  ┌───────────────────┐   │
│  │   Router   │→ │   Service   │→ │       DB          │   │
│  │  (HTTP)    │  │ (business)  │  │  (SQLAlchemy)     │   │
│  └────────────┘  └─────────────┘  └───────────────────┘   │
│                         │                                   │
│  ┌──────────────────────┼──────────────────────────────┐   │
│  │  Workflow Dispatcher                                  │   │
│  │  ┌─────────┐  ┌────────────┐  ┌─────────────────┐   │   │
│  │  │  none   │  │  dynamic   │  │     yaml        │   │   │
│  │  │(stream) │  │(agent loop)│  │(step templates) │   │   │
│  │  └─────────┘  └────────────┘  └─────────────────┘   │   │
│  └──────────────────────────────────────────────────────┘   │
│                         │                                   │
│  ┌──────────────────────┼──────────────────────────────┐   │
│  │              Tool Registry                           │   │
│  │  execute_sql │ list_tables │ get_table_schema      │   │
│  │  read_file   │ list_files                            │   │
│  └──────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘
                          │
          ┌───────────────┼───────────────┐
          ▼               ▼               ▼
   ┌───────────┐  ┌────────────┐  ┌───────────────┐
   │ PostgreSQL│  │  OpenAI-   │  │   Object      │
   │  / SQLite │  │  compatible│  │   Storage     │
   │           │  │  LLM API   │  │   (TOS/S3)    │
   └───────────┘  └────────────┘  └───────────────┘

Tech Stack

Layer	Technology
Frontend	React 19, TypeScript, Vite, TailwindCSS, Zustand
Backend	Python 3.12, FastAPI, SQLAlchemy (async)
Database	PostgreSQL (production) / SQLite (development)
LLM	Any OpenAI-compatible API (DeepSeek, GPT-4o, MiniMax, etc.)
Vector DB	LanceDB
Object Storage	Volcengine TOS (S3-compatible)

Quick Start

Prerequisites

Docker and Docker Compose
An API key from an OpenAI-compatible LLM provider

Docker Compose (Recommended)

# 1. Configure environment
cp backend/.env.example backend/.env
# Edit backend/.env with your LLM_API_KEY and other settings

# 2. Start services
bash scripts/build.sh

# 3. Access the application
# Frontend: http://localhost:9280
# Backend API: http://localhost:8010

This starts PostgreSQL 16, the FastAPI backend, and the React frontend behind Nginx.

For a clean rebuild (stop and remove all containers, remove old images, rebuild and restart):

bash scripts/build.sh

Local Development

Backend:

cd backend
uv sync
cp .env.example .env   # Configure your LLM_API_KEY
uv run uvicorn app.main:app --reload --port 8010

Frontend:

cd frontend
npm install
npm run dev   # Proxies /api to localhost:8010

The dev server runs at http://localhost:3000 and proxies API requests to the backend via Vite.

Testing

cd backend
uv run pytest tests/ -v

Configuration

All settings are in backend/conf.yml with environment variable overrides available. Key settings:

Setting	Env Variable	Description
`llm.api_base`	`LLM_API_BASE`	OpenAI-compatible API endpoint
`llm.api_key`	`LLM_API_KEY`	API key for the LLM provider
`llm.model`	`LLM_MODEL`	Model name (e.g., `gpt-4o-mini`)
`auth.jwt_secret`	`JWT_SECRET`	Secret for signing JWT tokens
`workflow.mode`	`WORKFLOW_MODE`	`none`, `dynamic`, or `yaml`
`workflow.max_iterations`	`WORKFLOW_MAX_ITERATIONS`	Max tool-calling loop iterations

Workflow Modes

Mode	Description
`none`	Simple chat with streaming responses, no tool calls
`dynamic`	ReAct agent loop -- the LLM autonomously decides which tools to call, up to `max_iterations`
`yaml`	Predefined workflows with keyword matching and template-based step resolution

You can also override the mode per request via the workflow_mode field in the chat API.

Built-in Tools

Tool	Description
`execute_sql`	Execute read-only SELECT queries on configured datasources
`list_tables`	List all tables across configured datasources
`get_table_schema`	Get column names and types for a specific table
`read_file`	Read file content from object storage
`list_files`	List files and directories in object storage

Project Structure

MaoDataAgent/
├── backend/
│   ├── app/
│   │   ├── main.py              # FastAPI entry point, lifespan events
│   │   ├── config.py            # Configuration (YAML + env vars)
│   │   ├── auth.py              # JWT + bcrypt authentication
│   │   ├── models/schemas.py    # Pydantic request/response models
│   │   ├── db/                  # SQLAlchemy ORM models + CRUD operations
│   │   ├── router/              # HTTP route handlers (thin layer)
│   │   ├── service/             # Business logic
│   │   │   ├── chat.py          # Chat streaming + mode dispatcher
│   │   │   ├── agent.py         # Dynamic ReAct agent loop
│   │   │   ├── engine.py        # YAML workflow engine
│   │   │   ├── tools.py         # Tool registry + implementations
│   │   │   └── ...
│   │   ├── ontology/            # Business ontology management
│   │   └── embedding/           # LanceDB vector embeddings
│   ├── skills/                  # YAML skill definitions
│   ├── workflows/               # YAML workflow definitions
│   ├── conf.yml                 # Configuration file
│   ├── pyproject.toml
│   └── tests/                   # pytest test suite
├── frontend/
│   ├── src/
│   │   ├── App.tsx               # Root component (auth gating)
│   │   ├── store.ts              # Zustand state management
│   │   ├── api.ts                # API client + SSE streaming
│   │   └── components/           # React components
│   │       ├── Layout.tsx        # Three-column layout
│   │       ├── ChatArea.tsx      # Chat + SSE streaming
│   │       ├── SidebarHistory.tsx
│   │       ├── ResourcePanel.tsx
│   │       ├── DebugDrawer.tsx
│   │       ├── WorkflowPanel.tsx
│   │       └── ...
│   ├── package.json
│   └── vite.config.ts
├── docs/                         # Documentation
│   ├── DESIGN.md                 # Technical design document
│   ├── Business_schema.md        # Procurement domain schema
│   └── Ontology_schema.md        # Ontology system schema
├── docker-compose.yml
└── LICENSE

API Overview

All endpoints return JSON except POST /api/chat/stream which returns SSE.

Method	Endpoint	Description
POST	`/api/auth/login`	Login, returns JWT + user info
GET	`/api/auth/me`	Current user profile
GET/POST/DELETE	`/api/conversations[/{id}]`	Conversation CRUD
POST	`/api/chat/stream`	Streaming chat (SSE)
GET	`/api/storage/list`	List files from object storage
GET	`/api/storage/read`	Read file from object storage
GET	`/api/datasource/tables`	List tables from datasources
GET	`/api/datasource/table/{table}/columns`	Get table column definitions
GET	`/api/debug/{message_id}`	Get LLM request/response debug info
GET	`/api/workflow/executions/{id}`	Get workflow execution details

See docs/DESIGN.md for full API specifications.

Security

JWT-based authentication on all endpoints except login
Password hashing via bcrypt
Resource ownership validation on all user-specific endpoints
SQL injection prevention through table name regex validation and parameterized queries
execute_sql tool restricted to SELECT statements (read-only)

License

This project is licensed under the Apache License 2.0. See LICENSE for details.

Documentation

Technical Design -- Full architecture, API specs, component tree
Business Schema -- Procurement/warehousing domain model
Ontology Schema -- Business ontology and metadata tables

FilesExpand file tree

README.md

Latest commit

History