NeuroDB is an experimental, high-performance SQL database engine that combines LSM-Tree storage with AI-learned indexes (RMI). It offers low read/write amplification, compact storage, and fast range scans. The codebase is ready to fork and adapt: use it for research, benchmarking, or as a base to train and extend your own project—take the code, modify it, and build on it.
- Learned SSTable: Recursive Model Index (RMI) at the SSTable layer instead of traditional block indexes.
- Python RMI trainer: Async ML training during L0→L1 compaction; Python exports
.limodels, Go runs inference. - Bounded fallback search: Binary search limited to
[pos - min_error, pos + max_error]per leaf (no global scan). - Cost-based optimizer (CBO): Uses RMI CDF for O(1) row count estimation.
- ACID + MVCC: Snapshot isolation, read-your-own-writes, durable WAL with undo/truncate recovery.
- LSM storage: Leveled compaction (L0→L1→L2→L3), tombstone GC, group commit, optional Z-Order spatial indexing.
- Go 1.24+
- Python 3.x (for RMI training; optional if you only use pre-trained
.lifiles) - Python deps:
numpy,scikit-learn(seepython/requirements.txt)
git clone https://github.com/MantraChen/neurodb.git
cd neurodb
go run cmd/server/main.go -config configs/neuro.yamlDefault config path is configs/neuro.yaml. Copy from configs/config.example.yaml if needed.
NeuroDB provides a Go client SDK with transaction support:
// Begin an isolated transaction
tx, _ := store.BeginTx()
// Writes are isolated in a WriteBatch
tx.Put([]byte("user_100"), []byte("Howie"))
// Read-your-own-writes within the transaction
val, _ := tx.Get([]byte("user_100"))
// Commit to make changes visible
tx.Commit()The project is designed so you can train and plug in your own models. The Python sidecar trains a two-layer RMI and exports a .li (JSON) file; the Go engine loads it for inference.
- Input: CSV of sorted keys (one key per line), e.g. exported by Go during compaction or prepared by you.
- Output:
.lifile with root/leaf weights and per-leaf error bounds.
From the project root:
pip install -r python/requirements.txt
python3 python/train_rmi.py --input keys.csv --output shard-0.li.new --fanout 256See ai_trainer/README.md and pkg/core/hybrid_store.go (triggerPythonTraining) for integration. You can change fanout, model type, or replace the trainer with your own script—the Go side only needs the .li format.
Example config (copy configs/config.example.yaml to configs/neuro.yaml):
| Section | Key | Description |
|---|---|---|
server |
addr, tcp_addr |
HTTP dashboard/REST and TCP binary |
storage |
path, wal_*, memtable_flush_threshold, compaction_threshold |
Data dir and LSM tuning |
system |
shard_count, bloom_* |
Shards and Bloom filter settings |
- MySQL wire protocol: Use standard MySQL clients and ORMs.
- Custom TCP binary: Low-latency protocol for the Go SDK.
- REST API: Stateless HTTP with session-based transactions (
BEGIN/COMMIT/ROLLBACKviasession_id).
A terminal-style web UI is available at http://localhost:8080 (configurable):
- Metrics: Global SeqNum, active transactions, GC watermark.
- SQL console: Multiline SQL with transaction boundaries.
- AI diagnostics: Learned index error heatmap for RMI precision.
cmd/server/main.go— Server entrypoint.pkg/core/hybrid_store.go— LSM + RMI integration; triggers Python training and loads.li.pkg/index/learned/— RMI load and inference (PredictWithBounds).python/train_rmi.py— Default RMI trainer; replace or adapt for your own models.ai_trainer/— Wrapper and docs for the Python trainer.
NeuroDB is under the MIT License. Copyright (c) 2026 HowieSun.
For academic use, you may cite:
NeuroDB: An Adaptive, Learned-Index Powered Relational Database Engine.