Skip to content

Latest commit

 

History

History
267 lines (203 loc) · 8.59 KB

README.md

File metadata and controls

267 lines (203 loc) · 8.59 KB

Contributors Forks Stars Issues License

🔱 Neleus DB

Local-first Merkle-DAG database for AI agents, reproducible runs, and verifiable state

A high-performance, content-addressed database designed for AI agent workflows with cryptographic proofs and immutable versioning.

Examples · Report Bug · Request Feature

Contents

Why neleus-db

AI agent systems need more than a key-value store:

  • deterministic replay from immutable inputs
  • versioned state snapshots with commit history
  • integrity proofs for auditability
  • local operation without network dependencies

neleus-db is built around those guarantees.

Who should use this

  • teams building agent runtimes or orchestrators
  • local/private RAG systems that need provenance
  • products requiring tamper-evident run history
  • developers who want Git-like history for agent state

Features

  • Content-addressed blob and object storage (blake3)
  • Strict canonical object encoding (DAG-CBOR) with golden-byte tests
  • Versioned segmented state store with Merkle commitments
  • Membership and non-membership state proofs
  • Git-like commit graph (parents, author, message, state_root, manifests)
  • WAL + atomic file writes + automatic WAL recovery on open
  • Lock-file protection for multi-process mutable operations
  • Optional verify-on-read integrity checks
  • Encryption at rest (AES-256-GCM / ChaCha20-Poly1305 + PBKDF2-HMAC-SHA256)
  • Native semantic and vector search as rebuildable derived indexes

Architecture at a glance

Immutable canonical layer:
  blobs/   -> raw bytes
  objects/ -> canonical manifests, state objects, commits

Mutable pointers:
  refs/heads/*  -> commit hashes
  refs/states/* -> staged state roots

Derived/rebuildable layer:
  index/<commit_hash>/search_index.json

Reliability:
  wal/*.wal
  meta/config.json

Installation

git clone https://github.com/auralshin/neleus-db
cd neleus-db
cargo build --release

Examples

See the examples/ directory for practical code demonstrating common patterns:

Run any example:

cargo run --example 01_basic_blob_storage

Quick start

# 1) initialize DB
cargo run -- db init /tmp/neleus_db

# 2) add a blob
cargo run -- --db /tmp/neleus_db blob put /path/to/file.txt

# 3) create a document manifest + deterministic chunks
cargo run -- --db /tmp/neleus_db manifest put-doc \
  --source local_file \
  --file /path/to/file.txt \
  --chunk-size 512 \
  --overlap 64

# 4) update versioned state
cargo run -- --db /tmp/neleus_db state set main 6b6579 /path/to/value.bin --key-encoding hex

# 5) commit snapshot
cargo run -- --db /tmp/neleus_db commit new \
  --head main \
  --author agent1 \
  --message "initial snapshot" \
  --manifest <manifest_hash>

# 6) build and query derived search indexes
cargo run -- --db /tmp/neleus_db index build --head main
cargo run -- --db /tmp/neleus_db search semantic --head main --query "systems programming" --top-k 5
cargo run -- --db /tmp/neleus_db search vector --head main --embedding-file /path/to/query_embedding.json --top-k 5

# 7) generate and verify state proof
cargo run -- --db /tmp/neleus_db proof state main 6b6579 --key-encoding hex

CLI

  • db init <path>
  • blob put <file>
  • blob get <hash> <out_file>
  • manifest put-doc --source ... --file ... --chunk-size ... [--overlap ...]
  • manifest put-run --model ... --prompt-file ... --io-hashes in:<hash> --io-hashes out:<hash>
  • state set <head> <key> <value-file> [--key-encoding utf8|hex|base64]
  • state get <head> <key> [--key-encoding utf8|hex|base64] [--out-file <path>]
  • state del <head> <key> [--key-encoding utf8|hex|base64]
  • state compact <head>
  • commit new --head <name> --author <id> --message <text> [--manifest <hash> ...]
  • index build --head <name>
  • search semantic --head <name> (--query <text> | --query-file <path>) [--top-k <n>]
  • search vector --head <name> --embedding-file <path> [--top-k <n>]
  • log <head>
  • proof state <head> <key> [--key-encoding utf8|hex|base64]

Global flags:

  • --db <path> (default: ./neleus_db)
  • --json machine-readable output

Public Rust API

  • hash - Hash, domain-separated hashing helpers
  • blob_store - immutable content-addressed blobs
  • manifest - typed manifests + deterministic chunking
  • state - segmented persistent KV + proofs + compaction
  • commit - commit objects + signing/verifier hooks
  • refs - atomic refs and staged roots
  • db - open/init, schema migration, WAL recovery orchestration
  • index - derived semantic/vector index build and query

Data layout

<db_root>/
  blobs/aa/bb/<fullhash>
  objects/cc/dd/<fullhash>
  refs/heads/<name>
  refs/states/<name>
  index/<commit_hash>/search_index.json
  wal/*.wal
  meta/config.json

Integrity and determinism

Hash domains:

  • H_blob = blake3("blob:" || bytes)
  • H_manifest = blake3("manifest:" || dag_cbor_bytes)
  • H_state_node = blake3("state_node:" || dag_cbor_bytes)
  • H_commit = blake3("commit:" || dag_cbor_bytes)
  • H_state_leaf = blake3("state_leaf:" || leaf_encoding)
  • H_merkle_node = blake3("merkle_node:" || left || right)

Canonical encoding:

  • DAG-CBOR via serde_ipld_dagcbor
  • deterministic bytes locked with golden tests in src/canonical.rs

Reliability and security

  • atomic temp-write + rename for persistent files
  • structured WAL with replay/rollback during Database::open
  • lock files for ref/state mutation safety across processes
  • optional verify-on-read mode for blobs and typed objects
  • authenticated encryption support for persisted payloads

To enable encryption at rest, set meta/config.json with an enabled encryption block and provide NELEUS_DB_ENCRYPTION_PASSWORD when running the CLI.

Example config snippet:

{
  "schema_version": 3,
  "hashing": "blake3",
  "verify_on_read": true,
  "encryption": {
    "enabled": true,
    "algorithm": "aes-256-gcm",
    "kdf": "pbkdf2",
    "key_size": 32,
    "salt_size": 16,
    "nonce_size": 12,
    "kdf_iterations": 210000
  }
}

Testing

cargo test

The suite covers determinism, state semantics, proof verification/tampering, WAL recovery, integrity checks, compaction behavior, and CLI-facing flows.

Integration guide

See INTEGRATION.md for Rust embedding patterns and CLI-based integration examples for TypeScript/JavaScript, Go, and Python.

Design document

See DESIGN.md for details on Merkle model, proof format, and recovery behavior.

Contributing

  1. Open an issue describing the behavior or change.
  2. Add tests with the change.
  3. Keep object encoding/hash behavior backward compatible unless schema migration is included.

License

MIT