Skip to content

AGENTS.md

Latest commit

 

History

History
469 lines (340 loc) · 16 KB

AGENTS.md

File metadata and controls

469 lines (340 loc) · 16 KB
Raw

AGENTS.md — AI Agent Guide for ExtendDB

This document provides AI agents with essential context for working on the ExtendDB project.

Project Overview

ExtendDB is an open source DynamoDB-compatible runtime backed by PostgreSQL storage, released under the Apache 2.0 License. ExtendDB is a clean-room implementation of the DynamoDB wire protocol, written in Rust and initially developed by AWS engineers. It is not a fork of DynamoDB and contains no DynamoDB source code. ExtendDB speaks the DynamoDB wire protocol: any AWS SDK, CLI, or tool that works with DynamoDB works with ExtendDB, unchanged.

  • Language: Rust (edition 2024, MSRV 1.85+)
  • Storage backend: PostgreSQL 14+
  • Architecture: Async (tokio), trait-based storage abstraction
  • Authentication: Mandatory SigV4 with built-in IAM (users, groups, roles, policies)
  • TLS: Mandatory (self-signed cert generated by default)

Use Cases

  • Local development without cloud dependency
  • CI/CD integration tests
  • Self-hosted/on-premises deployments
  • Multi-cloud portability
  • Air-gapped environments

Project Structure

extenddb/
├── crates/
│   ├── core/             # Pure sync types, validation, expression parsing (no async)
│   ├── engine/           # DynamoDB operation handlers (PutItem, Query, etc.)
│   ├── storage/          # Storage trait definitions (TableEngine trait)
│   ├── storage-postgres/ # PostgreSQL implementation of TableEngine
│   ├── auth/             # SigV4 verification, IAM policy engine
│   ├── server/           # HTTP server, management API, web console
│   └── bin/              # CLI, config, daemon lifecycle (extenddb binary)
├── tests/                # Python integration tests (pytest)
├── docs/
│   ├── design/           # Architecture and component design docs
│   ├── manuals/          # User-facing guides (PDF pipeline)
│   └── adr/              # Architecture decision records
├── scripts/              # Platform installers (Linux, macOS)
├── devtools/             # Development utilities (test runners, license checks)
└── samples/              # Example Python applications

Key Architectural Concepts

Storage Abstraction

The TableEngine trait in crates/storage/src/lib.rs defines the storage interface. All storage backends implement this trait:

  • Current: storage-postgres (PostgreSQL)

The trait uses RPITIT (return-position impl Trait in traits) for async methods — no #[async_trait] macro.

All operations are account-scoped via account_id parameter for multi-tenancy isolation.

Crate Dependencies

extenddb (bin)
  └─> extenddb-server
       ├─> extenddb-engine
       │    ├─> extenddb-core (pure sync, no async)
       │    └─> extenddb-storage (trait definitions)
       ├─> extenddb-auth
       └─> extenddb-storage-postgres
  • extenddb-core: Pure synchronous Rust. No async, no I/O. Types, validation, expression parsing.
  • extenddb-storage: Trait definitions only. No implementation.
  • extenddb-storage-postgres: Concrete PostgreSQL implementation.
  • extenddb-engine: Operation handlers that call storage traits.
  • extenddb-server: HTTP server, management API, web console.
  • extenddb-auth: SigV4 signature verification, IAM policy evaluation.
  • extenddb (bin): CLI, config parsing, daemon lifecycle.

Building and Running

Prerequisites

  • Rust 1.85+ (rustup update)
  • PostgreSQL 14+ running locally (see docs/local-postgres-setup.md)
  • Python 3.10+ for tests (python3 -m venv ~/venvs/extenddb-venv && source ~/venvs/extenddb-venv/bin/activate && pip install -r requirements.txt)

Build

cargo build --release

Binary: target/release/extenddb

Unit Test

cargo test --workspace

Lint

cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings

Initialize (first time only)

./target/release/extenddb init --config extenddb.toml

This creates:

  • PostgreSQL databases (extenddb_catalog, extenddb_account_<id>)
  • Admin user credentials (printed to stdout — save the password!)
  • Self-signed TLS certificate at ~/.extenddb/tls/cert.pem
  • Config file extenddb.toml

Start Server

./target/release/extenddb serve --config extenddb.toml

The server runs on https://127.0.0.1:8000 by default (TLS mandatory).

Stop Server

./target/release/extenddb stop --config extenddb.toml

Check Status

./target/release/extenddb status --config extenddb.toml

Verify Deployment

./target/release/extenddb verify --config extenddb.toml

Destroy (tear down all databases)

Caution: this is a destructive action.

./target/release/extenddb destroy --config extenddb.toml

Testing

Python Integration Tests

Tests live in tests/ and use pytest. They run against either ExtendDB or Amazon DynamoDB.

Against ExtendDB (recommended workflow):

# 1. Build and initialize (first time)
cargo build --release
./target/release/extenddb init --config extenddb.toml
# Save the admin password!

# 2. Start server
./target/release/extenddb serve --config extenddb.toml

# 3. Provision test credentials and run tests
export EXTENDDB_TEST_ENDPOINT=https://127.0.0.1:8000
export EXTENDDB_ADMIN_USER=admin
export EXTENDDB_ADMIN_PASSWORD=<password-from-init>
eval $(python3 devtools/provision-test-credentials)
pytest tests/ -v

Against Amazon DynamoDB:

# Ensure AWS credentials are configured and EXTENDDB_TEST_ENDPOINT is unset
unset EXTENDDB_TEST_ENDPOINT
pytest tests/ -v

Test Runner Script

# Run all test suites against local ExtendDB
devtools/run-tests --extenddb --all

# Run specific suites
devtools/run-tests --extenddb --rust
devtools/run-tests --extenddb --pytest
devtools/run-tests --extenddb --external
devtools/run-tests --extenddb --comprehensive

# Run against Amazon DynamoDB
devtools/run-tests --real-dynamodb --pytest

# Use release build
devtools/run-tests --extenddb --rust --release

# Filter tests
devtools/run-tests --extenddb --pytest --filter test_put_item

The script requires:

  • EXTENDDB_TEST_ENDPOINT (e.g., https://127.0.0.1:8000)
  • EXTENDDB_ADMIN_PASSWORD (from extenddb init)
  • EXTENDDB_ADMIN_USER (optional, defaults to admin)

The script automatically provisions test credentials, configures runtime settings, and writes artifacts to discussions/test-*-<hash>.txt.

External Java Tests

Java tests live in tests/external/java/ and require Java 17+ and Maven 3.6+.

devtools/run-external-tests --endpoint https://127.0.0.1:8000

Coverage

devtools/run-coverage

License Checks

devtools/check-licenses

Configuration

extenddb init generates extenddb.toml. See extenddb.sample.toml for all options.

Environment variable overrides use EXTENDDB__ prefix:

export EXTENDDB__SERVER__PORT=9000
export EXTENDDB__AUTH__PROVIDER=builtin

Runtime settings (no restart required):

extenddb settings --config extenddb.toml set log_level debug
extenddb settings --config extenddb.toml list

TLS and Self-Signed Certificates

TLS is mandatory. extenddb init generates a self-signed cert at ~/.extenddb/tls/cert.pem.

To use with AWS CLI/SDKs:

export AWS_CA_BUNDLE=~/.extenddb/tls/cert.pem
aws dynamodb list-tables --endpoint-url https://127.0.0.1:8000 --region us-east-1

Python tests automatically use verify=False when endpoint is HTTPS.

Authentication

ExtendDB uses built-in IAM by default (auth.provider = "builtin"). All requests must be SigV4-signed with valid credentials.

Management API

Create users, groups, roles, policies, and access keys via CLI:

extenddb manage --user admin --password <pw> create-account --account-name myaccount
extenddb manage --user admin --password <pw> create-user --account-id <id> --user-name alice
extenddb manage --user admin --password <pw> create-access-key --account-id <id> --user-name alice

Or use the web console at https://127.0.0.1:8000/console/.

Monitoring

# Health check
curl --cacert ~/.extenddb/tls/cert.pem https://127.0.0.1:8000/health

# Prometheus metrics
curl --cacert ~/.extenddb/tls/cert.pem https://127.0.0.1:8000/metrics

# Syslog (Linux)
journalctl -t extenddb -f

# Syslog (macOS)
log stream --predicate 'processImagePath ENDSWITH "extenddb"' --level info

Supported DynamoDB Operations

Table Operations

CreateTable, DeleteTable, DescribeTable, ListTables, UpdateTable, DescribeEndpoints, DescribeLimits

Item Operations

PutItem, GetItem, DeleteItem, UpdateItem (SET, REMOVE, ADD, DELETE actions; condition expressions; all ReturnValues modes)

Query & Scan

Query, Scan (key conditions, filters, projections, pagination, index selection)

Batch & Transactions

BatchGetItem (100 keys), BatchWriteItem (25 ops), TransactGetItems (100 items), TransactWriteItems (100 ops)

Streams

ListStreams, DescribeStream, GetShardIterator, GetRecords

Other

UpdateTimeToLive, DescribeTimeToLive, TagResource, UntagResource, ListTagsOfResource, ImportTable, ExportTableToPointInTime

Common Development Tasks

Adding a New DynamoDB Operation

  1. Add types to crates/core/src/types.rs (input/output structs)
  2. Add handler to crates/engine/src/ (e.g., put_item.rs)
  3. Add storage trait method to crates/storage/src/lib.rs if needed
  4. Implement in crates/storage-postgres/src/
  5. Wire up HTTP route in crates/server/src/
  6. Add integration test in tests/

Modifying Expression Parsing

Expression parsing lives in crates/core/src/expression/. This is pure sync Rust with no async or I/O.

  • Key conditions: KeyCondition enum
  • Filter expressions: Expr enum
  • Update expressions: UpdateAction enum
  • Condition expressions: Expr enum

Debugging

  • Set RUST_LOG=debug for verbose logging
  • Use extenddb settings set log_level debug for runtime log level changes
  • Check syslog: journalctl -t extenddb -f (Linux) or log stream (macOS)
  • Health-check endpoint: curl --cacert ~/.extenddb/tls/cert.pem https://127.0.0.1:8000/health

Documentation

Document Path Description
Getting Started docs/getting-started.md Full setup walkthrough
Architecture Guide docs/manuals/01-architecture-guide.md System design and crate structure
Admin Guide docs/manuals/05-admin-guide.md Server lifecycle, configuration, IAM
Security Model docs/manuals/10-security-model.md Threat model, auth, authz
Deployment Guide docs/manuals/11-deployment-guide.md Self-hosted, multi-cloud, air-gapped
Differences from DynamoDB docs/differences-from-dynamodb.md Behavioral differences
Troubleshooting docs/troubleshooting.md Common errors and solutions
Storage Component Design docs/design/04-component-storage.md Storage trait design
Testing Design docs/design/09-testing.md Test strategy and infrastructure
Extending Storage 12-backend-plugin-architecture.md Guide to implementing storage backends

Build PDFs:

python3 docs/build-docs.py

Code Style and Conventions

  • Rust edition: 2024
  • MSRV: 1.85
  • Async: tokio, no #[async_trait] (use RPITIT)
  • Error handling: thiserror for library errors, anyhow for application errors
  • Serialization: serde + serde_json
  • Database: sqlx with compile-time query checking (PostgreSQL)
  • HTTP: axum + tower + hyper
  • Logging: tracing + tracing-subscriber
  • Metrics: metrics + metrics-exporter-prometheus
  • TLS: rustls + axum-server

Naming Conventions

  • Crates: extenddb-<component> (e.g., extenddb-core, extenddb-storage-postgres)
  • Modules: snake_case
  • Types: PascalCase
  • Functions: snake_case
  • Constants: SCREAMING_SNAKE_CASE

Testing Conventions

  • Integration tests in tests/ (Python pytest)
  • Unit tests in src/ modules (#[cfg(test)])
  • No target-specific branching in tests — if behavior differs, it's a bug
  • All tests clean up resources, even on failure
  • Auth tests use management_helpers.py for credential provisioning

Skills

This skill covers the full ExtendDB lifecycle: from a cold clone to a running server with working CRUD, sample app walkthroughs, and troubleshooting. It dispatches to domain-specific reference files based on user intent.

.agents/skills/extenddb/
├── SKILL.md              Dispatcher — routes by user intent
├── scripts/
│   └── detect-state.sh  Read-only environment state detection
└── references/
    ├── setup/            Build, init, serve, IAM user creation
    ├── postgres/         PostgreSQL readiness and installation
    ├── first-request/    AWS CLI/SDK configuration, first CRUD
    ├── samples/          sample_app.py and stream_consumer.py
    └── troubleshooting/  Symptom-to-fix lookup (16 indexed errors)

Activate when the user asks about installing, configuring, running, or debugging ExtendDB.

Important Files

File Purpose
Cargo.toml Workspace definition and shared dependencies
extenddb.toml Runtime configuration (generated by extenddb init)
extenddb.sample.toml Configuration template with all options
requirements.txt Python dependencies for tests and docs
pytest.ini Pytest configuration
external-suites.toml External test suite configuration
crates/storage/src/lib.rs Storage trait definitions
crates/core/src/types.rs DynamoDB type definitions
crates/engine/src/ Operation handlers
crates/bin/src/main.rs CLI entry point

Environment Variables

Variable Purpose
EXTENDDB_TEST_ENDPOINT Target endpoint for tests (e.g., https://127.0.0.1:8000)
EXTENDDB_ADMIN_USER Admin username for test credential provisioning (default: admin)
EXTENDDB_ADMIN_PASSWORD Admin password for test credential provisioning
EXTENDDB_PASSWORD Password for extenddb manage commands (avoids process listing exposure)
EXTENDDB_TEST_PG_CONNECTION_STRING PostgreSQL connection string for CLI lifecycle tests
AWS_CA_BUNDLE Path to CA cert for TLS (e.g., ~/.extenddb/tls/cert.pem)
RUST_LOG Rust logging level (e.g., debug, info)
EXTENDDB__<SECTION>__<KEY> Config overrides (e.g., EXTENDDB__SERVER__PORT=9000)

Common Pitfalls

  1. TLS is mandatory — server refuses to start without it. Use AWS_CA_BUNDLE for self-signed certs.
  2. Auth is mandatory — all requests must be SigV4-signed. Use extenddb manage or web console to create credentials.
  3. Account isolation — all operations are scoped to account_id. Different accounts can have tables with the same name.
  4. PostgreSQL must be running — extenddb init and extenddb serve require a running PostgreSQL instance.
  5. Python venv — activate the venv before running tests: source ~/venvs/extenddb-venv/bin/activate
  6. Test credentials — run devtools/provision-test-credentials before pytest to create test users and keys.

Getting Help

  • Troubleshooting: docs/troubleshooting.md
  • Design docs: docs/design/
  • Manuals: docs/manuals/
  • ADRs: docs/adr/

License

Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. See LICENSE for the full text.

This software is provided "as is" without warranty of any kind.

Last Updated: 2026-05-18