expand nostrdb-rs docs, add mdbook

README.md

@@ -1,9 +1,100 @@
-
 # nostrdb-rs
 
 [![ci](https://github.com/damus-io/nostrdb-rs/actions/workflows/rust.yml/badge.svg)](https://github.com/damus-io/nostrdb-rs/actions)
 [![docs](https://img.shields.io/docsrs/nostrdb)](https://docs.rs/nostrdb)
 
-[nostrdb][nostrdb] in Rust!
+Rust bindings for [nostrdb], the unfairly fast LMDB-backed nostr datastore.
+This crate exposes safe wrappers around the C engine—open a database, ingest
+events, run zero-copy filters, and subscribe to updates from async Rust.
 
 [nostrdb]: https://github.com/damus-io/nostrdb
+
+## Documentation
+
+- **Engine reference**: the upstream [nostrdb mdBook](https://github.com/damus-io/nostrdb/tree/master/docs/book)
+  covers architecture, metadata formats, and CLI workflows. Build it locally with
+  `mdbook serve docs/book --open`.
+- **Crate docs**: `cargo doc --open` or [docs.rs/nostrdb](https://docs.rs/nostrdb) for the
+  Rust-specific API surface (re-exported types, builders, async helpers).
+- **Rust guide**: see [`docs/rust.md`](docs/rust.md) for environment notes,
+  binding regeneration, and idiomatic usage patterns.
+
+## Requirements
+
+- Rust 1.75+ (edition 2021)
+- `clang`/`libclang` for `bindgen` (`LIBCLANG_PATH` may be required on macOS/Nix)
+- `cmake`, `pkg-config`, `make`, and a C11 compiler (used to build the vendored nostrdb)
+- Optional: `zstd`, `curl`, and the nostrdb fixtures if you run the examples end-to-end
+
+The `nostrdb` C sources live in the `nostrdb/` submodule; run
+`git submodule update --init --recursive` after cloning.
+
+## Quick start
+
+```bash
+git clone https://github.com/damus-io/nostrdb-rs.git
+cd nostrdb-rs
+git submodule update --init --recursive
+cargo test               # builds the C core and runs the Rust test suite
+
+# Try the examples (see docs/rust.md for details)
+cargo run --example ingest -- testdata/many-events.json
+cargo run --example query  -- --kind 1 --search nostrdb --limit 5
+```
+
+Examples reuse the same fixtures described in the nostrdb mdBook Getting Started
+chapter. To grab them quickly:
+
+```bash
+make -C nostrdb testdata/many-events.json
+```
+
+## Usage snapshot
+
+```rust
+use nostrdb::{Config, Filter, Ndb, NoteBuilder};
+
+let mut config = Config::default();
+config.skip_verification(true);
+config.writer_scratch_size(4 * 1024 * 1024);
+let ndb = Ndb::open("./data", &config)?;
+
+// ingest a JSON event (skipping signatures here)
+let raw = include_str!("../testdata/sample-event.json");
+ndb.process_event(raw)?;
+
+// build a filter (kind 1 + text search) and iterate results
+let filter = Filter::new()
+    .kinds([1])
+    .search("nostrdb")
+    .build()?;
+let mut txn = ndb.txn()?;
+for note in txn.query(&filter)? {
+    println!("{}: {}", note.id_hex(), note.content());
+}
+```
+
+See [`examples/`](examples) and the mdBook *CLI Guide* for richer workflows
+(thread queries, relay metadata, async subscriptions).
+
+Available examples:
+
+- `ingest` – load LDJSON fixtures while tagging the source relay.
+- `query` – mimic `ndb query` with `--kind`, `--search`, and `--limit` options.
+- `subscription` – spawn an async stream that prints new notes as they arrive.
+
+## Regenerating bindings
+
+When the `schemas/*.fbs` or C headers change upstream:
+
+```bash
+cargo clean
+cargo build --features bindgen   # or run build.rs manually with BINDGEN=1
+```
+
+This reruns `bindgen` against the vendored nostrdb headers. CI uses the
+pre-generated bindings by default to avoid requiring libclang everywhere.
+
+## License
+
+GPL-3.0-or-later (same as upstream nostrdb).

docs/rust.md

@@ -0,0 +1,61 @@
+# Rust Guide
+
+This document bridges the nostrdb mdBook (architecture, metadata, CLI workflows)
+and the Rust crate API. Use it as a checklist when wiring nostrdb-rs into an
+application.
+
+## Environment
+
+1. Install a recent Rust toolchain (1.75+ recommended).
+2. Install `clang`/`libclang`. `bindgen` will look for `libclang.so` / `libclang.dylib`;
+   set `LIBCLANG_PATH` if it lives outside your system default.
+3. Ensure a C build toolchain is available: `cmake`, `pkg-config`, `make`, and a
+   C11 compiler (Clang or GCC). The build script compiles the vendored `nostrdb`
+   sources located in `./nostrdb`.
+4. Clone submodules: `git submodule update --init --recursive`.
+5. Optional: install [`mdbook`](https://rust-lang.github.io/mdBook/) if you want to
+   read the upstream documentation locally: `mdbook serve docs/book --open` from the
+   nostrdb repo.
+
+## Building & testing
+
+```bash
+cargo build              # builds Rust + C artifacts
+cargo test               # runs the integration tests
+```
+
+To regenerate bindings after changing headers or schemas:
+
+```bash
+cargo clean
+cargo build --features bindgen
+```
+
+This sets the `BINDGEN` cfg path in `build.rs` so the C headers are parsed again.
+Otherwise the pre-generated Rust bindings in `src/bindings*.rs` are used.
+
+## Examples
+
+The `examples/` directory mirrors the workflows described in the nostrdb mdBook
+*Getting Started* and *CLI Guide* chapters:
+
+- `ingest.rs` – import LDJSON using `Ndb::process_events_with`.
+- `query.rs` – build nostr filters, execute queries, and print JSON.
+- `subscription.rs` – subscribe to filters asynchronously with `SubscriptionStream`.
+
+Run them with `cargo run --example <name> -- [args...]`. See the example files for
+CLI flags; they intentionally match the upstream `ndb` tool.
+
+## Mapping chapters to Rust types
+
+| mdBook chapter | Rust focus |
+| --- | --- |
+| Getting Started | `Config`, `Ndb`, `Ndb::process_event(s)` |
+| Architecture | `Ndb`, `Transaction`, `Note`, metadata structs |
+| API Tour | `FilterBuilder`, `NoteBuilder`, `Subscription`, `NoteMetadataBuilder` |
+| CLI Guide | Examples + `query.rs`, `subscription.rs` |
+| Metadata | `NoteMetadata`, `NoteMetadataBuilder`, `ReactionEntry` |
+| Language Bindings | Build scripts (`build.rs`, `src/bindings_*`) |
+
+Whenever the mdBook changes, update references in this guide plus the Rust inline
+docs so both stay synchronized.

examples/ingest.rs

@@ -0,0 +1,74 @@
+use nostrdb::{Config, IngestMetadata, Ndb};
+use std::env;
+use std::error::Error;
+use std::fs::File;
+use std::io::{BufRead, BufReader};
+use std::path::PathBuf;
+
+fn main() -> Result<(), Box<dyn Error>> {
+    let args = Args::parse();
+
+    let config = Config::new().skip_verification(true);
+    let ndb = Ndb::new(&args.db_dir, &config)?;
+
+    let file = File::open(&args.input)?;
+    let reader = BufReader::new(file);
+    let mut processed = 0usize;
+
+    for line in reader.lines() {
+        let line = line?;
+        if line.trim().is_empty() {
+            continue;
+        }
+        let meta = IngestMetadata::new().relay(&args.relay);
+        ndb.process_event_with(&line, meta)?;
+        processed += 1;
+    }
+
+    println!(
+        "Imported {processed} events from {} into {}",
+        args.input.display(),
+        args.db_dir
+    );
+
+    Ok(())
+}
+
+struct Args {
+    input: PathBuf,
+    db_dir: String,
+    relay: String,
+}
+
+impl Args {
+    fn parse() -> Self {
+        let mut input = None;
+        let mut db_dir = String::from("./data");
+        let mut relay = String::from("fixture");
+
+        let mut iter = env::args().skip(1);
+        while let Some(arg) = iter.next() {
+            match arg.as_str() {
+                "--db" => db_dir = iter.next().expect("missing value for --db"),
+                "--relay" => relay = iter.next().expect("missing value for --relay"),
+                "-h" | "--help" => Args::print_help(),
+                other if input.is_none() => input = Some(PathBuf::from(other)),
+                _ => Args::print_help(),
+            }
+        }
+
+        let input = input.unwrap_or_else(|| Args::print_help());
+        Args {
+            input,
+            db_dir,
+            relay,
+        }
+    }
+
+    fn print_help() -> ! {
+        eprintln!(
+            "usage: cargo run --example ingest -- <path-to-ldjson> [--db ./data] [--relay url]"
+        );
+        std::process::exit(1)
+    }
+}

examples/query.rs

@@ -0,0 +1,81 @@
+use nostrdb::{Config, Filter, Ndb, Transaction};
+use std::env;
+use std::error::Error;
+
+fn main() -> Result<(), Box<dyn Error>> {
+    let args = Args::parse();
+
+    let config = Config::new().skip_verification(true);
+    let ndb = Ndb::new(&args.db_dir, &config)?;
+
+    let mut builder = Filter::new();
+    if let Some(kind) = args.kind {
+        builder = builder.kinds([kind]);
+    }
+    if let Some(search) = &args.search {
+        builder = builder.search(search);
+    }
+    builder = builder.limit(args.limit as u64);
+    let filter = builder.build();
+
+    let txn = Transaction::new(&ndb)?;
+    let results = ndb.query(&txn, &[filter], args.limit)?;
+
+    if results.is_empty() {
+        println!("no matches");
+        return Ok(());
+    }
+
+    for row in results {
+        let json = row.note.json()?;
+        println!("{}\n", json);
+    }
+
+    Ok(())
+}
+
+struct Args {
+    db_dir: String,
+    kind: Option<u32>,
+    search: Option<String>,
+    limit: i32,
+}
+
+impl Args {
+    fn parse() -> Self {
+        let mut db_dir = String::from("./data");
+        let mut kind = None;
+        let mut search = None;
+        let mut limit = 10;
+
+        let mut iter = env::args().skip(1);
+        while let Some(arg) = iter.next() {
+            match arg.as_str() {
+                "--db" => db_dir = iter.next().expect("missing value for --db"),
+                "--kind" => {
+                    let v = iter.next().expect("missing value for --kind");
+                    kind = Some(v.parse().expect("invalid kind"));
+                }
+                "--search" => search = Some(iter.next().expect("missing term")),
+                "--limit" => {
+                    let v = iter.next().expect("missing value for --limit");
+                    limit = v.parse().expect("invalid limit");
+                }
+                "-h" | "--help" => Args::print_help(),
+                _ => Args::print_help(),
+            }
+        }
+
+        Args {
+            db_dir,
+            kind,
+            search,
+            limit,
+        }
+    }
+
+    fn print_help() -> ! {
+        eprintln!("usage: cargo run --example query -- [--db ./data] [--kind 1] [--search term] [--limit 20]");
+        std::process::exit(1)
+    }
+}