Address review feedback: WAL tests, docs, and new-code cleanups

- tests/wal_provider_test.rs: add smoke tests for the redb WAL provider
  (creation + builder wiring/lifecycle), mirroring state_store_test.rs.
- instance_paths.rs: add a doc comment explaining the hex-encoding rationale
  and two edge-case tests (empty ID, happy-path ASCII).
- server.rs / instance_handlers.rs: hoist the duplicated instance_storage_key
  computation so the index and WAL paths share one safe_id.
- builder.rs: document that with_index_provider registers under the name
  "rocksdb".
- README.md / CLAUDE.md: document the always-on, redb-backed WAL, its
  ./data/<instance-key>/wal location, and the Docker volume implication.

The WAL remains always-on by design (durability of source events as a
baseline guarantee), not gated behind a config field.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: danielgerlag

CLAUDE.md

@@ -209,6 +209,23 @@ The REST API is exposed under `/api/v1/instances/{instanceId}/...` for multi-ins
 
 **Important**: Sources and reactions are plugins that must be provided programmatically or via the configuration file's tagged enum format. Queries can also be defined via configuration files.
 
+### Write-Ahead Log (WAL)
+
+Every DrasiLib instance is unconditionally wired with a durable, redb-backed
+write-ahead log for source events (`drasi-wal-redb::RedbWalProvider`, attached
+via `builder.with_wal_provider(...)`). The WAL is created under
+`./data/<instance-key>/wal/`, where `<instance-key>` is `instance_storage_key()`
+applied to the instance ID (a hex-encoded, path-traversal-safe form). The
+per-source redb files inside that directory are created lazily on first append.
+
+Unlike `persistIndex` (RocksDB query indexes) and `stateStore` (plugin state),
+the WAL is **always on** and there is currently no config field to disable it.
+This is an intentional design choice: durability of source events is treated as
+a baseline guarantee rather than an opt-in. The wiring lives in `server.rs`
+(config/startup path) and `instance_handlers.rs` (REST API create path).
+Operators should account for this directory in disk-usage planning and mount a
+volume for `./data/` in containerized deployments.
+
 ### Configuration Persistence
 
 Persistence uses a snapshot-based approach: when saving, `ConfigPersistence::save()` calls `snapshot_configuration()` on each DrasiLib instance via the ComponentGraph. The ComponentGraph is the single source of truth — there are no shadow caches or separate registration steps. Mutations flow through the ComponentGraph, and the persisted YAML is reconstructed from the current graph state at save time.

README.md

@@ -822,6 +822,16 @@ queries: []
 reactions: []
 ```
 
+> **Write-ahead log (always on):** Every instance maintains a durable
+> write-ahead log (WAL) for source events, backed by redb. It is written to
+> `./data/<instance-key>/wal/` (where `<instance-key>` is a hex-encoded form of
+> the instance ID), relative to the server's working directory. The WAL is
+> always enabled and there is currently no configuration option to disable it,
+> so account for this directory when planning disk usage and, in containerized
+> deployments, mount a volume for `./data/` to persist it across restarts. This
+> is separate from the optional `persistIndex` (query indexes) and `stateStore`
+> (plugin state) storage.
+
 ### Plugins Configuration
 
 The `plugins` section declares plugin dependencies that can be installed with `drasi-server plugin install --from-config`. Each entry specifies a plugin reference that supports three URI formats:

src/api/shared/handlers/instance_handlers.rs

@@ -89,9 +89,11 @@ pub async fn create_instance(
         builder = builder.with_dispatch_buffer_capacity(capacity);
     }
 
+    // Filesystem-safe key shared by the persistent index and WAL paths.
+    let safe_id = instance_storage_key(&instance_id);
+
     // Set up RocksDB persistent indexing if requested
     if persist_index {
-        let safe_id = instance_storage_key(&instance_id);
         let index_path = PathBuf::from(format!("./data/{safe_id}/index"));
         log::info!(
             "Enabling persistent indexing for instance '{}' with RocksDB at: {}",
@@ -107,7 +109,6 @@ pub async fn create_instance(
 
     // WAL provider for durable source event persistence
     {
-        let safe_id = instance_storage_key(&instance_id);
         let wal_path = PathBuf::from(format!("./data/{safe_id}/wal"));
         log::info!(
             "Enabling WAL provider for instance '{}' at: {}",

src/builder.rs

@@ -78,7 +78,8 @@ impl DrasiServerBuilder {
     /// Add an index provider for persistent storage
     ///
     /// By default, DrasiLib uses in-memory indexes. Use this method to inject
-    /// a persistent index provider like RocksDB.
+    /// a persistent index provider like RocksDB. The provider is registered as
+    /// the default index backend under the name `"rocksdb"`.
     pub fn with_index_provider(mut self, provider: Arc<dyn IndexBackendPlugin>) -> Self {
         let builder = self.primary_builder_mut();
         *builder = std::mem::take(builder).with_default_index_provider("rocksdb", provider);

src/instance_paths.rs

@@ -12,6 +12,15 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+/// Converts an arbitrary instance ID into a filesystem-safe storage key.
+///
+/// Each byte of the ID is hex-encoded as two lowercase digits and prefixed with
+/// `id-`. Hex encoding is used (rather than naive character substitution) because
+/// it is injective: it prevents path traversal (e.g. `../tenant` →
+/// `id-2e2e2f74656e616e74`), eliminates separator collisions (e.g. `a/b` and
+/// `a_b` map to distinct keys), and always yields a valid single-segment
+/// directory name on every platform. Used to derive the per-instance index and
+/// WAL directories under `./data/`.
 pub(crate) fn instance_storage_key(instance_id: &str) -> String {
     let mut key = String::with_capacity(3 + instance_id.len() * 2);
     key.push_str("id-");
@@ -36,6 +45,16 @@ mod tests {
         assert_ne!(instance_storage_key("a\\b"), instance_storage_key("a_b"));
     }
 
+    #[test]
+    fn instance_storage_key_encodes_empty_id() {
+        assert_eq!(instance_storage_key(""), "id-");
+    }
+
+    #[test]
+    fn instance_storage_key_encodes_ascii_id() {
+        assert_eq!(instance_storage_key("default"), "id-64656661756c74");
+    }
+
     #[test]
     fn instance_storage_key_encodes_path_traversal_characters() {
         let key = instance_storage_key("../tenant");

src/server.rs

@@ -454,9 +454,11 @@ impl DrasiServer {
                 builder = builder.with_dispatch_buffer_capacity(capacity);
             }
 
+            // Filesystem-safe key shared by the persistent index and WAL paths.
+            let safe_id = instance_storage_key(&instance.id);
+
             // Create and add RocksDB index provider if persist_index is enabled
             if instance.persist_index {
-                let safe_id = instance_storage_key(&instance.id);
                 let index_path = PathBuf::from(format!("./data/{safe_id}/index"));
                 info!(
                     "Enabling persistent indexing for instance '{}' with RocksDB at: {}",
@@ -484,7 +486,6 @@ impl DrasiServer {
 
             // Create WAL provider for durable source event persistence
             {
-                let safe_id = instance_storage_key(&instance.id);
                 let wal_path = PathBuf::from(format!("./data/{safe_id}/wal"));
                 info!(
                     "Enabling WAL provider for instance '{}' at: {}",

tests/wal_provider_test.rs

@@ -0,0 +1,74 @@
+// Copyright 2025 The Drasi Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Integration tests for the redb-backed WAL provider.
+//!
+//! The WAL provider is the primary feature added in this PR and is wired into
+//! every instance, so these tests verify:
+//! - `RedbWalProvider` can be created from a directory path without panicking
+//! - `DrasiLib::builder().with_wal_provider(...)` accepts the provider and the
+//!   instance reaches Running and stops cleanly
+//!
+//! Per-source WAL file/directory creation is exercised by the `drasi-wal-redb`
+//! crate's own tests, since it only happens once a source appends events.
+
+use anyhow::Result;
+use drasi_lib::DrasiLib;
+use drasi_wal_redb::RedbWalProvider;
+use std::sync::Arc;
+use tempfile::TempDir;
+
+/// Test that RedbWalProvider can be created with a valid directory path.
+#[test]
+fn test_redb_wal_provider_creation() {
+    let temp_dir = TempDir::new().expect("Failed to create temp directory");
+    let wal_path = temp_dir.path().join("wal");
+
+    let provider = RedbWalProvider::new(&wal_path);
+
+    // Provider should be created successfully.
+    drop(provider);
+}
+
+/// Test that DrasiLib builder accepts a redb WAL provider and the instance
+/// starts and stops cleanly.
+#[tokio::test]
+async fn test_drasi_lib_builder_with_redb_wal_provider() -> Result<()> {
+    let temp_dir = TempDir::new()?;
+    let wal_path = temp_dir.path().join("wal");
+
+    let provider = RedbWalProvider::new(&wal_path);
+
+    let core = DrasiLib::builder()
+        .with_id("test-wal-provider")
+        .with_wal_provider(Arc::new(provider))
+        .build()
+        .await?;
+
+    core.start().await?;
+    assert!(core.is_running().await);
+    drasi_lib::wait_for_status(
+        &core.component_graph(),
+        "__component_graph__",
+        &[drasi_lib::channels::ComponentStatus::Running],
+        std::time::Duration::from_secs(5),
+    )
+    .await
+    .expect("component graph should reach Running");
+
+    core.stop().await?;
+    assert!(!core.is_running().await);
+
+    Ok(())
+}