Skip to content

Expose storageBackends top-level config to enable named storage backend references #106

@danielgerlag

Description

@danielgerlag

Summary

drasi-lib supports a storage_backends config at the DrasiLib level that allows defining named storage backends (e.g., rocksdb, redis, memory) which queries can reference by name via storageBackend: <name>. However, drasi-server does not expose this as a top-level config field — the server config schema only accepts:

apiVersion, id, host, port, logLevel, persistConfig, persistIndex, enableUi, solutionsDir,
stateStore, secretStore, defaultPriorityQueueCapacity, defaultDispatchBufferCapacity,
pluginRegistry, autoInstallPlugins, plugins, verifyPlugins, trustedIdentities,
hotReloadPlugins, hotReloadDebounceMs, corsAllowedOrigins, sources, queries, reactions,
identityProviders, instances

Current Behavior

  • Per-query inline storageBackend objects work (parsed as serde_json::Value and deserialized to StorageBackendRef::Inline):

    queries:
      - id: my-query
        storageBackend:
          backend_type: rocksdb
          path: /data/drasi/indexes
  • A string value like storageBackend: my-rocks deserializes as StorageBackendRef::Named("my-rocks"), but always fails at runtime because no named backends are defined.

  • The legacy persistIndex: true creates a single RocksDB at ./data/{instance_id}/index for the entire instance, but this is all-or-nothing and doesn't allow per-query backend choice.

Desired Behavior

Expose the storageBackends top-level field (matching what drasi-lib already supports in its config schema at DrasiLibConfig::storage_backends):

storageBackends:
  - id: persistent
    backend_type: rocksdb
    path: /data/drasi/indexes

  - id: distributed
    backend_type: redis
    connection_string: "redis://redis:6379"

queries:
  - id: my-query
    storageBackend: persistent   # Named reference
  - id: other-query
    storageBackend: distributed  # Different backend

Why

  1. DRY — With multiple queries, repeating the full inline object on each query is verbose and error-prone.
  2. Flexibility — Different queries may need different backends (e.g., high-throughput queries on Redis, others on local RocksDB).
  3. Consistency — drasi-lib already validates named references (DrasiLibConfig::validate() checks that all referenced backend names exist). The server should pass these through.

References

  • drasi-lib config schema: drasi-core/lib/src/config/schema.rsDrasiLibConfig::storage_backends: Vec<StorageBackendConfig>
  • drasi-lib index factory: drasi-core/lib/src/indexes/factory.rsIndexFactory::new(backends, ...)
  • drasi-lib builder: drasi-core/lib/src/builder.rsadd_storage_backend()
  • Docs PR documenting this (ahead of server support): Add named storage backends configuration docs#215

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels
    No fields configured for Feature.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions