[Feature Request] Support Custom Backends for Toolkits

[Wendong-Fan]

Required prerequisites

• I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• Consider asking first in a Discussion.

Motivation

Enable CAMEL Toolkits to use pluggable storage backends, allowing flexible data persistence across local filesystem, cloud storage, databases, and more.

Motivation

Currently, toolkits that need file/data operations are tightly coupled to specific storage implementations. Users need flexibility to:

• Use in-memory storage for temporary/ephemeral workspaces
• Store data in cloud storage (S3, Azure Blob, GCS)
• Use databases (PostgreSQL, Redis) as storage backends
• Route different paths to different backends (e.g., /temp/ → memory, /data/ → S3)
• Apply access control policies across any backend

Proposed Design

1. Backend Protocol

class BaseBackend(ABC):
    """Abstract base class for all backends."""

    @abstractmethod
    def read(self, path: str, offset: int = 0, limit: int = 2000) -> str: ...

    @abstractmethod
    def write(self, path: str, content: str) -> WriteResult: ...

    @abstractmethod
    def edit(self, path: str, old: str, new: str) -> EditResult: ...

    @abstractmethod
    def delete(self, path: str, recursive: bool = False) -> DeleteResult: ...

    @abstractmethod
    def ls_info(self, path: str = "/") -> List[FileInfo]: ...

    @abstractmethod
    def glob_info(self, pattern: str, path: str = "/") -> List[FileInfo]: ...

    @abstractmethod
    def grep_raw(self, pattern: str, path: str = None) -> List[GrepMatch]: ...

    @abstractmethod
    def exists(self, path: str) -> bool: ...

    @abstractmethod
    def mkdir(self, path: str) -> WriteResult: ...

2. Built-in Backends

Backend	Description
FilesystemBackend	Local disk with sandboxing support
StateBackend	In-memory ephemeral storage
S3Backend	AWS S3 object storage
CompositeBackend	Route paths to different backends
PolicyWrapper	Add access control policies

3. Usage Example

from camel.toolkits.backends import (
    FilesystemBackend, StateBackend, S3Backend, CompositeBackend
)

# Example 1: FileToolkit with S3 backend
from camel.toolkits import FileToolkit
toolkit = FileToolkit(backend=S3Backend(bucket="my-data"))

# Example 2: Any backend-aware toolkit with composite routing
toolkit = SomeToolkit(
    backend=CompositeBackend(
        default=StateBackend(),  # temp data in memory
        routes={
            "/persist/": FilesystemBackend(root_dir="/data"),
            "/cloud/": S3Backend(bucket="results")
        }
    )
)

# Example 3: Custom toolkit using backend
class MyCustomToolkit(BackendAwareToolkit):
    def save_result(self, data: str) -> str:
        result = self.backend.write("/output/result.json", data)
        return "Saved" if result.success else f"Error: {result.error}"

Implementation Tasks

• Define data types (types.py)
• Implement BaseBackend protocol (base.py)
• Implement FilesystemBackend with path sandboxing
• Implement StateBackend for in-memory storage
• Implement CompositeBackend for path routing
• Implement PolicyWrapper for access control
• (Optional) Implement S3Backend
• Create BackendAwareToolkit base class
• Integrate backend support into existing toolkits (e.g., FileToolkit)
• Add unit tests
• Add documentation and examples

Solution

No response

Alternatives

No response

Additional context

No response