Selective replication for Kafka. Filter, transform, redact, and route records between topics and clusters without deploying Kafka Connect.
StreamForge moves only the records and fields that downstream systems need. One source topic can feed analytics, lake, and lower-trust destinations with an independent filter and transform for each route.
Kafka source ──► filter ──► transform ──┬──► analytics topic
└──► PII-safe topic
- Route records by payload, key, headers, and timestamps.
- Reshape events and remove or hash sensitive fields before delivery.
- Fan out one source topic into destination-specific representations.
- Run opt-in, digest-pinned WebAssembly filters and transforms inside explicit resource and capability limits.
- Run as a standalone Rust binary or through the Kubernetes operator.
- Observe delivery, errors, lag, retries, and dead-letter records with Prometheus metrics.
Prerequisites: a Rust toolchain, Podman, and a Podman Compose provider.
podman compose -f examples/redpanda/docker-compose.yml up -d
cargo run --quiet --bin streamforge-validate -- \
examples/redpanda/selective-replication.yaml
CONFIG_FILE=examples/redpanda/selective-replication.yaml \
cargo run --release --bin streamforgeKeep StreamForge running, then follow the five-minute quickstart to create the topics, publish one order, and inspect the two destination-specific outputs.
| Goal | Start here |
|---|---|
| Understand the product boundary | When to use StreamForge |
| Build a selective replication pipeline | Usage guide |
| Learn the filter and transform language | DSL reference |
| Build a sandboxed custom filter or transform | WebAssembly UDFs |
| Deploy with Podman or Kubernetes | Deployment guide |
| Configure TLS and SASL | Security configuration |
| Operate and troubleshoot a pipeline | Operations |
| Browse the complete public documentation | StreamForge documentation |
Use the binary or container when configuration is managed directly by your deployment system. Start with Podman.
Use the operator and StreamforgePipeline custom resource when pipelines
should be managed declaratively. Start with
Kubernetes or the
Helm chart.
StreamForge targets Kafka-compatible brokers. Kafka is the primary target in the current documentation; Redpanda is covered for the selective-replication workflows exercised by this repository.
StreamForge is not positioned as a replacement for MirrorMaker 2 active-active replication and offset-sync workflows, or as a general-purpose stateful stream processor. See Compatibility for the tested scope.
Performance depends on message shape, partitions, broker configuration, delivery guarantees, and hardware. The public documentation therefore provides measurement and tuning guidance, not a universal throughput claim. Results are published only after a reproducible, like-for-like comparison passes record-count and delivery validation.
See the contributing guide for development setup, testing, and pull-request expectations. Use GitHub Discussions for questions and GitHub Issues for reproducible defects or feature proposals.
StreamForge is licensed under the Apache License 2.0.