Skip to content

README.md

Latest commit

 

History

History
104 lines (76 loc) · 4.27 KB

README.md

File metadata and controls

104 lines (76 loc) · 4.27 KB
Raw

StreamForge

Selective replication for Kafka. Filter, transform, redact, and route records between topics and clusters without deploying Kafka Connect.

Version CI Docs License

StreamForge moves only the records and fields that downstream systems need. One source topic can feed analytics, lake, and lower-trust destinations with an independent filter and transform for each route.

Kafka source ──► filter ──► transform ──┬──► analytics topic
                                       └──► PII-safe topic

Why StreamForge

  • Route records by payload, key, headers, and timestamps.
  • Reshape events and remove or hash sensitive fields before delivery.
  • Fan out one source topic into destination-specific representations.
  • Run opt-in, digest-pinned WebAssembly filters and transforms inside explicit resource and capability limits.
  • Run as a standalone Rust binary or through the Kubernetes operator.
  • Observe delivery, errors, lag, retries, and dead-letter records with Prometheus metrics.

Run the local demo

Prerequisites: a Rust toolchain, Podman, and a Podman Compose provider.

podman compose -f examples/redpanda/docker-compose.yml up -d

cargo run --quiet --bin streamforge-validate -- \
  examples/redpanda/selective-replication.yaml

CONFIG_FILE=examples/redpanda/selective-replication.yaml \
  cargo run --release --bin streamforge

Keep StreamForge running, then follow the five-minute quickstart to create the topics, publish one order, and inspect the two destination-specific outputs.

Choose a path

Goal Start here
Understand the product boundary When to use StreamForge
Build a selective replication pipeline Usage guide
Learn the filter and transform language DSL reference
Build a sandboxed custom filter or transform WebAssembly UDFs
Deploy with Podman or Kubernetes Deployment guide
Configure TLS and SASL Security configuration
Operate and troubleshoot a pipeline Operations
Browse the complete public documentation StreamForge documentation

Deployment modes

Standalone

Use the binary or container when configuration is managed directly by your deployment system. Start with Podman.

Kubernetes

Use the operator and StreamforgePipeline custom resource when pipelines should be managed declaratively. Start with Kubernetes or the Helm chart.

Compatibility and boundaries

StreamForge targets Kafka-compatible brokers. Kafka is the primary target in the current documentation; Redpanda is covered for the selective-replication workflows exercised by this repository.

StreamForge is not positioned as a replacement for MirrorMaker 2 active-active replication and offset-sync workflows, or as a general-purpose stateful stream processor. See Compatibility for the tested scope.

Performance policy

Performance depends on message shape, partitions, broker configuration, delivery guarantees, and hardware. The public documentation therefore provides measurement and tuning guidance, not a universal throughput claim. Results are published only after a reproducible, like-for-like comparison passes record-count and delivery validation.

Contributing

See the contributing guide for development setup, testing, and pull-request expectations. Use GitHub Discussions for questions and GitHub Issues for reproducible defects or feature proposals.

License

StreamForge is licensed under the Apache License 2.0.