Skip to content

feat(wasm): Add sandboxed UDF runtime - #34

Merged
rahulbsw merged 2 commits into
mainfrom
feature/wasm-udf
Jul 26, 2026
Merged

feat(wasm): Add sandboxed UDF runtime#34
rahulbsw merged 2 commits into
mainfrom
feature/wasm-udf

Conversation

@rahulbsw

Copy link
Copy Markdown
Owner

Summary

Adds production-oriented, stateless WebAssembly UDF support to StreamForge
using the Wasmtime component model.

  • adds versioned WIT worlds for envelope-aware filters, JSON value transforms,
    and mutable envelope transforms
  • composes stages as filter → value transform → envelope mutations, with native
    logic preceding the corresponding UDF stage
  • verifies digest-pinned local components before Kafka startup and rejects WASI
    or other unresolved imports
  • enforces execution, input/output, memory, table, stack, artifact, and
    concurrency limits
  • integrates typed fail, DLQ, skip-and-log, and continue policies without
    rerunning successful destinations
  • adds bounded-label metrics, operator ConfigMap/PVC delivery, hardened pod
    settings, and rollout checksums
  • includes a Rust guest SDK, runnable fixtures, WIT ABI, usage documentation,
    configuration schema, and roadmap updates

Why

StreamForge needs custom policy and transformation logic beyond the native DSL
without loading native plugins or granting ambient filesystem, network,
environment, clock, or random access. The component boundary provides a
versioned ABI while digest verification and explicit resource limits keep the
extension point bounded and auditable.

Wasmtime is pinned to the security-patched 36.0.10 release. Dependency audit
failures now block CI, and CI compiles all guest examples and verifies fixture
digests.

User and developer impact

UDFs are opt-in. Configurations without a top-level wasm registry retain the
existing native path and do not initialize Wasmtime.

This aligns native destination processing with the documented order: value
transforms now precede native key/header/timestamp envelope mutations. Pipelines
that derived envelope fields from values removed by the value transform must
retain those inputs or update their envelope rules.

See docs/WASM_UDFS.md and udf-sdk/rust/README.md for configuration, ABI,
build, deployment, security, and performance guidance.

Validation

  • cargo test --all --locked --offline --no-fail-fast: 525 passed, 30 ignored,
    0 failed
  • cargo clippy --all-targets --all-features --locked --offline -- -D warnings
  • cargo audit --no-fetch --ignore RUSTSEC-2024-0437: zero vulnerabilities;
    seven warnings inherited from main
  • release build for streamforge and streamforge-validate
  • release validator loaded, verified, compiled, linked, and probed the
    digest-pinned smoke component
  • operator: 7 tests, warnings-denied Clippy, formatting, and Helm lint
  • all three Rust guests compile for wasm32-unknown-unknown
  • fixture SHA-256 manifest, JSON schema, workflow/CRD YAML, package contents,
    formatting, and diff checks passed
  • complete WASM Criterion target and isolated no-UDF controls completed; local
    controls remained within the 3% diagnostic budget
  • loopback Kafka smoke reconciled 7,599,000 records exactly with zero processing
    errors; the 30-second dirty-worktree result is diagnostic and not a
    publication claim

Controlled same-revision A/B performance runs remain the release gate;
shared/local timings are retained as development evidence only.

Add digest-pinned Wasmtime UDF filters and transforms, bounded execution, pipeline policies, operator delivery, SDK fixtures, documentation, CI security gates, and performance coverage.

Co-Authored-By: claude-flow <ruv@ruv.net>
Use the standard integer multiple check accepted by the current stable Clippy lint set.

Co-Authored-By: claude-flow <ruv@ruv.net>
@rahulbsw rahulbsw added the security Security-related changes label Jul 26, 2026
@rahulbsw rahulbsw changed the title feat(wasm): add sandboxed UDF runtime feat(wasm): Add sandboxed UDF runtime Jul 26, 2026
@rahulbsw rahulbsw added the skip-size-check Bypass PR size check for intentionally large PRs label Jul 26, 2026
@rahulbsw
rahulbsw marked this pull request as ready for review July 26, 2026 01:19
@rahulbsw
rahulbsw merged commit f8be712 into main Jul 26, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant