v1.6.0

Major Features 🔥

• Rust Dataplane is the default Dataplane (Golang will be deprecated in next release). To switch to Golang Runtime, set NUMAFLOW_RUNTIME

    - name: vtx-name
      containerTemplate:
        env:
          - name: NUMAFLOW_RUNTIME
            value: "golang"

Recommend SDK

• Please use 0.10 and above. Refer Compatibility Matrix
• Contextual debugging only available for MonoVertex
• Python MultiProc Map won't work

🚨 Breaking Change 🚨

• If you have a Reduce vertex with persistence, pipeline should be recreated. The WAL format has changed.

What's Changed

• doc: accumulator doc by @vigith in #2609
• feat: enable pod template support for serving deployment by @whynowy in #2620
• feat: exponential backoff retry strategy for sink by @adarsh0728 in #2614
• fix: potential incorrect k8s minor version by @whynowy in #2647
• feat: Kafka source - Rust implementation by @BulkBeing in #2636
• feat: Aligned Window for Reduce Async Data Movement by @yhl25 in #2618
• feat: Rust pipeline metrics by @adarsh0728 in #2666
• fix: conditional forwarding in Async Dataplane by @vigith in #2668
• fix: return 400 if serving ID header contains '.' by @vigith in #2669
• fix: retry attempts calculation and logs in retry strategy by @adarsh0728 in #2653
• feat(doc): update quick start and UI docs by @adarsh0728 in #2622
• feat: Kafka sink implementation in Rust by @BulkBeing in #2672
• feat: extension for http source by @vigith in #2687
• feat: optionally enable Wire compression to optimize network and ISB stability by @vigith in #2709
• feat: add serving http server by @cosmic-chichu in #2633
• feat: http source should honor response status codes sent to clients by @yhl25 in #2732
• feat: insert id into generator payload by @vigith in #2735
• feat: Pulsar Sink implementation by @BulkBeing in #2745
• docs: enhance user-defined sinks docs with message-response contract by @sapkota-aayush in #2747
• test(ui): partially fix UI test by @tmenjo in #2726
• fix(UI): pending and ackPending buffer info on Edges by @yogesh1801 in #2746
• feat: Unaligned Window for Async Data Movement by @yhl25 in #2699
• docs: Add user guide for GPU annotation and resource requests in vertices by @sapkota-aayush in #2753
• fix: use kafka timestamp as event time by @yhl25 in #2781
• fix: jetstream source should use published timestamp by @yhl25 in #2778
• doc: Example PDB and Anti-Affinity config for ISB by @vigith in #2787
• feat: e2e tests with rust runtime by @adarsh0728 in #2693
• fix: kafka writes messages to same partition when setKey is not set by @yhl25 in #2797
• feat: Daemon server to use http based watermark fetcher by @yhl25 in #2798
• fix(UI): health filter & support for mono-vertex health on pipeline listing page by @adarsh0728 in #2761
• doc: User documentation for Jetstream source by @BulkBeing in #2826
• fix: Detect tokio worker threads by @BulkBeing in #2831
• fix: handle stream already existing when making consumer by @saty9 in #2832
• fix: add serving pipeline transformer config by @cosmic-chichu in #2835
• feat: Filter subjects and deliver policy as user options by @BulkBeing in #2837
• feat: enable rust runtime by default by @yhl25 in #2833
• feat: unpause pipelines vertices and mvtx from min replicas by @kohlisid in #2840
• fix: graceful shutdown during reduce udf panic by @yhl25 in #2845
• fix: requeue reconciliation after seeing container restart by @whynowy in #2855

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.6.0/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.6.0/validating-webhook-install.yaml

New Contributors

• @yogesh1801 made their first contribution in #2729
• @sapkota-aayush made their first contribution in #2744
• @saty9 made their first contribution in #2832

Full Changelog: v1.5.0...v1.6.0