v1.8.0

Major Features 🔥

• Distributed Tracing: Numaflow can now export platform traces using OpenTelemetry Protocol (OTLP)
• Ordered Processing (input-order preservation): A new opt-in ordered processing mode provides partitioned FIFO semantics
• OnSuccess Sink: A new OnSuccess sink allows users to write a follow-up message to a secondary sink after the primary sink has successfully accepted the original message.
• MonoVertex Bypass Routing: Mvtx now supports bypass / short-circuit routing, letting users skip the UDF and/or primary sink and send messages directly to a fallback sink or onSuccess sink based on user-defined tags.
• Dark mode theme

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.8.0/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.8.0/validating-webhook-install.yaml

New Contributors

• @juniemariam made their first contribution in #3086
• @txmxthy made their first contribution in #3169
• @bobo made their first contribution in #3187
• @JagjotB made their first contribution in #3376
• @suryapratap-01 made their first contribution in #3407
• @DattoDarragh made their first contribution in #3430

Full Changelog: v1.7.0...v1.8.0

What's Changed

• docs: updated CHANGELOG.md by @github-actions[bot] in #3055
• chore: default to use structured logging by @whynowy in #3056
• chore: Fix code formatting in example code in user documentation by @BulkBeing in #3058
• fix: Kafka source/sink - pass client certs for mTLS when insecureSkipVerify is false by @BulkBeing in #3059
• feat: add support for OnSuccess sink by @vaibhavtiwari33 in #3040
• doc: add map to mvtx example by @vigith in #3066
• fix: cron parser uses 6 field format by @vigith in #3068
• fix(test): we need 5 sec ticker in test by @vigith in #3069
• chore: revert to use ReadWriteOnce as default accessMode to avoid breaking changes by @whynowy in #3070
• fix: build issues in #3071 by @BulkBeing in #3072
• fix: add back drop metrics after sink refactor by @adarsh0728 in #3074
• docs: updated CHANGELOG.md by @github-actions[bot] in #3073
• doc: a bit more content to pipeline document by @vigith in #3075
• fix: use std context by @vigith in #3081
• chore: bump crypto from 0.36.0 to 0.45.0 by @vigith in #3082
• chore: remove unused eval which uses expr that has CVE by @vigith in #3083
• doc: update watermark doc with init source delay by @vaibhavtiwari33 in #3084
• doc: Update timeout doc for accumulator where idle detection config is required by @vaibhavtiwari33 in #3085
• feat: enable OTLP metrics exporter for daemon services by @juniemariam in #3086
• chore(deps): bump node-forge from 1.3.1 to 1.3.2 in /ui by @dependabot[bot] in #3088
• fix: alignment and text wrapping in Errors tab by @adarsh0728 in #3078
• chore: Add a small elaboration for cron schedule parsing error by @vaibhavtiwari33 in #3094
• test: Add e2e test for on-success-sink by @vaibhavtiwari33 in #3067
• fix: graceful shutdown of sigkills in map udf container by @yhl25 in #3096
• doc: add fallback container for mvtx sink configuration by @vigith in #3097
• fix: Remove godebug env var initialization by @vaibhavtiwari33 in #3092
• chore: replace kafka with redpanda for e2e testing by @whynowy in #3077
• chore: e2e tests for metadata changes by @adarsh0728 in #3065
• chore: clean up managedFields from k8s objects from UI display by @whynowy in #3098
• doc: Documentation for on-success sink by @vaibhavtiwari33 in #3107
• chore: downgrade health check error log to warn by @adarsh0728 in #3108
• feat: Spec changes for short-circuiting mvtx by @vaibhavtiwari33 in #3105
• feat (UI): support onSuccess sink in MonoVertex graph by @adarsh0728 in #3110
• chore: update onSuccess sink image by @adarsh0728 in #3120
• chore: Upgrade Rust toolchain to 1.92 by @BulkBeing in #3124
• fix: avoid panic in sink code by @yhl25 in #3128
• chore(refactor): more idiomatic rust for server info by @KeranYang in #3130
• chore: refactor map component by @yhl25 in #3132
• feat: Mvtx short-circuiting using a bypass router struct by @vaibhavtiwari33 in #3126
• docs: updated CHANGELOG.md by @github-actions[bot] in #3137
• docs: Documentation for metric exposition format differences b/w numaflow versions by @vaibhavtiwari33 in #3142
• chore: add dummy rust daemon server by @KeranYang in #3138
• chore: Metrics changes for Mvtx UDF, on-success sink and drop metrics by @vaibhavtiwari33 in #3139
• refactor: define errors for ISB Reader by @vigith in #3148
• chore(deps): bump lodash from 4.17.21 to 4.17.23 in /ui by @dependabot[bot] in #3149
• refactor: error types and handling for Writer by @vigith in #3151
• test: E2E test for bypass scenarios for map and source transformer in monovertex by @vaibhavtiwari33 in #3134
• feat: SQS System Attributes and Custom Attributes Propagation by @cosmic-chichu in #3095
• chore: code cleanup and cargo fixes by @yhl25 in #3153
• lint: fail if cargo check has warnings by @vigith in #3156
• docs: Documentation for monovertex bypass routing by @vaibhavtiwari33 in #3152
• lint(clippy): clippy fix 144/165 by @vigith in #3157
• lint(clippy): remaining(105/165) by @vigith in #3158
• lint(clippy): remaining(12/165) by @vigith in #3159
• chore: support gRPC for rust mvtx daemon server by @KeranYang in #3143
• lint(clippy): fix all the remaining clippy issues by @vigith in #3161
• ci: block CI on clippy errors by @vigith in #3163
• doc: Update FAQs with a scenario for Server Info File not ready by @vaibhavtiwari33 in #3165
• docs(map): call out unary, batch, and streaming by @vigith in #3164
• chore: add critical error metric for numaplane analysis template by @adarsh0728 in #3154
• Add Starboard to USERS.md by @txmxthy in #3169
• chore: adding Playerdata to the list of users by @Koalk in #3171
• chore: write a macro for emitting critical errors by @vigith in #3168
• chore: support for keys in http source by @yhl25 in #3172
• chore: Preallocate Bytes/BytesMut when possible by @BulkBeing in #3173
• feat: add ISB Reader Trait and add it as Associated Type in NumaflowTypeConfig by @vigith in #3175
• feat: ISB Writer Trait integration (propagated via NumaflowTypeConfig) by @vigith in #3178
• chore(deps): bump bytes from 1.10.1 to 1.11.1 in /rust by @dependabot[bot] in #3186
• feat(ui): add dark mode theme by @bobo in htt...

v1.7.5

What's Changed

• fix: Handle udsource disconnect when ACK is invoked by @BulkBeing in #3265
• fix: start lastScaledAt as creationTime by default by @Koalk in #3358
• fix: Pod deletion (scale down) caused transient unhealthy vertex/mvtx by @whynowy in #3366
• chore: downgrade health check error log to warn by @adarsh0728 in #3108

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.4/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.4/validating-webhook-install.yaml

Full Changelog: v1.7.4...v1.7.5

v1.7.4

What's Changed

• chore: set max message size to 64mb for udsource by @yhl25 in #3312

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.4/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.4/validating-webhook-install.yaml

Full Changelog: v1.7.3...v1.7.4

v1.7.3

What's Changed

• fix: sum of rate metrics across all partitions by @yhl25 in #3243

Full Changelog: v1.7.2...v1.7.3

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.3/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.3/validating-webhook-install.yaml

v1.7.2

What's Changed

Bug Fixes

• Source Autoscaling fix #3205
• InitContainerStatuses in health checks #3201
• Pending chart for source vertex in UI #3209
• Restart on non-retryable kafka source ack errors #3194
• Structured Logging by default #3056
• Numaplane critical error metric for better Analysis Template #3154
• #3167 (part of this PR that fixes race condition)
• SQS Metadata support - #3095
• SQS Sink Optimizations - #3153 - only the sqs sink
• Reduce Watermark Spurious Log #3225
• chore: minor fixes for reduce #3229

Full Changelog: v1.7.0...v1.7.2

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.2/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.2/validating-webhook-install.yaml

v1.7.1

Fixes

• b72b8f4 fix: use std context (#3081)
• ac8bfeb fix: avoid panic in sink code (#3128)
• eafcf14 fix(cve): remove unused eval which uses expr that has CVE (#3083)
• 5b017de chore: bump crypto from 0.36.0 to 0.45.0 (#3082)
• f713f24 fix: graceful shutdown of sigkills in map udf container (#3096)
• e7b62d2 fix: alignment and text wrapping in Errors tab (#3078)
• b1e5996 fix: add back drop metrics after sink refactor (#3074)

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.1/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.1/validating-webhook-install.yaml

v1.7.0

Major Features 🔥

• Enables fine-grained control over UDF invocation rates using Distributed Throttling.
• Map Container Support for MonoVertex
• Graceful shutdown for sources, we have exposed nack functionality to user-defined source (available in SDKs with version > 0.11)

Recommend SDK

• Please use 0.10 and above. Refer Compatibility Matrix
• Contextual debugging available for Pipeline

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.0/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.0/validating-webhook-install.yaml

New Contributors

• @romanalexander made their first contribution in #2865
• @Siddhant-K-code made their first contribution in #2864
• @hasmukhmparmar-27 made their first contribution in #2879
• @Koalk made their first contribution in #2919
• @turtlelovesshoes made their first contribution in #3036
• @IversenMichael made their first contribution in #3071

Full Changelog: v1.6.0...v1.7.0

v1.7.0-rc1

Major Features 🔥

• Enables fine-grained control over UDF invocation rates using Distributed Throttling.
• Map Container Support for MonoVertex
• Graceful shutdown for sources, we have exposed nack functionality to user-defined source (available in SDKs with version > 0.11)

Recommend SDK

• Please use 0.10 and above. Refer Compatibility Matrix
• Contextual debugging available for Pipeline

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.0-rc1/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.0-rc1/validating-webhook-install.yaml

New Contributors

• @yogesh1801 made their first contribution in #2729
• @sapkota-aayush made their first contribution in #2744
• @saty9 made their first contribution in #2832

Full Changelog: v1.6.0...v1.7.0-rc1

v1.5.3

Release Notes: Numaflow v1.5.3

🐛 Bug Fixes

SQS-related Fixes

• Fixed SQS sink batch entry ID handling (#2951)

Core Platform Fixes

• Fixed fallback sink validation (#2947)
• Added spec.replicas field to minimal CRDs (#2910)
• Fixed controller reconciliation after container restarts (#2855)

---

Full Changelog: v1.5.2...v1.5.3

v1.6.0

Major Features 🔥

• Rust Dataplane is the default Dataplane (Golang will be deprecated in next release). To switch to Golang Runtime, set NUMAFLOW_RUNTIME

    - name: vtx-name
      containerTemplate:
        env:
          - name: NUMAFLOW_RUNTIME
            value: "golang"

Recommend SDK

• Please use 0.10 and above. Refer Compatibility Matrix
• Contextual debugging only available for MonoVertex
• Python MultiProc Map won't work

🚨 Breaking Change 🚨

• If you have a Reduce vertex with persistence, pipeline should be recreated. The WAL format has changed.

What's Changed

• doc: accumulator doc by @vigith in #2609
• feat: enable pod template support for serving deployment by @whynowy in #2620
• feat: exponential backoff retry strategy for sink by @adarsh0728 in #2614
• fix: potential incorrect k8s minor version by @whynowy in #2647
• feat: Kafka source - Rust implementation by @BulkBeing in #2636
• feat: Aligned Window for Reduce Async Data Movement by @yhl25 in #2618
• feat: Rust pipeline metrics by @adarsh0728 in #2666
• fix: conditional forwarding in Async Dataplane by @vigith in #2668
• fix: return 400 if serving ID header contains '.' by @vigith in #2669
• fix: retry attempts calculation and logs in retry strategy by @adarsh0728 in #2653
• feat(doc): update quick start and UI docs by @adarsh0728 in #2622
• feat: Kafka sink implementation in Rust by @BulkBeing in #2672
• feat: extension for http source by @vigith in #2687
• feat: optionally enable Wire compression to optimize network and ISB stability by @vigith in #2709
• feat: add serving http server by @cosmic-chichu in #2633
• feat: http source should honor response status codes sent to clients by @yhl25 in #2732
• feat: insert id into generator payload by @vigith in #2735
• feat: Pulsar Sink implementation by @BulkBeing in #2745
• docs: enhance user-defined sinks docs with message-response contract by @sapkota-aayush in #2747
• test(ui): partially fix UI test by @tmenjo in #2726
• fix(UI): pending and ackPending buffer info on Edges by @yogesh1801 in #2746
• feat: Unaligned Window for Async Data Movement by @yhl25 in #2699
• docs: Add user guide for GPU annotation and resource requests in vertices by @sapkota-aayush in #2753
• fix: use kafka timestamp as event time by @yhl25 in #2781
• fix: jetstream source should use published timestamp by @yhl25 in #2778
• doc: Example PDB and Anti-Affinity config for ISB by @vigith in #2787
• feat: e2e tests with rust runtime by @adarsh0728 in #2693
• fix: kafka writes messages to same partition when setKey is not set by @yhl25 in #2797
• feat: Daemon server to use http based watermark fetcher by @yhl25 in #2798
• fix(UI): health filter & support for mono-vertex health on pipeline listing page by @adarsh0728 in #2761
• doc: User documentation for Jetstream source by @BulkBeing in #2826
• fix: Detect tokio worker threads by @BulkBeing in #2831
• fix: handle stream already existing when making consumer by @saty9 in #2832
• fix: add serving pipeline transformer config by @cosmic-chichu in #2835
• feat: Filter subjects and deliver policy as user options by @BulkBeing in #2837
• feat: enable rust runtime by default by @yhl25 in #2833
• feat: unpause pipelines vertices and mvtx from min replicas by @kohlisid in #2840
• fix: graceful shutdown during reduce udf panic by @yhl25 in #2845
• fix: requeue reconciliation after seeing container restart by @whynowy in #2855

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.6.0/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.6.0/validating-webhook-install.yaml

New Contributors

• @yogesh1801 made their first contribution in #2729
• @sapkota-aayush made their first contribution in #2744
• @saty9 made their first contribution in #2832

Full Changelog: v1.5.0...v1.6.0