v1.7.5

What's Changed

• fix: Handle udsource disconnect when ACK is invoked by @BulkBeing in #3265
• fix: start lastScaledAt as creationTime by default by @Koalk in #3358
• fix: Pod deletion (scale down) caused transient unhealthy vertex/mvtx by @whynowy in #3366
• chore: downgrade health check error log to warn by @adarsh0728 in #3108

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.4/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.4/validating-webhook-install.yaml

Full Changelog: v1.7.4...v1.7.5

v1.7.4

What's Changed

• chore: set max message size to 64mb for udsource by @yhl25 in #3312

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.4/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.4/validating-webhook-install.yaml

Full Changelog: v1.7.3...v1.7.4

v1.7.3

What's Changed

• fix: sum of rate metrics across all partitions by @yhl25 in #3243

Full Changelog: v1.7.2...v1.7.3

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.3/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.3/validating-webhook-install.yaml

v1.7.2

What's Changed

Bug Fixes

• Source Autoscaling fix #3205
• InitContainerStatuses in health checks #3201
• Pending chart for source vertex in UI #3209
• Restart on non-retryable kafka source ack errors #3194
• Structured Logging by default #3056
• Numaplane critical error metric for better Analysis Template #3154
• #3167 (part of this PR that fixes race condition)
• SQS Metadata support - #3095
• SQS Sink Optimizations - #3153 - only the sqs sink
• Reduce Watermark Spurious Log #3225
• chore: minor fixes for reduce #3229

Full Changelog: v1.7.0...v1.7.2

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.2/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.2/validating-webhook-install.yaml

v1.7.1

Fixes

• b72b8f4 fix: use std context (#3081)
• ac8bfeb fix: avoid panic in sink code (#3128)
• eafcf14 fix(cve): remove unused eval which uses expr that has CVE (#3083)
• 5b017de chore: bump crypto from 0.36.0 to 0.45.0 (#3082)
• f713f24 fix: graceful shutdown of sigkills in map udf container (#3096)
• e7b62d2 fix: alignment and text wrapping in Errors tab (#3078)
• b1e5996 fix: add back drop metrics after sink refactor (#3074)

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.1/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.1/validating-webhook-install.yaml

v1.7.0

Major Features 🔥

• Enables fine-grained control over UDF invocation rates using Distributed Throttling.
• Map Container Support for MonoVertex
• Graceful shutdown for sources, we have exposed nack functionality to user-defined source (available in SDKs with version > 0.11)

Recommend SDK

• Please use 0.10 and above. Refer Compatibility Matrix
• Contextual debugging available for Pipeline

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.0/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.0/validating-webhook-install.yaml

New Contributors

• @romanalexander made their first contribution in #2865
• @Siddhant-K-code made their first contribution in #2864
• @hasmukhmparmar-27 made their first contribution in #2879
• @Koalk made their first contribution in #2919
• @turtlelovesshoes made their first contribution in #3036
• @IversenMichael made their first contribution in #3071

Full Changelog: v1.6.0...v1.7.0

v1.7.0-rc1

Major Features 🔥

• Enables fine-grained control over UDF invocation rates using Distributed Throttling.
• Map Container Support for MonoVertex
• Graceful shutdown for sources, we have exposed nack functionality to user-defined source (available in SDKs with version > 0.11)

Recommend SDK

• Please use 0.10 and above. Refer Compatibility Matrix
• Contextual debugging available for Pipeline

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.0-rc1/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.7.0-rc1/validating-webhook-install.yaml

New Contributors

• @yogesh1801 made their first contribution in #2729
• @sapkota-aayush made their first contribution in #2744
• @saty9 made their first contribution in #2832

Full Changelog: v1.6.0...v1.7.0-rc1

v1.5.3

Release Notes: Numaflow v1.5.3

🐛 Bug Fixes

SQS-related Fixes

• Fixed SQS sink batch entry ID handling (#2951)

Core Platform Fixes

• Fixed fallback sink validation (#2947)
• Added spec.replicas field to minimal CRDs (#2910)
• Fixed controller reconciliation after container restarts (#2855)

---

Full Changelog: v1.5.2...v1.5.3

v1.6.0

Major Features 🔥

• Rust Dataplane is the default Dataplane (Golang will be deprecated in next release). To switch to Golang Runtime, set NUMAFLOW_RUNTIME

    - name: vtx-name
      containerTemplate:
        env:
          - name: NUMAFLOW_RUNTIME
            value: "golang"

Recommend SDK

• Please use 0.10 and above. Refer Compatibility Matrix
• Contextual debugging only available for MonoVertex
• Python MultiProc Map won't work

🚨 Breaking Change 🚨

• If you have a Reduce vertex with persistence, pipeline should be recreated. The WAL format has changed.

What's Changed

• doc: accumulator doc by @vigith in #2609
• feat: enable pod template support for serving deployment by @whynowy in #2620
• feat: exponential backoff retry strategy for sink by @adarsh0728 in #2614
• fix: potential incorrect k8s minor version by @whynowy in #2647
• feat: Kafka source - Rust implementation by @BulkBeing in #2636
• feat: Aligned Window for Reduce Async Data Movement by @yhl25 in #2618
• feat: Rust pipeline metrics by @adarsh0728 in #2666
• fix: conditional forwarding in Async Dataplane by @vigith in #2668
• fix: return 400 if serving ID header contains '.' by @vigith in #2669
• fix: retry attempts calculation and logs in retry strategy by @adarsh0728 in #2653
• feat(doc): update quick start and UI docs by @adarsh0728 in #2622
• feat: Kafka sink implementation in Rust by @BulkBeing in #2672
• feat: extension for http source by @vigith in #2687
• feat: optionally enable Wire compression to optimize network and ISB stability by @vigith in #2709
• feat: add serving http server by @cosmic-chichu in #2633
• feat: http source should honor response status codes sent to clients by @yhl25 in #2732
• feat: insert id into generator payload by @vigith in #2735
• feat: Pulsar Sink implementation by @BulkBeing in #2745
• docs: enhance user-defined sinks docs with message-response contract by @sapkota-aayush in #2747
• test(ui): partially fix UI test by @tmenjo in #2726
• fix(UI): pending and ackPending buffer info on Edges by @yogesh1801 in #2746
• feat: Unaligned Window for Async Data Movement by @yhl25 in #2699
• docs: Add user guide for GPU annotation and resource requests in vertices by @sapkota-aayush in #2753
• fix: use kafka timestamp as event time by @yhl25 in #2781
• fix: jetstream source should use published timestamp by @yhl25 in #2778
• doc: Example PDB and Anti-Affinity config for ISB by @vigith in #2787
• feat: e2e tests with rust runtime by @adarsh0728 in #2693
• fix: kafka writes messages to same partition when setKey is not set by @yhl25 in #2797
• feat: Daemon server to use http based watermark fetcher by @yhl25 in #2798
• fix(UI): health filter & support for mono-vertex health on pipeline listing page by @adarsh0728 in #2761
• doc: User documentation for Jetstream source by @BulkBeing in #2826
• fix: Detect tokio worker threads by @BulkBeing in #2831
• fix: handle stream already existing when making consumer by @saty9 in #2832
• fix: add serving pipeline transformer config by @cosmic-chichu in #2835
• feat: Filter subjects and deliver policy as user options by @BulkBeing in #2837
• feat: enable rust runtime by default by @yhl25 in #2833
• feat: unpause pipelines vertices and mvtx from min replicas by @kohlisid in #2840
• fix: graceful shutdown during reduce udf panic by @yhl25 in #2845
• fix: requeue reconciliation after seeing container restart by @whynowy in #2855

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.6.0/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.6.0/validating-webhook-install.yaml

New Contributors

• @yogesh1801 made their first contribution in #2729
• @sapkota-aayush made their first contribution in #2744
• @saty9 made their first contribution in #2832

Full Changelog: v1.5.0...v1.6.0

v1.5.2

What's Changed

• chore: option to disable read ahead in udf (#2851)
• feat: option to unpause pipelines vertices and mvtx from min replicas (#2840)

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.2/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.2/validating-webhook-install.yaml

Full Changelog: v1.5.1...v1.5.2