v2.4.0-beta.3

What's Changed

• ci: fix perf action by @Yisaer in #3990
• ci: add explicit permissions to semgrep workflow by @ngjaying in #3992
• fix(test): resolve flaky FVT test by @ngjaying in #3993
• fix: enable race ci and fix problems by @ngjaying in #3996
• feat(trial): replace websocket with sse by @ngjaying in #3991
• fix: fix TestWindowState race test by @Yisaer in #3997
• test(http): add oauth test with header by @ngjaying in #3998

Full Changelog: v2.4.0-beta.2...v2.4.0-beta.3

v2.4.0-beta.2

What's Changed

• ci: add perf action by @Yisaer in #3985
• ci: add semgrep for nested lock detection by @ngjaying in #3988
• fix: add syncx wrapper and fix deadlocks by @ngjaying in #3989

Full Changelog: v2.4.0-beta.1...v2.4.0-beta.2

v2.4.0-beta.1

What's Changed

• docs: add Swagger API playground link by @ankur-paan in #3975
• fix(server): prevent cpu api panic by @ngjaying in #3978
• fix(stream): allow deleting corrupted streams by @ngjaying in #3979
• docs: update helm installation to use OCI registry by @ngjaying in #3980
• docs: update Kafka Docker image by @igor-soldev in #3977
• fix(conf): add log symlink validation by @ngjaying in #3981
• fix: replace path.Join with filepath.Join by @ngjaying in #3982
• fix: fix sql sink quote string value by @Yisaer in #3983
• fix(video): enforce image2pipe by @ngjaying in #3984

New Contributors

• @ankur-paan made their first contribution in #3975
• @igor-soldev made their first contribution in #3977

Full Changelog: v2.4.0-alpha.3...v2.4.0-beta.1

v2.4.0-alpha.3

Features

Video Source Refactoring

Refactored the video source to use a push model, improving efficiency and performance (#3959).

Sink Enhancements

Added HasFields support to sinks, enabling more flexible field validation (#3964).

Server Improvements

Implemented version-based conflict resolution for concurrent upserts (#3970).

---

Bug Fixes

• Fixed function channel timeout issues in Go SDK (#3965)
• Fixed function channel timeout issues in Python SDK (#3967)

---

Full Changelog: v2.4.0-alpha.2...v2.4.0-alpha.3

v2.4.0-alpha.2

⚠️ Breaking Changes

IMPORTANT: This release includes security enhancements that may affect existing deployments.

1. SSRF Protection Enabled by Default

   • The new enablePrivateNet configuration defaults to false, which blocks access to private network addresses (e.g., localhost, 127.0.0.1, internal IPs).
   • Action Required: If your rules rely on accessing local resources (local REST services, local databases, etc.), you must set enablePrivateNet: true in etc/kuiper.yaml.
   • Documentation

2. File Access Restriction Enabled by Default

   • The new allowExternalFileAccess configuration defaults to false, restricting file access to the data/uploads directory only.
   • Action Required: If your plugins or schemas need to access files outside the uploads directory, set allowExternalFileAccess: true.
   • Documentation

---

New Features

Temporary Streams (#3940)

Introduced temporary streams that exist only in memory and are not persisted. They are defined using TEMP="true" in the stream definition and are ideal for intermediate data processing or testing. Temporary streams cannot be replaced and can only be used by temporary rules.

• Documentation

State Window Partition By (#3936)

State windows now support the PARTITION BY clause, enabling data partitioning into separate window groups. This allows more granular state tracking across different partitions.

• Documentation

Tuple Sink Format Support (#3954)

Tuple-based sinks now support configurable output formats, providing more flexibility in data serialization.

Video Source Enhancements (#3955)

Added new properties to the video source for better ffmpeg control:

• debugResp: Output ffmpeg response to logs for debugging
• inputArgs: Custom input arguments for ffmpeg (e.g., rtsp_transport: tcp)
• Documentation

Global Configuration Provider (#3942)

Added a global configuration provider that allows portable plugins and external components to access eKuiper's configuration settings programmatically.

API ID Validation (#3951)

Added comprehensive validation for resource identifiers. Stream, table, rule, connection, plugin, schema, and service names are now validated to prevent invalid characters.

---

Security Enhancements

SSRF Protection

Implemented Server-Side Request Forgery (SSRF) protection across all HTTP clients. Private network access is blocked by default.

• Configuration

File Access Restriction (#3950)

Added configurable file access restrictions to prevent unauthorized file system access.

• Configuration

Path Traversal Prevention

• Fixed upload embedded path traversal (#3958)
• Fixed path traversal in file downloads
• Enforced safe path validation for user input (#3911)
• Safe unzip implementation (#3931)

---

Bug Fixes

• Fixed HTTP refresh token support (#3922)
• Fixed REST sink access token handling
• Fixed SQL lookup unsafe string (#3930)
• Fixed wildcard expander limit in slice mode (#3925)
• Fixed bool type conversion issues (#3917, #3918)
• Fixed state window with GROUP BY key (#3916)

---

Dependency Updates

• Upgraded Go version
• Upgraded FoundationDB client to 7.3 (#3938)
• Bumped logrus, paho.mqtt.golang, golang.org/x/crypto, jose2go, gorilla/schema

---

Full Changelog: v2.4.0-alpha.1...v2.4.0-alpha.2

v2.3.1

What's Changed

• fix(trial): rename rule trial mock stream by @ngjaying in #3920
• fix: merge bugfixes for v2.3.1 by @ngjaying in #3935

Full Changelog: v2.3.0...v2.3.1

v2.4.0-alpha.1

What's Changed

• feat: add extstatetype support for Pebble by @weiji6 in #3856
• feat: revise GetByPrefix by @weiji6 in #3906
• feat: revise websocket source by @Yisaer in #3905
• feat(io): add rollingSize option to file sink by @e7217 in #3910
• feat(op): ratelimit support batch input by @ngjaying in #3907
• feat(slice): support select * by @ngjaying in #3912
• feat: support ToUint32Slice by @Yisaer in #3908
• feat: split checkpoint db by @Yisaer in #3909
• fix(io): fix sql sink build sql by @Yisaer in #3913

New Contributors

• @weiji6 made their first contribution in #3856
• @e7217 made their first contribution in #3910

Full Changelog: v2.3.0...v2.4.0-alpha.1

v2.3.0

SQL & Functions

• New extract Function: Added the extract function to easily retrieve data from nested structures. [Docs]
• Single Conditional State Window: Introduced the single condition state window for advanced, event-driven windowing logic. [Docs]
• Enhanced round Function: The round function now supports specifying the number of decimal places. [Docs]
• Aggregate Function Access in WHERE Clause: Now supports accessing aggregate functions within the WHERE clause. (e.g., SELECT * FROM sim5 WHERE a > avg(a) GROUP BY countwindow(2)).
• Multiple Stream/Table JOIN Support: Enabled joining data from multiple streams and tables.
• Slice Mode Updates: Implemented updates and improvements to Slice mode functionality.Added validation to detect and prevent operations in unsupported scenarios.

I/O & Sources/Sinks

• HTTP Pull Source State: HTTP Pull Source now supports state management similar to SQL Source.
• Streaming JSON for Sinks: Sinks now support streaming writes for JSON format output.
• Kafka Sink Dynamic Topic: Added support for dynamic topic configuration in Kafka Sink.
• SQL Source Support: Added support for the PostgreSQL int4 data type in SQL Source.
• MQTT Source Multiple Topics: MQTT Source now supports subscribing to multiple topics using a comma-separated list (e.g., datasource="topic1,topic2"). [Docs]
• WebSocket Scheme Configuration: WebSocket Source/Sink now supports scheme configuration (e.g., ws:// or wss://). [Docs]
• New Source: Added the Nexmark Source for benchmarking and testing.
• TFLite Plugin: Updated the TensorFlow version for the TFLite plugin.
• MQTT Source Session: Fixed issues related to MQTT source session configuration.
• WebSocket Client: Resolved issues in WebSocket client mode.

Operation & Management (Op)

• Stream Enhancement: Stream definition supports VERSION and quick configuration options. [Docs]
• Preserved Rule Definition: Starting or stopping a rule no longer overwrites the original rule string definition.
• Plugin Configuration Location: Plugin configurations are now installed by default under the data directory.
• Pre-defined Rules Location: Pre-defined rules can now be configured under the etc directory.
• New API for Rule Schema: Introduced GET /rules/{id}/schema to retrieve a rule's output schema. [Docs]
• Schema Support: Schema now supports versioning. [Docs]
• Temporary Rules: Added support for temporary rules (temp=true) which are not persisted and disappear after an eKuiper restart.
• REST API Audit Log: Added an option to enable/disable REST API audit logs.
• pprof Integration: Support for configuring pprof on the same port as the REST service.
• Rule Restart Policy Update (Deprecation): Automatic rule restart is now handled by the timed global patrol mechanism. The legacy restartStrategy configuration is deprecated and no longer takes effect.
• Performance: Improved performance for reading source configurations.
• Support for EoF (End of File) Message.
• Service Shutdown Logic: Updated service shutdown logic to first close the REST service, preventing new requests while the service is stopping.

Build & Internal Refactoring

• Helm Chart Added: Officially released the Helm chart for eKuiper deployment. [Link]
• Shared Stream Refactoring: Major refactoring of the creation, startup, and stop logic for rules using shared streams. This resolves issues related to slow rule stopping and related start/stop anomalies.
• Schema Management Refactoring: Refactored schema management to support the registration of new schema types.

Full Change List

• feat: refactor sliding window by @Yisaer in #3744
• feat: support acc function condition by @Yisaer in #3749
• feat(slice): add slice mode by @ngjaying in #3756
• fix: fix lag default value by @Yisaer in #3758
• feat: support state window by @Yisaer in #3757
• fix(http): transport setting by @ngjaying in #3759
• fix(planner): donot share conn src by @ngjaying in #3765
• feat: portable plugin install in data dir by @Yisaer in #3773
• fix(planner): strict share conn by @ngjaying in #3781
• fix(edgex): tags field access by @ngjaying in #3782
• build(deps): bump oauth2 by @dependabot[bot] in #3789
• feat: support patch/put rule tags by @Yisaer in #3790
• fix: fix init plugin/sink meta by @Yisaer in #3791
• fix: validate window version by @Yisaer in #3793
• fix(window): state window logic by @ngjaying in #3794
• fix: graceful stop rule timeout by @Yisaer in #3792
• fix(kv): prepare ts query by @odaysec in #3767
• fix(checkpoint): gob register map[string]time.time by @ngjaying in #3795
• docs: file hook docs by @Yisaer in #3796
• fix(rule): upsert closed rule start failed by @ngjaying in #3799
• feat(contract): support EoF message by @ngjaying in #3800
• feat(source): support eof msg by @ngjaying in #3801
• refactor(schema): modularize schema by @ngjaying in #3802
• feat(schema): support zip by @ngjaying in #3805
• fix: fix roll done error handle by @Yisaer in #3806
• feat: support eval aggfunc in where condition by @Yisaer in #3803
• feat(mod): support register schema to format by @ngjaying in #3809
• fix: support sql source timestamp scan by @Yisaer in #3811
• feat(converter): support json writer by @ngjaying in #3812
• feat: support httppull state by @Yisaer in #3804
• feat(rest): API to get stream/rule runtime schema by @ngjaying in #3816
• fix: merge bugfixes by @ngjaying in #3817
• feat(rule): support temp rule by @ngjaying in #3818
• chore: merge slice mode updates by @ngjaying in #3819
• fix: remove gu plugin sink ack by @Yisaer in #3820
• test: fix random ut failure by @ngjaying in #3821
• feat(schema): schema message id multiple levels by @ngjaying in #3822
• fix(schema): compatible with path schema id by @ngjaying in #3823
• feat(schema): allow any type by @ngjaying in #3824
• fix(schema): merger schemaId parse by @ngjaying in #3825
• feat: support log rest api visit by @Yisaer in #3827
• feat: support dyn kafka sink topic by @Yisaer in #3831
• feat: sql source support pg int4 by @Yisaer in #3832
• feat: support mqtt subscribe multi topics by @Yisaer in #3830
• docs: add mqtt source multi topic doc by @Yisaer in #3835
• feat: support single condition state window by @Yisaer in #3836
• fix(schema): enable schema init by @ngjaying in #3837
• feat: add ci chart releaser by @eznix86 in #3833
• feat(module): pass props for merger by @ngjaying in #3839
• docs: add single condition statewindow doc by @Yisaer in #3838
• feat: support setreturning function extract by @Yisaer in #3840
• feat(schema): support versions by @ngjaying in #3841
• docs: add extract doc by @Yisaer in #3842
• feat(rule): init rule from both etc and data by @ngjaying in #3844
• docs: add inc blog by @Yisaer in #38...

v2.3.0-beta.7

What's Changed

• fix(topo): clean rule id for multi level subtopo by @ngjaying in #3895
• fix(stream): forbid to update stream shared option by @ngjaying in #3897
• fix(conv): register json writer by @ngjaying in #3899
• fix: fix tuple collector batcher by @Yisaer in #3898
• docs: rule and stream update by @ngjaying in #3896

Full Changelog: v2.3.0-beta.6...v2.3.0-beta.7

v2.3.0-beta.6

What's Changed

• refactor(rule): do not export topo by @ngjaying in #3873
• fix(io): fix websocket parse url rawQuery by @Yisaer in #3877
• feat(api): add run id to stream context by @ngjaying in #3886
• refactor(rule): state code structure by @ngjaying in #3879
• chore(rule): remove rule auto restart by @ngjaying in #3880
• feat: revise GetByPrefix by @Yisaer in #3884
• feat(topo): subtopo lifecycle refactor by @ngjaying in #3885
• ci: push chart to OCI by @Rory-Z in #3878
• fix: schema partial import save data into db by @Yisaer in #3888
• refactor(rule): run restart strategy periodically by @ngjaying in #3887
• fix(subtopo): runtime error should reject new rule by @ngjaying in #3890
• fix: revise CopyConfContent by @Yisaer in #3889
• docs: fix docs by @Yisaer in #3892
• feat: stop rest server first by @Yisaer in #3891

Full Changelog: v2.3.0-beta.5...v2.3.0-beta.6