v2.0.5

release v2.0.5

v2.1.0

Please use v2.1.2 instead

#20107 fixed a bug that may lead to data loss.

For installation and running instructions, see Get started.

Main changes

SQL features

• Query syntax:
  • Technical preview: Supports ASOF JOIN to join and find the closest matching record by the event time or another ordered property. #18683
  • Supports AGGREGATE: prefixed scalar function in streaming aggregation. #18205
  • Supports using user-defined aggregate functions as window function. #18181
  • Supports blocking subscription cursors and configuring cursor timeouts. #18675
• SQL commands:
  • Enhances observability of cursors and subscription cursors by improving output results of SHOW SUBSCRIPTION CURSORS and SHOW CURSORS commands. #18896
• SQL functions & operators:
  • Technical Preview: Supports the TVF postgres_query . #18811
• System catalog:
  • Supports pg_catalog.pg_sequence. #18714
  • Supports information_schema.table_constraints. #18716
  • Supports information_schema.schemata. #18709

Connectors

• Breaking change: Changes scan.startup.mode=latest for NATS source connector to start consuming from next available message instead of last one. #18733
• Public preview: Supports shared Kafka sources, which can be disabled by session variable streaming_use_shared_source. #18749
• Supports recursively scanning file sources. #18324
• Supports schemaless ingestion for data in JSON format from Kafka sources by using INCLUDE payload clause. #18437
• Adds a set of options for NATS source connector based on the async_nats crate. #17615
• Adds a required option, consumer.durable_name, for NATS source connector. #18873
• Supports option max_packet_size for MQTT sources. #18520
• Supports option database.encrypt for SQL Server CDC sources. #18912
• Supports ingesting data from a partitioned table for PostgreSQL CDC sources. #18456
• Supports option auto.schema.change for PostgreSQL CDC sources to enable replicating Postgres table schema change. #18760
• Requires upstream table name to also be prefixed with database name when creating a SQL Server CDC table. #18868
• Adds JSON encode for file sinks, allowing users to sink JSON files into object storage. #18744
• Supports create_table_if_not_exists option for Iceberg sink connector. #18362
• Supports WebHDFS sinks. #18293
• Removes option bulk_write_max_entries for MongoDB sink and option default_max_batch_rows for DynamoDB sink. Adds options max_batch_item_nums and max_future_send_nums for DynamoDB sink. #17645
• Sets sink decoupling as the default policy for MongoDB, DynamoDB, and Redis sink connectors. #17645
• Supports option routing_column for ElasticSearch sinks, allowing a column to be set as a routing key. #18698
• Supports specifying batching strategy when sinking data in Parquet format. #18472

Installation and deployment

• Adds a CLI argument of --license-key-path for the meta node, enabling a background task to watch and reload license key from the specified file. #18768

Cluster configuration changes

• When visibility_mode is set to all, the latest uncommitted data will be queried, but consistency is no longer guaranteed between the tables. #18230
• Supports SET TIME ZONE INTERVAL '+00:00' HOUR TO MINUTE as equivalent to SET TIME ZONE UTC. #18705
• The etcd metastore is fully deprecated and unsupported. Users previously utilizing etcd metastore must manually migrate to a SQL backend (PostgreSQL, MySQL, or SQLite) to upgrade to v2.1.0

Full Changelog: v2.0.4...v2.1.0

v2.0.4

release v2.0.4

v2.0.3

release v2.0.3

v2.1.0-rc.2

release v2.1.0-rc.2

v2.1.0-rc.1

release v2.1.0-rc.1

v2.0.2

release v2.0.2

v2.0.1

Please be advised that we have withdrawn version v2.0.0 due to a critical bug affecting upgrades of RW clusters from older versions. We sincerely apologize for the inconvenience.

The PRs for the bug fix are:
#18589
#18581

For installation and running instructions, see Get started.

Main changes

SQL features

• Query syntax:
  • Public preview: Supports AS CHANGELOG to convert any stream into an append-only changelog. #17132
  • Supports time travel query to access historical data at a specific point in time. #17665, #17621.
  • Supports CORRESPONDING specification in set operations. #17891
• SQL commands:
  • Breaking change: DECLARE cursor_name SUBSCRIPTION CURSOR is the same as DECLARE cursor_name SUBSCRIPTION CURSOR since now(), which will be consumed from the current time. DECLARE cursor_name SUBSCRIPTION CURSOR FULL will start consuming data from stock. The type of operation has changed to varchar. It is one of Insert, Delete, UpdateInset, or UpdateDelete. #18217
  • Public preview: Supports managing database credentials securely. #17456
  • SHOW CURSORS will return all query cursors in the current session. SHOW SUBSCRIPTION CURSORS will return all subscription cursorS and subscriptions in the current session. #18217
  • Allows ALTER TABLE on tables with generated columns. #17652
  • Allows dropping generated columns from tables created with a schema registry. #17689
  • Supports using scalar functions with list inputs as aggregate functions. #17622
  • Supports altering the backfill rate limit for materialized views. #17911
• SQL functions & operators:
  • Public preview: Supports approx_percentile(). #17814, #17873.
  • Public preview: Supports native map type and related functions for map, and ingesting AVRO MAP type into RisingWave map type. #17986
  • Public preview: Supports scanning a directory of parquet files. #17811
  • Supports pg_index_column_has_property() to query index column properties. #17275
  • Supports continuous timestamp generation in streaming mode. #17371
  • Supports acosd(). #9876
  • Supports function rw_recovery_status() and pg_is_in_recovery() to retrieve the meta node status. #17641
• System catalog:
  • Adds append_only column in RisingWave catalogs rw_tables and rw_materialized_views. #17598
  • Adds RisingWave catalog rw_catalog.rw_secrets. #17726

Connectors

• Public preview: Supports ingesting Avro map type for source connectors. ****#17980
• Public preview: Supports encoding parquet for file source. #17201
• Public preview: Supports batch reading S3 Parquet files. #17625, #17673.
• Supports AWS Glue schema registry with aws.glue.schema_arn parameter. #17605
• Supports creating tables and sources with format upsert encode protobuf. #17624
• Supports ingesting Avro Union type for source connectors. #17485
• Supports reading files compressed in gzip format. #16538
• Adds the option to use a semicolon as the delimiter for CSV encode. #17356
• Uses OpenDAL to connect to S3 object store state backend. #18011
• Public preview: Supports replicating DDL for MySQL CDC source. #17876
• Supports parameter refresh.interval.sec option for S3, GCS, and POSIX sources. #18184
• Supports parameter group.id.prefix for Kafka sources. #18115
• Validates slot name of PostgreSQL CDC sources. #17949
• Supports altering backfill_rate_limit of CDC tables. #17989
• Public preview: Supports sinking data to file systems in parquet format. #17311
• Supports upsert Protobuf type sinks, which requires KEY ENCODE TEXT. #18024
• Adds option jsonb.handling.mode under WITH options for sinks in JSON format. #17693
• Public preview: Supports Azure Blob sinks. #18244
• Public preview: Supports MongoDB sinks. #17102
• Supports Azure Blob file sources. #18295
• Supports glue catalog for iceberg sink and source. #17477
• Adds jdbc.query.timeout for JDBC sinks to set the timeout for queries. #18430
• Changes default Kafka sink message timeout from five seconds to five minutes. #18304
• Adds new parameters retry_on_conflict, batch_size_kb, batch_num_messages, and concurrent_requests for ElasticSearch sink.#17867
• Supports parameter bigquery.retry_times for BigQuery sink. #17237
• Supports parameter bigquery.auto_create_table for BigQuery sink. #17393
• Supports parameter doris.partial_columns for Doris sink. #16821
• Supports ClickHouse sink checkpoint decouple. #17491
• Sets sink decouple as default for all sinks. #18182
• Uses S3's SQS notification to complete the import of data from Snowflake instead of the Snowflake HTTP client. #17627
• Ensures at-least-once delivery semantic and eventual consistency for Kinesis sink. #17983
• Supports backfilling by consuming a fixed snapshot of upstream table and then the upstream data epoch by epoch. #17735

Installation and deployment

• Supports configuring the SQL metastore using username, password, and database separately. #17530
• Supports more seamless scaling-in in Kubernetes deployments. #17802

Cluster configuration changes

• Breaking change: Refactors streaming_rate_limit into source_rate_limit and backfill_rate_limit. #17796
• Breaking change: Adds a default soft and hard limit on actor count per worker parallelism. When the hard limit is reached, streaming workloads will fail. #18383
• Introduces batch.developer.exchange_connection_pool_size and streaming.developer.exchange_connection_pool_size to configure streaming and batch remote exchange between two nodes #17768
• Introduces system parameter license_key used to enable enterprise features. #17396

Fixes

• Deletes related cursors when deleting a subscription. #17232

Full Changelog: v1.10.1...v2.0.1

v1.10.2

release v1.10.2

v2.0.0-rc.1

For installation and running instructions, see Get started.

Main changes

SQL features

• Query syntax:
  • Public preview: Supports AS CHANGELOG to convert any stream into an append-only changelog. #17132
  • Supports time travel query to access historical data at a specific point in time. #17665, #17621.
  • Supports CORRESPONDING specification in set operations. #17891
• SQL commands:
  • Breaking change: DECLARE cursor_name SUBSCRIPTION CURSOR is the same as DECLARE cursor_name SUBSCRIPTION CURSOR since now(), which will be consumed from the current time. DECLARE cursor_name SUBSCRIPTION CURSOR FULL will start consuming data from stock. The type of operation has changed to varchar. It is one of Insert, Delete, UpdateInset, or UpdateDelete. #18217
  • Public preview: Supports managing database credentials securely. #17456
  • SHOW CURSORS will return all query cursors in the current session. SHOW SUBSCRIPTION CURSORS will return all subscription cursorS and subscriptions in the current session. #18217
  • Allows ALTER TABLE on tables with generated columns. #17652
  • Allows dropping generated columns from tables created with a schema registry. #17689
  • Supports using scalar functions with list inputs as aggregate functions. #17622
  • Supports altering the backfill rate limit for materialized views. #17911
• SQL functions & operators:
  • Public preview: Supports approx_percentile(). #17814, #17873.
  • Public preview: Supports native map type and related functions for map, and ingesting AVRO MAP type into RisingWave map type. #17986
  • Public preview: Supports scanning a directory of parquet files. #17811
  • Supports pg_index_column_has_property() to query index column properties. #17275
  • Supports continuous timestamp generation in streaming mode. #17371
  • Supports acosd(). #9876
  • Supports function rw_recovery_status() and pg_is_in_recovery() to retrieve the meta node status. #17641
• System catalog:
  • Adds append_only column in RisingWave catalogs rw_tables and rw_materialized_views. #17598
  • Adds RisingWave catalog rw_catalog.rw_secrets. #17726

Connectors

• Public preview: Supports ingesting Avro map type for source connectors. ****#17980
• Public preview: Supports encoding parquet for file source. #17201
• Public preview: Supports batch reading S3 Parquet files. #17625, #17673.
• Supports AWS Glue schema registry with aws.glue.schema_arn parameter. #17605
• Supports creating tables and sources with format upsert encode protobuf. #17624
• Supports ingesting Avro Union type for source connectors. #17485
• Supports reading files compressed in gzip format. #16538
• Adds the option to use a semicolon as the delimiter for CSV encode. #17356
• Uses OpenDAL to connect to S3 object store state backend. #18011
• Public preview: Supports replicating DDL for MySQL CDC source. #17876
• Supports parameter refresh.interval.sec option for S3, GCS, and POSIX sources. #18184
• Supports parameter group.id.prefix for Kafka sources. #18115
• Validates slot name of PostgreSQL CDC sources. #17949
• Supports altering backfill_rate_limit of CDC tables. #17989
• Public preview: Supports sinking data to file systems in parquet format. #17311
• Supports upsert Protobuf type sinks, which requires KEY ENCODE TEXT. #18024
• Adds option jsonb.handling.mode under WITH options for sinks in JSON format. #17693
• Public preview: Supports Azure Blob sinks. #18244
• Public preview: Supports MongoDB sinks. #17102
• #18295
• Supports glue catalog for iceberg sink and source. #17477
• #18304
• Supports parameter bigquery.retry_times for BigQuery sink connector. #17237
• Supports parameter bigquery.auto_create_table for BigQuery sink. #17393
• Supports parameter doris.partial_columns for Doris sink connector. #16821
• Supports ClickHouse sink checkpoint decouple. #17491
• Sets sink decouple as default for all sinks. #18182
• Uses S3's SQS notification to complete the import of data from Snowflake instead of the Snowflake HTTP client. #17627
• Ensures at-least-once delivery semantic and eventual consistency for Kinesis sink. #17983
• Supports backfilling by consuming a fixed snapshot of upstream table and then the upstream data epoch by epoch. #17735

Installation and deployment

• Supports configuring the SQL metastore using username, password, and database separately. #17530
• Supports more seamless scaling-in in Kubernetes deployments. #17802

Cluster configuration changes

• Breaking change: Refactors streaming_rate_limit into source_rate_limit and backfill_rate_limit. #17796
• Introduces batch.developer.exchange_connection_pool_size and streaming.developer.exchange_connection_pool_size to configure streaming and batch remote exchange between two nodes #17768
• Introduces system parameter license_key used to enable enterprise features. #17396

Fixes

• Deletes related cursors when deleting a subscription. fix(subscription): fix drop subscription not clear cursor

Full Changelog: v1.10.1...v2.0.0-rc.1