Skip to content

Commit 1fef980

Browse files
authored
[website] update roadmap to account latest requirements (#1214)
* update roadmap to account latest requirements * Address Jark comment * minor fix * address giannis comments
1 parent 2b1cce5 commit 1fef980

File tree

1 file changed

+19
-39
lines changed

1 file changed

+19
-39
lines changed

website/src/pages/roadmap.md

Lines changed: 19 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -17,56 +17,36 @@
1717
-->
1818

1919
# Fluss Roadmap
20-
21-
This roadmap means to provide users and contributors with a high-level summary of ongoing efforts in the Fluss community.
22-
The roadmap contains both efforts working in process as well as completed efforts, so that users may get a better impression of the overall status and direction of those developments.
23-
20+
This roadmap means to provide users and contributors with a high-level summary of ongoing efforts in the Fluss community. The roadmap contains both efforts working in process as well as completed efforts, so that users may get a better impression of the overall status and direction of those developments.
2421
## Kafka Protocol Compatibility
25-
2622
Fluss will support the Kafka network protocol to enable users to use Fluss as a drop-in replacement for Kafka. This will allow users to leverage Fluss's real-time storage capabilities while maintaining compatibility with existing Kafka applications.
27-
2823
## Flink Integration
29-
30-
Fluss will provide deep integration with Apache Flink, enabling users a single engine experience for building real-time analytics applications.
31-
The integration will include:
32-
- Support for Flink **DataStream API** to read/write data from/to Fluss.
33-
- Support new [Delta Join](https://cwiki.apache.org/confluence/display/FLINK/FLIP-486%3A+Introduce+A+New+DeltaJoin) to address the pain-points of Stream-Stream Join.
34-
- More pushdown optimizations: Filter Pushdown ([#197](https://github.com/alibaba/fluss/issues/197)), Partition Pruning ([#196](https://github.com/alibaba/fluss/issues/196)), Aggregation Pushdown, etc.
24+
Fluss will provide deep integration with Apache Flink, enabling users a single engine experience for building real-time analytics applications. The integration will include:
25+
- Upgrade Flink version to 2.x
26+
- Support new Delta Join to address the pain-points of Stream-Stream Join.
27+
- More pushdown optimizations: Filter Pushdown ([#197](https://github.com/alibaba/fluss/issues/197)), Aggregation Pushdown, etc.
3528
- Upgrade the Rule-Based Optimization into Cost-Based Optimization in Flink SQL streaming planner with leveraging statistics in Fluss tables.
36-
37-
3829
## Streaming Lakehouse
39-
40-
- Support for Iceberg ([#102](https://github.com/alibaba/fluss/issues/102)) as Lakehouse Storage. And DeltaLake, Hudi as well.
30+
- Support for Iceberg ([#452](https://github.com/alibaba/fluss/issues/452)) as Lakehouse Storage. And DeltaLake, Hudi as well.
4131
- Support Union Read for Spark, Trino, StarRocks.
42-
- Avoid data shuffle in compaction service to directly compact Arrow files of Fluss into Parquet files of data lakes ([#107](https://github.com/alibaba/fluss/issues/107)).
43-
44-
## ZooKeeper Removal
45-
46-
Fluss currently utilizes **ZooKeeper** for cluster coordination, metadata storage, and cluster configuration management.
47-
In upcoming releases, **ZooKeeper will be replaced** by **KvStore** for metadata storage and **Raft** for cluster coordination and ensuring consistency.
48-
This transition aims to streamline operations and enhance system reliability.
49-
32+
- Support for Lance ([#1155](https://github.com/alibaba/fluss/issues/1155)) as Lakehouse Storage to enable integration with AI/ML workflows for multi-modal data processing.
33+
## Spark Integration
34+
- Support for Spark connector ([#155](https://github.com/alibaba/fluss/issues/155)) to enable seamless data processing and analytics workflows.
35+
## Python Client
36+
- Support Python SDK to connect with Python ecosystems, including PyArrow, Pandas, Lance, and DuckDB.
5037
## Storage Engine
51-
5238
- Support for complex data types: Array ([#168](https://github.com/alibaba/fluss/issues/168)), Map ([#169](https://github.com/alibaba/fluss/issues/169)), Struct ([#170](https://github.com/alibaba/fluss/issues/170)), Variant/JSON.
5339
- Support for schema evolution.
54-
- Support for secondary index for Delta Join with Flink (~~[#65](https://github.com/alibaba/fluss/issues/65)~~).
55-
- Support for buckets rescale.
40+
## ZooKeeper Removal
41+
Fluss currently utilizes ZooKeeper for cluster coordination, metadata storage, and cluster configuration management. In upcoming releases, ZooKeeper will be replaced by KvStore for metadata storage and Raft for cluster coordination and ensuring consistency. This transition aims to streamline operations and enhance system reliability.
5642

5743
## Zero Disks
58-
59-
Fluss currently utilizes a tiered storage architecture to significantly reduce storage costs and operational complexities.
60-
However, the Fluss community is actively investing in the Zero Disk architecture,
61-
which aims to completely replace local disks with S3 storage. This transition will enable Fluss to achieve a
62-
serverless, stateless, and elastic design, significantly minimizing operational overhead while eliminating inter-zone networking costs.
63-
44+
Fluss currently utilizes a tiered storage architecture to significantly reduce storage costs and operational complexities. However, the Fluss community is actively investing in the Zero Disk architecture, which aims to completely replace local disks with S3 storage. This transition will enable Fluss to achieve a serverless, stateless, and elastic design, significantly minimizing operational overhead while eliminating inter-zone networking costs.
6445
## Maintenance
65-
66-
- Re-balance Cluster
67-
- Gray Upgrade
68-
46+
- Re-balance Cluster: Automatic cluster rebalancing capabilities for optimal resource distribution.
47+
- Gray Upgrade: Rolling upgrade support enabling zero-downtime system updates.
6948
## Miscellaneous
49+
- Upgrade programming language to Java 11.
50+
- Support for more connectors: Trino, DuckDB, etc.
7051

71-
- Upgrade programing language to Java 17 ([#203](https://github.com/alibaba/fluss/issues/203)).
72-
- Support for more connectors: Spark ([#155](https://github.com/alibaba/fluss/issues/155)), Presto, DuckDB, etc.
52+
*This roadmap is subject to change based on community feedback, technical discoveries, and evolving requirements. For the most up-to-date information, please refer to the GitHub milestone boards and project issues.*

0 commit comments

Comments
 (0)