Merge main into sync PR 79503

Signed-off-by: luohaha <18810541851@163.com>

Author: luohaha

conf/be.conf

@@ -58,9 +58,8 @@ starlet_port = 9070
 # eg:
 # JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"
 # used for query iceberg metadata and Java UDF for JDK17
-# -Djava.security.manager=allow is required on JDK 18-23 (JEP 411) for the UDF security
-# manager that UDFClassLoader installs when java.security.policy is set. It is a no-op on
-# JDK 17. Drop it before moving to JDK 24 or later: JEP 486 makes the JVM refuse to start
-# on any value other than 'disallow'. bin/start_backend.sh appends the flag automatically
-# on JDK 18-23 when it is unset.
-JAVA_OPTS="--add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED -Djava.security.manager=allow"
+# bin/start_backend.sh appends -Djava.security.manager=allow on JDK 18-23 (JEP 411) for the
+# UDF security manager that UDFClassLoader installs when java.security.policy is set. Do not
+# put the flag here: a value set in JAVA_OPTS always wins over the script, and JDK 24+
+# (JEP 486) refuses to start on any value other than 'disallow'.
+JAVA_OPTS="--add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED"

conf/cn.conf

@@ -57,9 +57,8 @@ starlet_port = 9070
 # eg:
 # JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"
 # used for query iceberg metadata and Java UDF for JDK17
-# -Djava.security.manager=allow is required on JDK 18-23 (JEP 411) for the UDF security
-# manager that UDFClassLoader installs when java.security.policy is set. It is a no-op on
-# JDK 17. Drop it before moving to JDK 24 or later: JEP 486 makes the JVM refuse to start
-# on any value other than 'disallow'. bin/start_backend.sh appends the flag automatically
-# on JDK 18-23 when it is unset, so uncommenting this line is only needed to override it.
-# JAVA_OPTS="--add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED -Djava.security.manager=allow"
+# bin/start_backend.sh appends -Djava.security.manager=allow on JDK 18-23 (JEP 411) for the
+# UDF security manager that UDFClassLoader installs when java.security.policy is set. Do not
+# put the flag here: a value set in JAVA_OPTS always wins over the script, and JDK 24+
+# (JEP 486) refuses to start on any value other than 'disallow'.
+# JAVA_OPTS="--add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED"

conf/fe.conf

@@ -24,11 +24,11 @@
 LOG_DIR = ${STARROCKS_HOME}/log
 
 DATE = "$(date +%Y%m%d-%H%M%S)"
-# -Djava.security.manager=allow is required on JDK 18-23 (JEP 411) to install the UDF
-# security manager while analyzing CREATE FUNCTION. It is a no-op on JDK 17. Drop it before
-# moving to JDK 24 or later: JEP 486 makes the JVM refuse to start on any value other than
-# 'disallow'. bin/start_fe.sh appends the flag automatically on JDK 18-23 when it is unset.
-JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time -XX:ErrorFile=${LOG_DIR}/hs_err_pid%p.log -Djava.security.policy=${STARROCKS_HOME}/conf/udf_security.policy -Djava.security.manager=allow"
+# bin/start_fe.sh appends -Djava.security.manager=allow on JDK 18-23 (JEP 411), which the FE
+# needs to install the UDF security manager while analyzing CREATE FUNCTION. Do not put the
+# flag here: a value set in JAVA_OPTS always wins over the script, and JDK 24+ (JEP 486)
+# refuses to start on any value other than 'disallow'.
+JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time -XX:ErrorFile=${LOG_DIR}/hs_err_pid%p.log -Djava.security.policy=${STARROCKS_HOME}/conf/udf_security.policy"
 
 ##
 ## the lowercase properties are read by main program.

docs/en/_assets/commonMarkdown/Edition_Specific_Flink_StarRocks_Connection.mdx

@@ -0,0 +1,8 @@
+Set the StarRocks connection options to addresses of the FE:
+
+- `jdbc-url`: `jdbc:mysql://<fe_host>:<fe_query_port>`. The query port defaults to `9030`.
+- `load-url`: `<fe_host>:<fe_http_port>`. The HTTP port defaults to `8030`.
+
+For `<fe_host>`, use an address that stays the same when an FE restarts or is replaced, such as a DNS name or a load balancer in front of the FEs, not the IP address of an FE. If the address stops leading to an FE, the Flink job can no longer load data.
+
+The Flink TaskManagers must be able to reach these FE ports, and also the HTTP port (default `8040`) of every BE or CN, because the FE redirects each load request to a BE or CN.

docs/en/integrations/loading_tools/SMT.md

@@ -6,7 +6,7 @@ description: "StarRocks Migration Tool (SMT) is a data migration tool provided b
 
 # StarRocks Migration Tool (SMT) 
 
-StarRocks Migration Tool (SMT) is a data migration tool provided by StarRocks to load data from source databases through Flink into StarRocks. SMT mainly can:
+StarRocks Migration Tool (SMT) is a command-line tool that generates the SQL you need to load data from a source database into StarRocks through Flink. SMT mainly can:
 - Generate statements to create tables in StarRocks based on information of the source database and the target StarRocks cluster.
 - Generate SQL statements that can be executed in Flink's SQL client to submit Flink jobs for synchronizing data, which simplifies full or incremental data synchronization in the pipeline. Currently, SMT supports the following source databases:
 
@@ -22,6 +22,22 @@ StarRocks Migration Tool (SMT) is a data migration tool provided by StarRocks to
 
 Download link: https://cdn-thirdparty.starrocks.com/smt.tar.gz?r=2
 
+## What SMT does and does not do
+
+SMT generates SQL files and then exits. It reads the table definitions in your source database and writes two kinds of files to the `result` directory:
+
+- `starrocks-create.*.sql`: CREATE TABLE statements, which you run in StarRocks.
+- `flink-create.*.sql`: Flink SQL that defines a source table and a sink table and submits an `INSERT INTO ... SELECT` job, which you run in the Flink SQL client.
+
+SMT does not start, run, or connect to Flink, and it does not move any data. You still install and run a Flink cluster with the Flink CDC connector for your source database and the [Flink connector for StarRocks](../../loading/Flink-connector-starrocks.md), and you run the generated files yourself.
+
+### Synchronize data without SMT
+
+SMT is optional. Instead, you can:
+
+- Use a [Flink CDC pipeline](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/data-pipeline/). A pipeline is defined in a single YAML file, creates the StarRocks tables itself, and runs the synchronization. See [Realtime synchronization from PostgreSQL](../../loading/Flink_cdc_postgres.md), or for MySQL, [Streaming ELT from MySQL to StarRocks](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.6/docs/get-started/quickstart-for-1.20/mysql-to-starrocks/) in the Flink CDC documentation.
+- Create the StarRocks tables yourself, and write the Flink SQL with the [Flink connector for StarRocks](../../loading/Flink-connector-starrocks.md) as the sink. This works for any source that Flink can read.
+
 ## Steps to use SMT
 
 The generally involved steps are as follows: 
@@ -225,6 +241,12 @@ As shown in the image, SMT can automatically generate CREATE TABLE statements of
 
 ## Synchronize PostgreSQL to StarRocks
 
+:::tip
+
+With Flink CDC 3.5 or later, a Flink CDC pipeline can synchronize PostgreSQL to StarRocks without SMT, and creates the StarRocks tables itself. For a step-by-step guide, see [Realtime synchronization from PostgreSQL](../../loading/Flink_cdc_postgres.md).
+
+:::
+
 ### Introduction
 
 Flink CDC connector and SMT can synchronize data from PostgreSQL within subsecond.
@@ -334,9 +356,9 @@ Flink CDC connector reads the WAL of PostgreSQL and Flink-connector-starrocks wr
 - How to enable PostgreSQL WAL?
 
    ```Bash
-   # Open connection permissions
-   echo "host all all 0.0.0.0/32 trust" >> pg_hba.conf
-   echo "host replication all 0.0.0.0/32 trust" >> pg_hba.conf
+   # Allow the Flink hosts to connect. Replace 10.0.0.0/24 with their address range.
+   echo "host all all 10.0.0.0/24 scram-sha-256" >> pg_hba.conf
+   echo "host replication all 10.0.0.0/24 scram-sha-256" >> pg_hba.conf
    # Enable wal logical replication
    echo "wal_level = logical" >> postgresql.conf
    echo "max_wal_senders = 2" >> postgresql.conf

docs/en/loading/Flink-connector-starrocks.md

@@ -6,6 +6,8 @@ description: "How to use the StarRocks Flink connector to continuously load data
 
 # Continuously load data from Apache Flink®
 
+import FlinkStarRocksConnection from '../_assets/commonMarkdown/Edition_Specific_Flink_StarRocks_Connection.mdx'
+
 StarRocks provides a self-developed connector named StarRocks Connector for Apache Flink® (Flink connector for short) to help you load data into a StarRocks table by using Flink. The basic principle is to accumulate the data and then load it all at a time into StarRocks through [STREAM LOAD](../sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md).
 
 The Flink connector supports DataStream API, Table API & SQL, and Python API. It has a higher and more stable performance than [flink-connector-jdbc](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/jdbc/) provided by Apache Flink®.
@@ -90,6 +92,10 @@ In your Maven project's `pom.xml` file, add the Flink connector as a dependency
 >
 > The name of Flink connector which is not formally released contains the `SNAPSHOT` suffix.
 
+## Connect to StarRocks
+
+<FlinkStarRocksConnection />
+
 ## Options
 
 ### General Options

docs/en/loading/Flink_cdc_load.md

@@ -12,6 +12,7 @@ description: "How to use Flink CDC to capture MySQL change data in real time and
 # Realtime synchronization from MySQL
 
 import InsertPrivNote from '../_assets/commonMarkdown/insertPrivNote.mdx'
+import FlinkStarRocksConnection from '../_assets/commonMarkdown/Edition_Specific_Flink_StarRocks_Connection.mdx'
 
 StarRocks supports multiple methods to synchronize data from MySQL to StarRocks in real time, delivering low latency real-time analytics of massive data.
 
@@ -191,6 +192,12 @@ To synchronize data from MySQL in real time, the system needs to read data from
     1 row in set (0.00 sec)
     ```
 
+### Connect to StarRocks
+
+The SMT configuration file sets these addresses in `flink.starrocks.jdbc-url` and `flink.starrocks.load-url`.
+
+<FlinkStarRocksConnection />
+
 ## Synchronize database & table schema
 
 1. Edit the SMT configuration file.