Skip to content

Commit a890e2d

Browse files
edcloledclol
authored
Merge branch 'apache:dev' into dev
2 parents 7012bdd + 8bec503 commit a890e2d

File tree

112 files changed

+284
-13505
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

112 files changed

+284
-13505
lines changed

‎.github/workflows/docs.yml

+7-1
Original file line numberDiff line numberDiff line change
@@ -55,12 +55,18 @@ jobs:
5555
timeout-minutes: 30
5656
steps:
5757
- uses: actions/checkout@v4
58+
with:
59+
submodules: true
60+
- name: Collect Workflow Telemetry
61+
uses: ./.github/actions/workflow-telemetry-action
62+
with:
63+
comment_on_pr: false
5864
- run: sudo npm install -g [email protected]
5965
- run: sudo apt install plocate -y
6066
# NOTE: Change command from `find . -name "*.md"` to `find . -not -path "*/node_modules/*" -not -path "*/.tox/*" -name "*.md"`
6167
# if you want to run check locally
6268
- run: |
63-
for file in $(locate "$PWD*/*.md" | grep -v ./deploy/terraform/aws/README.md); do
69+
for file in $(locate "$PWD*/*.md" | grep -v ./deploy/terraform/aws/README.md | grep -v ./.github); do
6470
markdown-link-check -c .dlc.json -q "$file" &
6571
done
6672
wait

‎.github/workflows/unit-test.yml

+4
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,10 @@ jobs:
6262
- uses: actions/checkout@v4
6363
with:
6464
submodules: true
65+
- name: Collect Workflow Telemetry
66+
uses: ./.github/actions/workflow-telemetry-action
67+
with:
68+
comment_on_pr: false
6569
- name: Sanity Check
6670
uses: ./.github/actions/sanity-check
6771
with:

‎docs/docs/en/guide/parameter/context.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ DolphinScheduler allows parameter transfer between tasks. Currently, transfer di
1414
* [SQL](../task/sql.md)
1515
* [Procedure](../task/stored-procedure.md)
1616
* [Python](../task/python.md)
17-
* [SubWorkflow](../task/sub-workflow)
17+
* [SubWorkflow](../task/sub-workflow.md)
1818
* [Kubernetes](../task/kubernetes.md)
1919

2020
When defining an upstream node, if there is a need to transmit the result of that node to a dependency related downstream node. You need to set an `OUT` direction parameter to [Custom Parameters] of the [Current Node Settings]. If it is a sub-workflow node, there is no need to set a parameter in [Current Node Settings], but an `OUT` direction parameter needs to be set in the workflow definition of the sub-workflow.

‎docs/docs/zh/guide/parameter/context.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ DolphinScheduler 允许在任务间进行参数传递，目前传递方向仅支
1414
* [SQL](../task/sql.md)
1515
* [Procedure](../task/stored-procedure.md)
1616
* [Python](../task/python.md)
17-
* [SubWorkflow](../task/sub-workflow)
17+
* [SubWorkflow](../task/sub-workflow.md)
1818
* [Kubernetes](../task/kubernetes.md)
1919

2020
当定义上游节点时，如果有需要将该节点的结果传递给有依赖关系的下游节点，需要在【当前节点设置】的【自定义参数】设置一个方向是 OUT 的变量。如果是 SubWorkflow 节点无需在【当前节点设置】中设置变量，需要在子流程的工作流定义中设置一个方向是 OUT 的变量。

‎dolphinscheduler-alert/dolphinscheduler-alert-server/pom.xml

+7-1
Original file line numberDiff line numberDiff line change
@@ -28,15 +28,20 @@
2828
<name>${project.artifactId}</name>
2929
<dependencies>
3030
<!-- dolphinscheduler -->
31-
3231
<dependency>
3332
<groupId>org.apache.dolphinscheduler</groupId>
3433
<artifactId>dolphinscheduler-meter</artifactId>
3534
</dependency>
3635

36+
<dependency>
37+
<groupId>org.apache.dolphinscheduler</groupId>
38+
<artifactId>dolphinscheduler-alert-api</artifactId>
39+
</dependency>
40+
3741
<dependency>
3842
<groupId>org.apache.dolphinscheduler</groupId>
3943
<artifactId>dolphinscheduler-alert-all</artifactId>
44+
<scope>provided</scope>
4045
</dependency>
4146

4247
<dependency>
@@ -58,6 +63,7 @@
5863
<groupId>org.apache.dolphinscheduler</groupId>
5964
<artifactId>dolphinscheduler-common</artifactId>
6065
</dependency>
66+
<!-- dolphinscheduler -->
6167

6268
<dependency>
6369
<groupId>com.google.guava</groupId>

‎dolphinscheduler-api-test/pom.xml

+7
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@
4848
<jackson.version>2.13.2</jackson.version>
4949
<checkstyle.version>3.1.2</checkstyle.version>
5050
<junit-pioneer.version>2.2.0</junit-pioneer.version>
51+
<http.components.version>4.5.13</http.components.version>
5152
</properties>
5253

5354
<dependencies>
@@ -74,6 +75,12 @@
7475
<scope>test</scope>
7576
</dependency>
7677

78+
<dependency>
79+
<groupId>org.apache.httpcomponents</groupId>
80+
<artifactId>httpmime</artifactId>
81+
<version>${http.components.version}</version>
82+
<scope>test</scope>
83+
</dependency>
7784

7885
<dependency>
7986
<groupId>org.testcontainers</groupId>

‎dolphinscheduler-api/pom.xml

+45-6
Original file line numberDiff line numberDiff line change
@@ -61,32 +61,60 @@
6161
<artifactId>dolphinscheduler-meter</artifactId>
6262
</dependency>
6363

64+
<dependency>
65+
<groupId>org.apache.dolphinscheduler</groupId>
66+
<artifactId>dolphinscheduler-common</artifactId>
67+
<version>${project.version}</version>
68+
</dependency>
69+
70+
<dependency>
71+
<groupId>org.apache.dolphinscheduler</groupId>
72+
<artifactId>dolphinscheduler-datasource-api</artifactId>
73+
<version>${project.version}</version>
74+
</dependency>
75+
6476
<dependency>
6577
<groupId>org.apache.dolphinscheduler</groupId>
6678
<artifactId>dolphinscheduler-datasource-all</artifactId>
6779
<version>${project.version}</version>
80+
<scope>provided</scope>
81+
</dependency>
82+
83+
<dependency>
84+
<groupId>org.apache.dolphinscheduler</groupId>
85+
<artifactId>dolphinscheduler-task-api</artifactId>
6886
</dependency>
6987

7088
<dependency>
7189
<groupId>org.apache.dolphinscheduler</groupId>
7290
<artifactId>dolphinscheduler-task-all</artifactId>
91+
<scope>provided</scope>
7392
</dependency>
7493

7594
<dependency>
7695
<groupId>org.apache.dolphinscheduler</groupId>
7796
<artifactId>dolphinscheduler-ui</artifactId>
7897
</dependency>
98+
7999
<dependency>
80100
<groupId>org.apache.dolphinscheduler</groupId>
81101
<artifactId>dolphinscheduler-registry-all</artifactId>
82102
</dependency>
103+
83104
<dependency>
84105
<groupId>org.apache.dolphinscheduler</groupId>
85106
<artifactId>dolphinscheduler-scheduler-all</artifactId>
86107
</dependency>
108+
109+
<dependency>
110+
<groupId>org.apache.dolphinscheduler</groupId>
111+
<artifactId>dolphinscheduler-storage-api</artifactId>
112+
</dependency>
113+
87114
<dependency>
88115
<groupId>org.apache.dolphinscheduler</groupId>
89116
<artifactId>dolphinscheduler-storage-all</artifactId>
117+
<scope>provided</scope>
90118
</dependency>
91119

92120
<dependency>
@@ -100,13 +128,16 @@
100128
</dependency>
101129

102130
<dependency>
103-
<groupId>org.codehaus.janino</groupId>
104-
<artifactId>janino</artifactId>
131+
<groupId>org.apache.dolphinscheduler</groupId>
132+
<artifactId>dolphinscheduler-extract-alert</artifactId>
105133
</dependency>
134+
<!-- dolphinscheduler -->
135+
106136
<dependency>
107-
<groupId>com.amazonaws</groupId>
108-
<artifactId>aws-java-sdk-s3</artifactId>
137+
<groupId>org.codehaus.janino</groupId>
138+
<artifactId>janino</artifactId>
109139
</dependency>
140+
110141
<!--springboot-->
111142
<dependency>
112143
<groupId>org.springframework.boot</groupId>
@@ -228,10 +259,18 @@
228259
<groupId>com.azure.resourcemanager</groupId>
229260
<artifactId>azure-resourcemanager-datafactory</artifactId>
230261
</dependency>
262+
231263
<dependency>
232-
<groupId>org.apache.dolphinscheduler</groupId>
233-
<artifactId>dolphinscheduler-extract-alert</artifactId>
264+
<groupId>com.azure</groupId>
265+
<artifactId>azure-identity</artifactId>
266+
<exclusions>
267+
<exclusion>
268+
<groupId>net.java.dev.jna</groupId>
269+
<artifactId>jna-platform</artifactId>
270+
</exclusion>
271+
</exclusions>
234272
</dependency>
273+
235274
</dependencies>
236275
<build>
237276
<testResources>

‎dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/AuditServiceImpl.java

+1-2
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,6 @@
2525
import org.apache.dolphinscheduler.dao.entity.AuditLog;
2626
import org.apache.dolphinscheduler.dao.mapper.AuditLogMapper;
2727

28-
import org.apache.parquet.Strings;
29-
3028
import java.util.ArrayList;
3129
import java.util.Arrays;
3230
import java.util.Date;
@@ -40,6 +38,7 @@
4038

4139
import com.baomidou.mybatisplus.core.metadata.IPage;
4240
import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
41+
import com.google.common.base.Strings;
4342

4443
@Service
4544
@Slf4j

‎dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/TenantServiceImpl.java

+4-2
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@
3030
import org.apache.dolphinscheduler.api.utils.RegexUtils;
3131
import org.apache.dolphinscheduler.common.constants.Constants;
3232
import org.apache.dolphinscheduler.common.enums.AuthorizationType;
33+
import org.apache.dolphinscheduler.common.enums.WorkflowExecutionStatus;
3334
import org.apache.dolphinscheduler.dao.entity.Queue;
3435
import org.apache.dolphinscheduler.dao.entity.Schedule;
3536
import org.apache.dolphinscheduler.dao.entity.Tenant;
@@ -255,8 +256,9 @@ public void deleteTenantById(User loginUser, int id) throws Exception {
255256
}
256257

257258
private List<WorkflowInstance> getWorkflowInstancesByTenant(Tenant tenant) {
258-
return workflowInstanceMapper.queryByTenantCodeAndStatus(tenant.getTenantCode(),
259-
org.apache.dolphinscheduler.service.utils.Constants.NOT_TERMINATED_STATES);
259+
return workflowInstanceMapper.queryByTenantCodeAndStatus(
260+
tenant.getTenantCode(),
261+
WorkflowExecutionStatus.getNotTerminalStatus());
260262
}
261263

262264
/**

‎dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/WorkerGroupServiceImpl.java

+5-4
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
import org.apache.dolphinscheduler.common.constants.Constants;
2828
import org.apache.dolphinscheduler.common.enums.AuthorizationType;
2929
import org.apache.dolphinscheduler.common.enums.UserType;
30+
import org.apache.dolphinscheduler.common.enums.WorkflowExecutionStatus;
3031
import org.apache.dolphinscheduler.common.utils.JSONUtils;
3132
import org.apache.dolphinscheduler.dao.entity.EnvironmentWorkerGroupRelation;
3233
import org.apache.dolphinscheduler.dao.entity.Schedule;
@@ -66,7 +67,7 @@
6667
import org.springframework.transaction.annotation.Transactional;
6768

6869
import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
69-
import com.facebook.presto.jdbc.internal.guava.base.Strings;
70+
import com.google.common.base.Strings;
7071

7172
@Service
7273
@Slf4j
@@ -391,9 +392,9 @@ public Map<String, Object> deleteWorkerGroupById(User loginUser, Integer id) {
391392
putMsg(result, Status.DELETE_WORKER_GROUP_NOT_EXIST);
392393
return result;
393394
}
394-
List<WorkflowInstance> workflowInstances = workflowInstanceMapper
395-
.queryByWorkerGroupNameAndStatus(workerGroup.getName(),
396-
org.apache.dolphinscheduler.service.utils.Constants.NOT_TERMINATED_STATES);
395+
List<WorkflowInstance> workflowInstances = workflowInstanceMapper.queryByWorkerGroupNameAndStatus(
396+
workerGroup.getName(),
397+
WorkflowExecutionStatus.getNotTerminalStatus());
397398
if (CollectionUtils.isNotEmpty(workflowInstances)) {
398399
List<Integer> workflowInstanceIds =
399400
workflowInstances.stream().map(WorkflowInstance::getId).collect(Collectors.toList());

‎dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/WorkflowDefinitionServiceImpl.java

+3-5
Original file line numberDiff line numberDiff line change
@@ -118,7 +118,6 @@
118118
import org.apache.dolphinscheduler.plugin.task.api.parameters.DependentParameters;
119119
import org.apache.dolphinscheduler.plugin.task.api.parameters.SqlParameters;
120120
import org.apache.dolphinscheduler.plugin.task.api.utils.TaskTypeUtils;
121-
import org.apache.dolphinscheduler.plugin.task.sql.SqlTaskChannelFactory;
122121
import org.apache.dolphinscheduler.service.model.TaskNode;
123122
import org.apache.dolphinscheduler.service.process.ProcessService;
124123

@@ -1061,9 +1060,8 @@ private void workflowDefinitionUsedInOtherTaskValid(WorkflowDefinition workflowD
10611060
}
10621061

10631062
// check workflow instances is already running
1064-
List<WorkflowInstance> workflowInstances = workflowInstanceService
1065-
.queryByWorkflowDefinitionCodeAndStatus(workflowDefinition.getCode(),
1066-
org.apache.dolphinscheduler.service.utils.Constants.NOT_TERMINATED_STATES);
1063+
List<WorkflowInstance> workflowInstances = workflowInstanceService.queryByWorkflowDefinitionCodeAndStatus(
1064+
workflowDefinition.getCode(), WorkflowExecutionStatus.getNotTerminalStatus());
10671065
if (CollectionUtils.isNotEmpty(workflowInstances)) {
10681066
throw new ServiceException(Status.DELETE_WORKFLOW_DEFINITION_EXECUTING_FAIL, workflowInstances.size());
10691067
}
@@ -1438,7 +1436,7 @@ private TaskDefinitionLog buildNormalSqlTaskDefinition(String taskName, DataSour
14381436
sqlParameters.setLocalParams(Collections.emptyList());
14391437
taskDefinition.setTaskParams(JSONUtils.toJsonString(sqlParameters));
14401438
taskDefinition.setCode(CodeGenerateUtils.genCode());
1441-
taskDefinition.setTaskType(SqlTaskChannelFactory.NAME);
1439+
taskDefinition.setTaskType("SQL");
14421440
taskDefinition.setFailRetryTimes(0);
14431441
taskDefinition.setFailRetryInterval(0);
14441442
taskDefinition.setTimeoutFlag(TimeoutFlag.CLOSE);

‎dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/WorkerGroupControllerTest.java

+2-1
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
2525

2626
import org.apache.dolphinscheduler.api.utils.Result;
27+
import org.apache.dolphinscheduler.common.enums.WorkflowExecutionStatus;
2728
import org.apache.dolphinscheduler.common.utils.JSONUtils;
2829
import org.apache.dolphinscheduler.dao.entity.WorkerGroup;
2930
import org.apache.dolphinscheduler.dao.mapper.WorkerGroupMapper;
@@ -134,7 +135,7 @@ public void testDeleteById() throws Exception {
134135
workerGroup.setName("测试");
135136
Mockito.when(workerGroupMapper.selectById(12)).thenReturn(workerGroup);
136137
Mockito.when(workflowInstanceMapper.queryByWorkerGroupNameAndStatus("测试",
137-
org.apache.dolphinscheduler.service.utils.Constants.NOT_TERMINATED_STATES))
138+
WorkflowExecutionStatus.getNotTerminalStatus()))
138139
.thenReturn(null);
139140
Mockito.when(workerGroupMapper.deleteById(12)).thenReturn(1);
140141
Mockito.when(workflowInstanceMapper.updateWorkflowInstanceByWorkerGroupName("测试", "")).thenReturn(1);

‎dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/TenantServiceTest.java

+2-1
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232
import org.apache.dolphinscheduler.api.utils.PageInfo;
3333
import org.apache.dolphinscheduler.common.enums.AuthorizationType;
3434
import org.apache.dolphinscheduler.common.enums.UserType;
35+
import org.apache.dolphinscheduler.common.enums.WorkflowExecutionStatus;
3536
import org.apache.dolphinscheduler.dao.entity.Queue;
3637
import org.apache.dolphinscheduler.dao.entity.Schedule;
3738
import org.apache.dolphinscheduler.dao.entity.Tenant;
@@ -191,7 +192,7 @@ public void testDeleteTenantById() {
191192
baseServiceLogger)).thenReturn(true);
192193
when(tenantMapper.queryById(1)).thenReturn(getTenant());
193194
when(workflowInstanceMapper.queryByTenantCodeAndStatus(tenantCode,
194-
org.apache.dolphinscheduler.service.utils.Constants.NOT_TERMINATED_STATES))
195+
WorkflowExecutionStatus.getNotTerminalStatus()))
195196
.thenReturn(getInstanceList());
196197
when(scheduleMapper.queryScheduleListByTenant(tenantCode)).thenReturn(getScheduleList());
197198
when(userMapper.queryUserListByTenant(3)).thenReturn(getUserList());

‎dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/service/WorkerGroupServiceTest.java

+3-2
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
import org.apache.dolphinscheduler.common.constants.Constants;
3030
import org.apache.dolphinscheduler.common.enums.AuthorizationType;
3131
import org.apache.dolphinscheduler.common.enums.UserType;
32+
import org.apache.dolphinscheduler.common.enums.WorkflowExecutionStatus;
3233
import org.apache.dolphinscheduler.dao.entity.User;
3334
import org.apache.dolphinscheduler.dao.entity.WorkerGroup;
3435
import org.apache.dolphinscheduler.dao.entity.WorkflowInstance;
@@ -243,7 +244,7 @@ public void giveRunningProcess_whenDeleteWorkerGroupById_expectFailed() {
243244
List<WorkflowInstance> workflowInstances = new ArrayList<WorkflowInstance>();
244245
workflowInstances.add(workflowInstance);
245246
Mockito.when(workflowInstanceMapper.queryByWorkerGroupNameAndStatus(workerGroup.getName(),
246-
org.apache.dolphinscheduler.service.utils.Constants.NOT_TERMINATED_STATES))
247+
WorkflowExecutionStatus.getNotTerminalStatus()))
247248
.thenReturn(workflowInstances);
248249

249250
Map<String, Object> deleteFailed = workerGroupService.deleteWorkerGroupById(loginUser, 1);
@@ -261,7 +262,7 @@ public void giveValidParams_whenDeleteWorkerGroupById_expectSuccess() {
261262
WorkerGroup workerGroup = getWorkerGroup(1);
262263
Mockito.when(workerGroupMapper.selectById(1)).thenReturn(workerGroup);
263264
Mockito.when(workflowInstanceMapper.queryByWorkerGroupNameAndStatus(workerGroup.getName(),
264-
org.apache.dolphinscheduler.service.utils.Constants.NOT_TERMINATED_STATES)).thenReturn(null);
265+
WorkflowExecutionStatus.getNotTerminalStatus())).thenReturn(null);
265266

266267
Mockito.when(workerGroupMapper.deleteById(1)).thenReturn(1);
267268

‎dolphinscheduler-bom/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -389,8 +389,8 @@
389389
</dependency>
390390

391391
<dependency>
392-
<groupId>mysql</groupId>
393-
<artifactId>mysql-connector-java</artifactId>
392+
<groupId>com.mysql</groupId>
393+
<artifactId>mysql-connector-j</artifactId>
394394
<version>${mysql-connector.version}</version>
395395
<scope>test</scope>
396396
</dependency>

‎dolphinscheduler-dao-plugin/dolphinscheduler-dao-mysql/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -33,8 +33,8 @@
3333
</dependency>
3434

3535
<dependency>
36-
<groupId>mysql</groupId>
37-
<artifactId>mysql-connector-java</artifactId>
36+
<groupId>com.mysql</groupId>
37+
<artifactId>mysql-connector-j</artifactId>
3838
</dependency>
3939
</dependencies>
4040

‎dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-doris/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -47,8 +47,8 @@
4747
</dependency>
4848

4949
<dependency>
50-
<groupId>mysql</groupId>
51-
<artifactId>mysql-connector-java</artifactId>
50+
<groupId>com.mysql</groupId>
51+
<artifactId>mysql-connector-j</artifactId>
5252
</dependency>
5353
</dependencies>
5454

0 commit comments

Comments
 (0)