【pressure test】the number of ”running” tasks and processes increases slowly

[kindragos]

version:
1.3.9
Test scenario:
The operation of the platform when a large number of tasks are executed at the same time.

Test method:
Start a springboot project on the additional virtual machine, expose a delay interface, and return after 30 minutes. At the same time, the maximum number of threads of this project is configured as server "tomcat. max-threads=100000"
A workflow is defined in DS. There is only one shell type task node. The shell calls the above delay interface through curl. Execute tasks on only one worker.
JMeter Ramp-up Period is 1 second, and 1000 threads are connected to the API interface to run the above DS workflow.

Machine configuration:
ds1：4C16G； ds2：16C32G； ds3：16C32G

Deployment method:
API deployed on ds1, jvm-0.5g; The master is deployed on ds1 and ds2, jvm-2g; The worker is deployed on ds3, jvm-4g.

Other DS configurations:
master. exec. threads=1000
master. exec. task. num=1000
worker. exec. threads=100000

Other deployment server configurations:
max user processes (-u) 65535

Phenomena obtained:
After receiving the request, the API should store data in the t_ds_command table normally and quickly. However, the subsequent process after the master reads data from the t_ds_command table is slow, which leads to the slow growth of the number of "running" tasks and processes seen on the UI home page. Refresh pages per second. When there are hundreds of pages, even one page per second is added.

Other test methods:
Besides API, workflow can be run through scheduled tasks.
I made five workflows, each with a scheduled task running every second.
From the page UI, the phenomenon is the same; From the database, the start time of all instances of a workflow is 1 second at first, and then more than 10+ seconds later.

I'd like to discuss it.
I don't think this is normal. I try to improve "master.exec.threads", but it has no effect.
I don't think the start-up process should be so slow.

---

中文：
【压测】“正在运行”的任务及流程增长缓慢

版本：
1.3.9
测试场景：
大量任务同时执行时，平台的运行情况。
测试方式：
在额外的虚拟机上启一个springboot项目，暴露一个延迟接口，30分钟后返回。同时这个项目的最大线程数如此配置"server.tomcat.max-threads=100000"
DS里定义一个工作流，只有一个shell类型的任务节点，shell通过curl调用上述延迟接口。只在一个worker上执行任务。
jmeter预热1秒，1000个线程打到api接口上，运行上述DS工作流。
机器配置：
ds1：4C16G；ds2：16C32G；ds3：16C32G
部署方式：
api部署在ds1上，jvm-0.5G；master部署在ds1、ds2上，jvm-2G；worker部署在ds3上，jvm-4G。
其他DS配置：
master.exec.threads=1000
master.exec.task.num=1000
worker.exec.threads=100000
其他部署服务器配置：
max user processes (-u) 65535

得到的现象：
api收到请求后，向t_ds_command表存数据应该比较正常，很快。但master从t_ds_command表读数据之后的后续流程，不知道是哪里慢了，导致UI首页看到的“正在运行”的任务数和进程数增长缓慢。每秒刷新页面，在大几百数量的时候，甚至1秒才增加1个。

其他的测试方法：
运行工作流除了通过api，可以通过定时任务。
我搞了五个工作流，每个工作流都有一个每秒运行的定时任务。
从页面UI上看，现象一样；从数据库里看某一个工作流的所有实例的启动时间，刚开始还是1秒的间隔，后来就是十几秒的间隔了。

想讨论一下。
感觉这种情况不是很正常。我尝试提高"master.exec.threads"，但并没有任何作用。
我觉得启动流程不应该这么慢。

[davidzollo]

you can analysis the master and worker server runtime log for the detail info to find out the place where block the performance. IMO, I suggest you watch the database performance first, don't set the param master.exec.task.num so large, I think 20 will be better

By the way, if you need a better performance,
You can have a try on the new version 2.0.1 , 2.x is much better performance than 1.x, about >20x better