Alter parallelism of specific fragment

[maingoh]

Is your feature request related to a problem? Please describe.

It is currently possible to set the parallelism at the materialized view level. We can set a fixed value but all fragments will have the same parallelism. However some fragment could need more parallelism than others.
The adaptive value seem to try to balance the fragments but it is not very clear how it do it and give less control and predictability.

Describe the solution you'd like

-- First set parallelism 8 for all fragments of the MV
ALTER MATERIALIZED VIEW foo SET parallelism = 8; 
-- Then alter a specific fragment
ALTER FRAGMENT 123 SET PARALLELISM = 16;

Describe alternatives you've considered

No response

Additional context

No response

[BugenZhao]

Hi. RisingWave does support altering the parallelism of a specific fragment, but we currently have no plans to prioritize making this feature user-facing. This is mainly because such command

• requires user knowledge of how streaming pipeline is executed
• can be easily overridden by high-level scaling operation like altering the parallelism of a job
• doesn't seem to be that useful, given that we can increase the job parallelism to increase the fragment's parallelism

However, if you still need such advanced and fine-grained control of fragment parallelism, there's an API named reschedule in risectl might help. You can check the usage by running

risingwave ctl meta reschedule --help
(or) risectl meta reschedule --help

[maingoh]

Hi ! I understand it is quite advanced :)

To give a bit more context, from my observations, one of the fragment of my MV seemed to be the bottleneck (backpressure). Instead of spreading all the cores to all the fragment I thought it would be wiser to attribute the cores to only this fragment. This way fragment that are lightweight would use less parallelism, and compute intensive fragments would be allowed to use more cores. If I set it at the MV level, they will all use the same percentage of CPU no, how is the CPU balanced across the MV fragments ?