-
Notifications
You must be signed in to change notification settings - Fork 730
Description
Is your feature request related to a problem? Please describe.
Most memory parameters can be specified via the toml file (block_cache_capacity, meta_cache_capacity, shared_buffer_capacity) or via an env var RW_RESERVED_MEMORY_BYTES. However the compute_memory does not seem to be tunable.
My scenario seem to look a bit like #10777, my MV does not use much CPU (1.5 CPU), my node has 70 GB of memory. The latency improves a lot when I increase meta_cache_capacity, and backpressure decreases.
However, to increase it I need to lower block_cache_capacity and shared_buffer_capacity and the reserved memory. My node is consuming only around 25 GiB out of the 70 GiB available. It seem most of the unused memory is in the compute_memory which is automatically booked to 42 GiB.
So ideally I could lower the compute_memory booked and use a 30-40 GiB node instead, avoiding wasting memory.
Small additional question: most memory values defined by user are absolute values (in bytes or mb). Couldn't we define it as percentage of the limit so that it adapt better to the node specs. This would avoid to update the config when increasing or decreasing the memory limit.
Thank you :)
Describe the solution you'd like
Tune compute memory using an env var or a toml variable.
Describe alternatives you've considered
No response
Additional context
No response