Open
Description
Is your feature request related to a problem or challenge?
Now we're able to set session configurations through SQL interface:
- Doc about config: https://datafusion.apache.org/user-guide/configs.html
- Example to set it through executing an SQL query:
set datafusion.execution.batch_size = 1024; SELECT * FROM tbl ORDER BY c1;
However, runtime configurations like memory limit can't be set like this, it has to be set through Rust APIs during setup like the following example:
datafusion/datafusion/core/tests/memory_limit/mod.rs
Lines 455 to 461 in 28451b5
Although it's called
RuntimeConfiguration
, it's still some static parameters to be configured, instead of a long-running daemon. So I think it make sense to be able to change it from query to query, and also provide a easier to use API.Also, they don't have a user manual like SessionConfig.
Describe the solution you'd like
- Add documents for runtime configurations like memory limit similar to SessionConfig to the user-guide.
- Make it configurable through SQL interface.
Describe alternatives you've considered
No response
Additional context
No response