feat(datafusion): add max_temp_directory_size parameter for z-order and compact operations for DataFusion #10584

Job	Run time
Python Build (Python 3.10 PyArrow latest)	2m 33s
test-minimal	1m 53s
Python Build (Python 3.10 LakeFS Integration tests)	2m 8s
Python Build (Python 3.10 Unity Catalog Integration tests)	2m 30s
PySpark Integration Tests	2m 19s
Running with Python 3.9	1m 38s
Running with Python 3.10	1m 28s
Running with Python 3.11	1m 56s
Python Build (Python 3.10 Unity Catalog Integration tests)	1m 49s
Running with Python 3.12	2m 15s
	20m 29s

Provide feedback