feat(datafusion): add max_temp_directory_size parameter for z-order and compact operations for DataFusion #10639

Job	Run time
test-minimal	36s
Python Build (Python 3.10 Unity Catalog Integration tests)	4m 47s
Python Build (Python 3.10 LakeFS Integration tests)	4m 59s
Python Build (Python 3.10 PyArrow latest)	6m 10s
PySpark Integration Tests	5m 23s
Running with Python 3.11	4m 48s
Running with Python 3.10	4m 38s
Running with Python 3.12	4m 38s
Running with Python 3.9	4m 41s
Python Build (Python 3.10 Unity Catalog Integration tests)	4m 39s
	45m 19s

Provide feedback