chore: upgrade to delta-kernel-rs 0.16.0 and remove more dependencies #10232

Job	Run time
Python Build (Python 3.10 PyArrow latest)	4m 3s
test-minimal	4m 46s
Python Build (Python 3.10 LakeFS Integration tests)	6m 31s
Python Build (Python 3.10 Unity Catalog Integration tests)	6m 7s
PySpark Integration Tests	6m 57s
Python Build (Python 3.10 Unity Catalog Integration tests)	5m 56s
Running with Python 3.10	5m 47s
Running with Python 3.11	5m 47s
Running with Python 3.12	5m 41s
Running with Python 3.9	5m 37s
	57m 12s

Provide feedback