chore: unify inconsistent SessionState in datafusion operations

[abhi-airspace-intelligence]

Description

Unifies the SessionState management for the various DeltaOps by moving it to use the dyn Session interface, downcasting to SessionContext when relevant.

Documentation

Session trait in datafusion: https://docs.rs/datafusion/50.0.0/datafusion/catalog/trait.Session.html

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[codecov]

Codecov Report

❌ Patch coverage is 70.58824% with 60 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.32%. Comparing base (d4a70e3) to head (16f288c).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/core/src/operations/load_cdf.rs	11.76%	15 Missing ⚠️
crates/core/src/operations/delete.rs	53.84%	10 Missing and 2 partials ⚠️
crates/core/src/operations/load.rs	45.45%	10 Missing and 2 partials ⚠️
crates/core/src/operations/constraints.rs	71.42%	2 Missing and 2 partials ⚠️
crates/core/src/operations/update.rs	82.60%	2 Missing and 2 partials ⚠️
crates/core/src/operations/write/mod.rs	80.00%	3 Missing and 1 partial ⚠️
crates/core/src/operations/optimize.rs	88.00%	2 Missing and 1 partial ⚠️
crates/core/src/delta_datafusion/find_files.rs	71.42%	0 Missing and 2 partials ⚠️
crates/core/src/operations/merge/mod.rs	77.77%	1 Missing and 1 partial ⚠️
crates/core/src/operations/mod.rs	75.00%	0 Missing and 1 partial ⚠️
... and 1 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3816      +/-   ##
==========================================
- Coverage   74.40%   74.32%   -0.09%     
==========================================
  Files         147      147              
  Lines       39544    39585      +41     
  Branches    39544    39585      +41     
==========================================
- Hits        29423    29421       -2     
- Misses       8734     8775      +41     
- Partials     1387     1389       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[rtyler]

Marking this as draft until the conflicts are resolved 🙈

[abhi-airspace-intelligence]

Should be good to go.

@roeap do you have a preference on what the var should be for datafusion state? In some places it's session, state, or df_state. Would prefer to have one term.

[roeap]

Looking great @abhi-airspace-intelligence.

As for the name i think session is the way to go, as this is the traits name. state would be second, since we often used the implementation SessionState.

We also do have some fixable lints (unused imports). Granted these are hard to track in our codebase right now, since there are SOOOO many deprecations. but maybe try to fix these :). - i.e. whatever makes check fail.

I'll remove the linked issue, since I believe we should also centralise setting up the session in a dedicated function o.a. since we still need to create session in several places. Important step towards consolidation though!!

[abhi-airspace-intelligence]

@roeap I will fix the lints and update the name tomorrow!

[roeap]

👍