fix: Restore cleanup of SPARK_LOCAL_IP and PYARROW_IGNORE_TIMEZONE after import

[mabunassar]

PR #2123 removed the finally block that cleaned up environment variables set temporarily for importing pyspark.pandas. This causes SPARK_LOCAL_IP=127.0.0.1 to persist in os.environ, breaking non-YARN distributed Spark setups where executors need to connect to the actual driver IP instead of localhost.

Restores the pre-v0.27.0 behavior: set env vars temporarily, import pyspark.pandas, then clean up.

Fixes #2344

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.55%. Comparing base (53e3a50) to head (7d44445).
⚠️ Report is 69 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2345   +/-   ##
=======================================
  Coverage   83.54%   83.55%           
=======================================
  Files         190      190           
  Lines       16654    16662    +8     
=======================================
+ Hits        13914    13922    +8     
  Misses       2740     2740           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jeremyk]

I would argue that a library like Pandera shouldn't be setting environment variables at all. In our case it breaks spark as we rely on the default behavior and this overrides it. Imo, if these are needed, it should be up to the user to set them.

[cosmicBboy]

@jeremyk do you think this should just be in the documentation instead of something that pandera does in the spark module?

[jeremyk]

@jeremyk do you think this should just be in the documentation instead of something that pandera does in the spark module?

Yeah, that would be my vote. I can see this being a stumbling block but I think documenting it is better than trying to automagically fix it and creating a new stumbling block for other users.

[cosmicBboy]

okay, so plan would be:

• delete any code in pandera that maniplates pyspark-related env vars
• update any unit tests that break (env vars being set in unit test is fine)
• update the documentation as needed.

I will need some help on thie @jeremyk @mabunassar or anyone else who is a pyspark user more familiar with how this works.

[jeremyk]

okay, so plan would be:

• delete any code in pandera that maniplates pyspark-related env vars
• update any unit tests that break (env vars being set in unit test is fine)
• update the documentation as needed.

I will need some help on thie @jeremyk @mabunassar or anyone else who is a pyspark user more familiar with how this works.

Happy to help! Just let me know what you need.