Added partial integration tests for pre-submit

[fangyh20]

Added github action for pre-submit
Added Service account based auth for integration test
Made the existing integration tests compatible with CI/CD
Made tests compatible with Spark4/runtime3.0
Changed pyspark.sql.functions -> pyspark.sql.connect.functions
Marked some of the tests skipped for now, pending for validation

[gemini-code-assist]

Summary of Changes

Hello @fangyh20, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the integration test suite by making it more robust and compatible with CI/CD environments. It introduces dynamic authentication based on the execution environment, refines test session management for better resource handling, and strategically skips tests that are not yet ready for automated execution, ensuring a smoother pre-submit validation process.

Highlights

• CI/CD Integration: Introduced logic to automatically detect if tests are running in a CI environment (e.g., GitHub Actions) and configure authentication to use service accounts accordingly. This streamlines the execution of integration tests in automated pipelines.
• Test Compatibility & Refactoring: Modified existing integration tests to be more compatible with CI/CD practices. This includes explicit imports for SparkConnect functions within test methods and ensuring proper cleanup of Spark sessions after each test run to prevent resource conflicts.
• Test Management & Skipping: Added new pytest markers (integration, ci_safe) for better test categorization. Several tests, such as test_add_artifacts_pypi_package and test_session_reuse_with_custom_id, have been marked to be skipped due to current infrastructure limitations or pending verification.
• Session Management Enhancements: The connect_session fixture now includes a cleanup mechanism to stop Spark sessions after tests complete. Additionally, session IDs in reuse tests are now randomized to avoid potential conflicts across test runs.

Ignored Files

• Ignored by pattern: .github/workflows/** (1)
  • .github/workflows/integration-tests.yaml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces several improvements to the integration tests to make them compatible with a CI/CD environment. Key changes include adding pytest markers, auto-detecting the authentication type based on the environment, and ensuring Spark sessions are properly cleaned up after tests. Some tests are temporarily skipped, and one is improved to use a randomized session ID to prevent conflicts. The changes are well-structured and significantly enhance the robustness of the test suite. I have one suggestion to improve error visibility during test cleanup.