Skip to content

Add Spark 4.0 support #297

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: develop
Choose a base branch
from
Draft

Add Spark 4.0 support #297

wants to merge 1 commit into from

Conversation

dolfinus
Copy link
Member

@dolfinus dolfinus commented Jul 25, 2024

Change Summary

  • Update CI matrix to include Spark 4.x. By default, it is not used for tests, unless some integration has been changed (like 2.x).
  • Spark 4.x also supports Java 22, include it to Readme & CI matrix.
  • Update JDBCConnection methods to be compatible with Spark 4.0 JDBCUtils methods signatures.
  • Update XML class get_packages() and check_if_supported() methods, as Spark 4.0 includes XML support.
  • Since Spark 4.0, DecimalType(38, 10).typeName() started returning decimal(38, 10) instead of decimal, which breaks some Oracle tests. Updated SparkTypeToHWM implementation to have direct mapping between Spark data type classes and HWM classes.

Notes:

  • Skip Excel tests on Spark 4.0, as there is no package version.

Related issue number

Checklist

  • Commit message and PR title is comprehensive
  • Keep the change as small as possible
  • Unit and integration tests for the changes exist
  • Tests pass on CI and coverage does not decrease
  • Documentation reflects the changes where applicable
  • docs/changelog/next_release/<pull request or issue id>.<change type>.rst file added describing change
    (see CONTRIBUTING.rst for details.)
  • My PR is ready to review.

@dolfinus dolfinus self-assigned this Jul 25, 2024
Copy link

codecov bot commented Jul 25, 2024

Codecov Report

Attention: Patch coverage is 83.33333% with 10 lines in your changes missing coverage. Please review.

Project coverage is 92.59%. Comparing base (84a85a4) to head (2101333).

Files with missing lines Patch % Lines
onetl/file/format/xml.py 41.66% 4 Missing and 3 partials ⚠️
onetl/hwm/store/hwm_class_registry.py 84.61% 1 Missing and 1 partial ⚠️
.../connection/db_connection/jdbc_mixin/connection.py 95.45% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #297      +/-   ##
===========================================
+ Coverage    92.36%   92.59%   +0.22%     
===========================================
  Files          229      229              
  Lines        10247    10272      +25     
  Branches      1021     1029       +8     
===========================================
+ Hits          9465     9511      +46     
+ Misses         587      569      -18     
+ Partials       195      192       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 0e55abd to b91db4e Compare July 25, 2024 18:52
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from b91db4e to 3d1d944 Compare July 25, 2024 19:02
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 3d1d944 to 469c1d9 Compare July 25, 2024 19:08
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 469c1d9 to dd21d0a Compare July 25, 2024 19:20
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from dd21d0a to 776f351 Compare July 25, 2024 19:26
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 776f351 to 0ef84bd Compare July 25, 2024 19:32
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 0ef84bd to d0c9090 Compare July 29, 2024 12:28
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from d0c9090 to 1a00a33 Compare July 29, 2024 12:47
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 1a00a33 to dca6397 Compare July 29, 2024 12:52
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from dca6397 to d050a2d Compare July 29, 2024 13:00
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from d050a2d to 849659e Compare July 29, 2024 13:45
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 849659e to 978dc31 Compare July 29, 2024 13:53
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 978dc31 to e68423d Compare July 29, 2024 14:06
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from e68423d to ffaba47 Compare July 29, 2024 14:11
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from d20d9e7 to 068fe01 Compare December 4, 2024 15:24
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 068fe01 to da0ae6b Compare December 5, 2024 12:02
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from da0ae6b to f5ee6f8 Compare December 5, 2024 12:09
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from f5ee6f8 to f23637a Compare December 5, 2024 12:17
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from f23637a to 6c3cb15 Compare January 10, 2025 12:07
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 6c3cb15 to dcc44ad Compare February 21, 2025 14:53
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from dcc44ad to a18bbfb Compare May 23, 2025 14:55
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from a18bbfb to 38c681f Compare May 23, 2025 14:58
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 38c681f to 11c35be Compare May 23, 2025 14:59
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 11c35be to adee2c6 Compare May 23, 2025 15:00
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from adee2c6 to 9b2c584 Compare May 23, 2025 15:05
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 9b2c584 to e3bfe08 Compare May 23, 2025 15:07
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from e3bfe08 to d4ebaf8 Compare May 23, 2025 15:13
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from d4ebaf8 to 2101333 Compare May 23, 2025 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant