-
Notifications
You must be signed in to change notification settings - Fork 82
DbType2 #1632
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
DbType2 #1632
Changes from 14 commits
Commits
Show all changes
33 commits
Select commit
Hold shift + click to select a range
67a1ef8
attempt at converting duckdb to column conversions. WIP
Jolanrensen 210af02
Refactored DbType to use `DbColumnTypeInformation`, generated from `T…
Jolanrensen d21dc79
converted DuckDb to new preprocessing DbType. Turns out I might need …
Jolanrensen 0ee7024
wip DuckDb nested preprocessing
Jolanrensen f96234d
added memoization for TableColumnMetadata -> AnyDbColumnTypeInformation
Jolanrensen d1b655b
renaming, added extra constructors for TypeInformation, restricting t…
Jolanrensen 4d8acfb
fixed duckdb tests
Jolanrensen 6539830
added jdbc source type parameter
Jolanrensen e114fca
struct parsing for duckdb working!
Jolanrensen 1a9f581
merging column creation and post processing dbType
Jolanrensen 6bf27fb
created AdvancedDbType so we can have "simple" and "advanced" db type…
Jolanrensen d678032
added resultSetReader option to JdbcToDataFrameConverter, converting …
Jolanrensen 736fe79
exploring struct/composite types for postgresql
Jolanrensen be83cc5
added duckDb STRUCT[] column to FrameColumn conversion
Jolanrensen 2a6b620
Merge branch 'master' into DbType2
Jolanrensen 7b1c5af
to support runtime json parsing for duckdb, allow targetSchema = null.
Jolanrensen 1d2494f
reverted changes from name-based jdbc columns back to order/index bas…
Jolanrensen df9dd90
Merge branch 'master' into DbType2
Jolanrensen 0aa0c15
Merge branch 'master' into DbType2
Jolanrensen a9aee70
made checkSchema run only in debug builds
Jolanrensen 5c8ccdc
simplified DbType and JdbcToDataFrameConverter typing situation, adde…
Jolanrensen 91dedd0
apidump
Jolanrensen 35dd613
Turns out getDataFrameCompatibleColumnNames was not necessary as DbTy…
Jolanrensen 699aa38
Fixed h2 test
Jolanrensen 05539d9
simplified api for creating JdbcToDataFrameConverter instances to bui…
Jolanrensen bf00824
enabled buildConfig for dataframe-jupyter too
Jolanrensen 6e04da6
Added some more kdocs to fetchAndConvertDataFromResultSet() explainin…
Jolanrensen cc64e3b
changed buildConfig convention plugin to generate unique BuildConfig …
Jolanrensen 0ef8f55
added CacheKey for AdvancedDbType as recommended by copilot
Jolanrensen d26940a
fixed parsing Struct types for DuckDb as recommended by copilot. Adde…
Jolanrensen 53e6efa
Merge branch 'master' into DbType2
Jolanrensen f8359a6
fixing types for local db tests
Jolanrensen 3ab6bd0
fixing postgres types and enabling all extension types #537
Jolanrensen File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
76 changes: 76 additions & 0 deletions
76
dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/AdvancedDbType.kt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| package org.jetbrains.kotlinx.dataframe.io.db | ||
|
|
||
| import org.jetbrains.kotlinx.dataframe.DataColumn | ||
| import org.jetbrains.kotlinx.dataframe.schema.ColumnSchema | ||
| import java.sql.ResultSet | ||
| import kotlin.reflect.KType | ||
|
|
||
| /** | ||
| * Alternative version of [DbType] that allows to customize type mapping | ||
| * by initializing a [JdbcToDataFrameConverter] instance for each JDBC type. | ||
| * | ||
| * This can be helpful for JDBC databases that support structured data, like [DuckDb] | ||
| * or that need to a lot of type mapping. | ||
| */ | ||
| public abstract class AdvancedDbType(dbTypeInJdbcUrl: String) : DbType(dbTypeInJdbcUrl) { | ||
|
|
||
| protected abstract fun generateConverter(tableColumnMetadata: TableColumnMetadata): AnyJdbcToDataFrameConverter | ||
|
|
||
| private val converterCache = mutableMapOf<TableColumnMetadata, AnyJdbcToDataFrameConverter>() | ||
|
|
||
| protected fun getConverter(tableColumnMetadata: TableColumnMetadata): AnyJdbcToDataFrameConverter = | ||
| converterCache.getOrPut(tableColumnMetadata) { | ||
| generateConverter(tableColumnMetadata) | ||
| } | ||
|
|
||
| final override fun getExpectedJdbcType(tableColumnMetadata: TableColumnMetadata): KType = | ||
| getConverter(tableColumnMetadata).expectedJdbcType | ||
|
|
||
| final override fun getPreprocessedValueType( | ||
| tableColumnMetadata: TableColumnMetadata, | ||
| expectedJdbcType: KType, | ||
| ): KType = getConverter(tableColumnMetadata).preprocessedValueType | ||
|
|
||
| final override fun getTargetColumnSchema( | ||
| tableColumnMetadata: TableColumnMetadata, | ||
| expectedValueType: KType, | ||
| ): ColumnSchema = getConverter(tableColumnMetadata).targetSchema | ||
|
|
||
| final override fun <J : Any> getValueFromResultSet( | ||
| rs: ResultSet, | ||
| columnIndex: Int, | ||
| tableColumnMetadata: TableColumnMetadata, | ||
| expectedJdbcType: KType, | ||
| ): J? = | ||
| getConverter(tableColumnMetadata).cast<J, Any, Any>() | ||
| .getValueFromResultSetOrElse(rs, columnIndex) { | ||
| try { | ||
| rs.getObject(columnIndex + 1) | ||
| } catch (_: Throwable) { | ||
| // TODO? | ||
| rs.getString(columnIndex + 1) | ||
| } as J? | ||
| } | ||
|
|
||
| final override fun <J : Any, D : Any> preprocessValue( | ||
| value: J?, | ||
| tableColumnMetadata: TableColumnMetadata, | ||
| expectedJdbcType: KType, | ||
| expectedPreprocessedValueType: KType, | ||
| ): D? = getConverter(tableColumnMetadata).cast<J, D, Any>().preprocessOrCast(value) | ||
|
|
||
| final override fun <D : Any, P : Any> buildDataColumn( | ||
| name: String, | ||
| values: List<D?>, | ||
| tableColumnMetadata: TableColumnMetadata, | ||
| targetColumnSchema: ColumnSchema, | ||
| inferNullability: Boolean, | ||
| ): DataColumn<P?> = | ||
| getConverter(tableColumnMetadata).cast<Any, D, P>() | ||
| .buildDataColumnOrNull(name, values, inferNullability) | ||
| ?: values.toDataColumn( | ||
| name = name, | ||
| targetColumnSchema = targetColumnSchema, | ||
| inferNullability = inferNullability, | ||
| ) | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.