Skip to content
Merged

DbType2 #1632

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
67a1ef8
attempt at converting duckdb to column conversions. WIP
Jolanrensen Dec 9, 2025
210af02
Refactored DbType to use `DbColumnTypeInformation`, generated from `T…
Jolanrensen Dec 10, 2025
d21dc79
converted DuckDb to new preprocessing DbType. Turns out I might need …
Jolanrensen Dec 10, 2025
0ee7024
wip DuckDb nested preprocessing
Jolanrensen Dec 10, 2025
f96234d
added memoization for TableColumnMetadata -> AnyDbColumnTypeInformation
Jolanrensen Dec 10, 2025
d1b655b
renaming, added extra constructors for TypeInformation, restricting t…
Jolanrensen Dec 11, 2025
4d8acfb
fixed duckdb tests
Jolanrensen Dec 11, 2025
6539830
added jdbc source type parameter
Jolanrensen Dec 12, 2025
e114fca
struct parsing for duckdb working!
Jolanrensen Dec 15, 2025
1a9f581
merging column creation and post processing dbType
Jolanrensen Dec 16, 2025
6bf27fb
created AdvancedDbType so we can have "simple" and "advanced" db type…
Jolanrensen Dec 16, 2025
d678032
added resultSetReader option to JdbcToDataFrameConverter, converting …
Jolanrensen Dec 17, 2025
736fe79
exploring struct/composite types for postgresql
Jolanrensen Dec 17, 2025
be83cc5
added duckDb STRUCT[] column to FrameColumn conversion
Jolanrensen Dec 17, 2025
2a6b620
Merge branch 'master' into DbType2
Jolanrensen Jan 27, 2026
7b1c5af
to support runtime json parsing for duckdb, allow targetSchema = null.
Jolanrensen Jan 28, 2026
1d2494f
reverted changes from name-based jdbc columns back to order/index bas…
Jolanrensen Feb 10, 2026
df9dd90
Merge branch 'master' into DbType2
Jolanrensen Feb 11, 2026
0aa0c15
Merge branch 'master' into DbType2
Jolanrensen Feb 11, 2026
a9aee70
made checkSchema run only in debug builds
Jolanrensen Feb 11, 2026
5c8ccdc
simplified DbType and JdbcToDataFrameConverter typing situation, adde…
Jolanrensen Feb 11, 2026
91dedd0
apidump
Jolanrensen Feb 11, 2026
35dd613
Turns out getDataFrameCompatibleColumnNames was not necessary as DbTy…
Jolanrensen Feb 11, 2026
699aa38
Fixed h2 test
Jolanrensen Feb 12, 2026
05539d9
simplified api for creating JdbcToDataFrameConverter instances to bui…
Jolanrensen Feb 12, 2026
bf00824
enabled buildConfig for dataframe-jupyter too
Jolanrensen Feb 12, 2026
6e04da6
Added some more kdocs to fetchAndConvertDataFromResultSet() explainin…
Jolanrensen Feb 12, 2026
cc64e3b
changed buildConfig convention plugin to generate unique BuildConfig …
Jolanrensen Feb 12, 2026
0ef8f55
added CacheKey for AdvancedDbType as recommended by copilot
Jolanrensen Feb 13, 2026
d26940a
fixed parsing Struct types for DuckDb as recommended by copilot. Adde…
Jolanrensen Feb 13, 2026
53e6efa
Merge branch 'master' into DbType2
Jolanrensen Feb 17, 2026
f8359a6
fixing types for local db tests
Jolanrensen Feb 17, 2026
3ab6bd0
fixing postgres types and enabling all extension types #537
Jolanrensen Feb 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
package org.jetbrains.kotlinx.dataframe.io.db

import org.jetbrains.kotlinx.dataframe.DataColumn
import org.jetbrains.kotlinx.dataframe.schema.ColumnSchema
import java.sql.ResultSet
import kotlin.reflect.KType

/**
* Alternative version of [DbType] that allows to customize type mapping
* by initializing a [JdbcToDataFrameConverter] instance for each JDBC type.
*
* This can be helpful for JDBC databases that support structured data, like [DuckDb]
* or that need to a lot of type mapping.
*/
public abstract class AdvancedDbType(dbTypeInJdbcUrl: String) : DbType(dbTypeInJdbcUrl) {

protected abstract fun generateConverter(tableColumnMetadata: TableColumnMetadata): AnyJdbcToDataFrameConverter

private val converterCache = mutableMapOf<TableColumnMetadata, AnyJdbcToDataFrameConverter>()

protected fun getConverter(tableColumnMetadata: TableColumnMetadata): AnyJdbcToDataFrameConverter =
converterCache.getOrPut(tableColumnMetadata) {
Comment thread
Jolanrensen marked this conversation as resolved.
Outdated
generateConverter(tableColumnMetadata)
}

final override fun getExpectedJdbcType(tableColumnMetadata: TableColumnMetadata): KType =
getConverter(tableColumnMetadata).expectedJdbcType

final override fun getPreprocessedValueType(
tableColumnMetadata: TableColumnMetadata,
expectedJdbcType: KType,
): KType = getConverter(tableColumnMetadata).preprocessedValueType

final override fun getTargetColumnSchema(
tableColumnMetadata: TableColumnMetadata,
expectedValueType: KType,
): ColumnSchema = getConverter(tableColumnMetadata).targetSchema

final override fun <J : Any> getValueFromResultSet(
rs: ResultSet,
columnIndex: Int,
tableColumnMetadata: TableColumnMetadata,
expectedJdbcType: KType,
): J? =
getConverter(tableColumnMetadata).cast<J, Any, Any>()
.getValueFromResultSetOrElse(rs, columnIndex) {
try {
rs.getObject(columnIndex + 1)
} catch (_: Throwable) {
// TODO?
rs.getString(columnIndex + 1)
} as J?
}

final override fun <J : Any, D : Any> preprocessValue(
value: J?,
tableColumnMetadata: TableColumnMetadata,
expectedJdbcType: KType,
expectedPreprocessedValueType: KType,
): D? = getConverter(tableColumnMetadata).cast<J, D, Any>().preprocessOrCast(value)

final override fun <D : Any, P : Any> buildDataColumn(
name: String,
values: List<D?>,
tableColumnMetadata: TableColumnMetadata,
targetColumnSchema: ColumnSchema,
inferNullability: Boolean,
): DataColumn<P?> =
getConverter(tableColumnMetadata).cast<Any, D, P>()
.buildDataColumnOrNull(name, values, inferNullability)
?: values.toDataColumn(
name = name,
targetColumnSchema = targetColumnSchema,
inferNullability = inferNullability,
)
}
Loading