DbType2

[Jolanrensen]

Fixes #1273
Fixes #1587
Fixes #461
Fixes #537

Helps #387

Takes care of #462, but only for DuckDB.

Follows up on #1266 and #462

DbType was overhauled greatly:

• Removed convertSqlTypeToColumnSchemaValue and convertSqlTypeToKType in favor of getExpectedJdbcType
• removed extractValueFromResultSet in favor of getValueFromResultSet
• introduced a new streamlined structure for reading from a ResultSet in DbType:
  • getValueFromResultSet() retrieves a value from a ResultSet given a column and row with the type given by getExpectedJdbcType()
  • Next, the value is optionally preprocessed by preprocessValue() with the new type given by getPreprocessedValueType()
  • Finally, the values are turned into a DataFrame column by buildDataColumn() with the new schema given by getTargetColumnSchema()
• In debug mode, the schema is checked at runtime.
• Column names might change, according to Use ColumnNameGenerator for consistent column name repair across IO readers #387
• By default, we now preprocess Java's LocalDateTime to Kotlin's LocalDateTime, java.sql.TimeStamp to Kotlin Instant and UUIDs to Kotlin Uuid. More preprocessing might follow later.

AdvancedDbType

• Introduced AdvancedDbType + JdbcToDataFrameConverter for easier Type-conversion logic

DuckDB

• Completely rewritten to AdvancedDbType
• Supports all DuckDB types including Structs, Maps, Lists/Arrays (with recursive conversion!) and JSON Define how JSON database types should be mapped to DataFrame (String vs ColumnGroup) #462
• This now makes it the most advanced SQL database we support completely!

Other:

• BuildConfig now includes the module name to not clash with other modules.
• Added support for all PostgreSQL extension types Add support for PostgreSQL geometry types #537
• Ran all local DB tests

[zaleslaw]

ExpectedJdbcTypes could be more complex structure, but better to keep order of column according indicies for debugging, not name-based, it could be edge object with fields: index, name, KType for example

[Jolanrensen]

That's certainly possible! Though that would look a bit more like AdvancedDbType which, for each column, requires you to provide an AnyJdbcToDataFrameConverter containing all information needed to read and convert that column. Maybe the concept could be merged

[zaleslaw]

Describe somewhere the processes

Expected->Preprocessed (what's the difference and why we need this step, what does it give)

[zaleslaw]

It's very safe and cheap way for any database, I moved from taking info from TableColumnMetadata, also lead to less error - prone in our codebase and make it flexible

[Jolanrensen]

Except for empty databases that just have a schema and no data. When building just the schema, we shouldn't have to look at the actual database contents. It's up to the DbType implementor to provide a watertight getExpectedJdbcType(), getPreprocessedValueType(), and getTargetColumnSchema() from just TableColumnMetadata. So if we manage to construct TableColumnMetadatas from connection, tableName and dbType, we can simply call those functions to get a schema without accessing any data.

[Jolanrensen]

...and for databases you may have no query access to

[Jolanrensen]

#1681

[zaleslaw]

J?

[Jolanrensen]

J-DBC. It's clearer in the context of AdvancedDbType:

• J-DBC type
• D-ataFrame type
• P-ost processed type

[Jolanrensen]

@zaleslaw I'm not sure what to do about java.time.OffsetDateTime and java.time.OffsetTime. Do you know if there's a kotlin equivalent?

[Copilot]

Pull request overview

This PR represents a major refactoring of the JDBC module's type inference and conversion system, introducing a new three-stage pipeline architecture for converting database types to DataFrame columns. The changes address several long-standing issues related to type conversions, nested types (STRUCT, ARRAY, MAP), and date/time handling.

Changes:

• Introduced JdbcToDataFrameConverter to encapsulate the three-stage type conversion pipeline (JDBC type → preprocessed value → final DataColumn)
• Added AdvancedDbType abstract class for databases with complex type systems (like DuckDB)
• Refactored DbType API: removed convertSqlTypeToKType/convertSqlTypeToColumnSchemaValue in favor of getExpectedJdbcType/getPreprocessedValueType/getTargetColumnSchema/buildDataColumn
• Updated DuckDB implementation to support STRUCT, MAP, ARRAY, and JSON types with proper conversions
• Changed column name deduplication from underscore-separated ("name_1") to no separator ("name1")
• Updated BuildConfig package naming to include project name
• Updated DuckDB version from 1.3.1.0 to 1.4.2.0
• Converted Java types to Kotlin equivalents (java.sql.Timestamp → kotlin.time.Instant, java.util.UUID → kotlin.uuid.Uuid, etc.)

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/DbType.kt	Core refactoring: new API methods for type conversion pipeline, removed old methods
dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/AdvancedDbType.kt	New abstract class for databases with complex type mappings using JdbcToDataFrameConverter
dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/JdbcToDataFrameConverter.kt	New converter infrastructure with type-safe preprocessing and column building
dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/DuckDb.kt	Major refactoring to extend AdvancedDbType, add support for STRUCT/MAP/ARRAY/JSON types
dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/readJdbc.kt	Refactored to use new three-stage pipeline, added schema validation in DEBUG mode
dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/readDataFrameSchema.kt	Updated to use new API methods
dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/*.kt	Updated all database implementations (PostgreSql, MySql, MariaDb, MsSql, H2, Sqlite) to new API
dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/local/duckDbTest.kt	Updated tests for new type conversions and nested types support
dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/h2/*.kt	Updated expected types to match Kotlin conversions (Timestamp → Instant)
build-logic/src/main/kotlin/dfbuild.buildConfig.gradle.kts	Changed BuildConfig package naming to include project name
gradle/libs.versions.toml	Updated DuckDB version to 1.4.2.0
core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/ColumnNameGenerator.kt	Column name deduplication now uses numeric suffixes without separators

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zaleslaw]

It looks like we have code in commonTestScenarios with the same intention, could we move this schema validation there unify it together somehow?

[Jolanrensen]

@zaleslaw I reverted DbType.getValueFromResultset() back to its original implementation. It turns out rs.getObject(i, javaClass) fails when javaClass is a supertype the actual type of the column in question. That delivers a whole bunch of unnecessary exceptions.

Instead, I fixed #537 and its "money" type by only calling rs.getObject(i, javaClass) inside PostgreSql : DbType for the PGobject extension types. This satisfies all tests consistently without causing potential issues with other SQL types.

All local tests pass now