[#6656] fix(catalog): support read time/datetime/timestamp columns with precision #6657

predator4ann · 2025-03-10T09:34:47Z

Why are the changes needed?

Fix: #6656

Does this PR introduce any user-facing change?

No

How was this patch tested?

UT already added.

...n/java/org/apache/gravitino/trino/connector/catalog/jdbc/mysql/MySQLDataTypeTransformer.java

jerryshao · 2025-03-18T11:41:12Z

@diqiu50 would you please help to review this PR again?

diqiu50

How do you handle the type of Timestamp without precision? Should a parameter be added to indicate using the default precision

predator4ann · 2025-03-24T09:11:05Z

How do you handle the type of Timestamp without precision? Should a parameter be added to indicate using the default precision

Hi @diqiu50, I understand your point of view. Yes, I also think that controlling whether to use default precision through parameters is a good way. I will revise it again.

jerqi · 2025-04-10T07:21:15Z

How do you handle the type of Timestamp without precision? Should a parameter be added to indicate using the default precision

Hi @diqiu50, I understand your point of view. Yes, I also think that controlling whether to use default precision through parameters is a good way. I will revise it again.

Gently ping. Are you working on this issue?

predator4ann · 2025-04-16T07:29:00Z

How do you handle the type of Timestamp without precision? Should a parameter be added to indicate using the default precision

Hi @diqiu50, I understand your point of view. Yes, I also think that controlling whether to use default precision through parameters is a good way. I will revise it again.

Gently ping. Are you working on this issue?

Apologies for the delay! I will prioritize resolving this issue by this week.

predator4ann · 2025-04-20T15:22:09Z

hi @diqiu50, could you please take another look and let me know if everything meets the requirements, or if there's anything else to be adjusted? Thx.

api/src/main/java/org/apache/gravitino/rel/types/Types.java

...c-mysql/src/main/java/org/apache/gravitino/catalog/mysql/operation/MysqlTableOperations.java

predator4ann · 2025-04-22T02:09:12Z

hi @diqiu50, I've implemented updates to address the previously failed IT tests. Could you kindly review the latest version when convenient?

diqiu50 · 2025-04-23T08:12:44Z

Please fix the error in CI

api/src/main/java/org/apache/gravitino/rel/types/Types.java

...n/java/org/apache/gravitino/trino/connector/catalog/jdbc/mysql/MySQLDataTypeTransformer.java

mchades

Each catalog has its own default value for the time type, and we should handle this default value when converting the field type of the catalog.

api/src/main/java/org/apache/gravitino/rel/types/Types.java

mchades

should we also update the behavior of the Spark connector?

BTW, don't forget to update the user doc and open-api doc

mchades · 2025-06-01T08:42:12Z

...bc-common/src/main/java/org/apache/gravitino/catalog/jdbc/operation/JdbcTableOperations.java

+   * @return the precision of the time/datetime/timestamp type
+   */
+  public Integer calculateDatetimePrecision(String typeName, int columnSize, int scale) {
+    return null;


why always return null?

It provides a default behavior for catalogs that do not implement this method.

If so, please add a comment on the method to specify when the developer should return null or non-null.

mchades · 2025-06-01T08:44:10Z

...dbc-doris/src/main/java/org/apache/gravitino/catalog/doris/converter/DorisTypeConverter.java

-        return Types.TimestampType.withoutTimeZone();
+        return Optional.ofNullable(typeBean.getDatetimePrecision())
+            .map(Types.TimestampType::withoutTimeZone)
+            .orElseGet(Types.TimestampType::withoutTimeZone);


always return Types.TimestampType::withoutTimeZone ?

Yes, Doris only supports the timezone-free time type datetime

...c-doris/src/main/java/org/apache/gravitino/catalog/doris/operation/DorisTableOperations.java

mchades · 2025-06-01T08:57:00Z

...-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDorisIT.java

could you plz add tests for datetime/timestamp column with specific precision?

mchades · 2025-06-01T09:10:34Z

...dbc-mysql/src/main/java/org/apache/gravitino/catalog/mysql/converter/MysqlTypeConverter.java

        // MySQL converts TIMESTAMP values from the current time zone to UTC for storage, and back
        // from UTC to the current time zone for retrieval. (This does not occur for other types
        // such as DATETIME.) see more details:
        // https://dev.mysql.com/doc/refman/8.0/en/datetime.html
      case TIMESTAMP:
-        return Types.TimestampType.withTimeZone();
+        return Optional.ofNullable(typeBean.getDatetimePrecision())


when will typeBean.getDatetimePrecision() return null?

from our implementation, it should not be possible

mchades · 2025-06-01T09:16:07Z

...n/java/org/apache/gravitino/trino/connector/catalog/jdbc/mysql/MySQLDataTypeTransformer.java

+      default:
+        throw new TrinoException(
+            GravitinoErrorCode.GRAVITINO_ILLEGAL_ARGUMENT,
+            "Invalid MySQL timestamp precision: " + precision + ". Valid values are 0, 3, 6");


You can add a comment to describe the source of this behavior.

mchades

BTW, after this PR merged, we also need to update the Python client (since #7241 has been merged), so could you plz open an new issue to track this?

Signed-off-by: zacsun <[email protected]>

mchades · 2025-06-05T07:25:01Z

docs/jdbc-doris-catalog.md

+| `Double`       | `Double`      |
+| `Decimal`      | `Decimal`     |
+| `Date`         | `Date`        |
+| `Timestamp[p]` | `Datetime[p]` |


Timestamp(p)

mchades

missing the doc

gravitino/docs/manage-relational-metadata-using-gravitino.md

Line 898 in f0f54c6

    
           | Timestamp                 | `Types.TimestampType.withoutTimeZone()`                                  | `timestamp`                                                                                                                          | Timestamp type, indicates a timestamp without timezone                                                                                                                     |

jerqi requested a review from diqiu50 March 10, 2025 11:11

diqiu50 reviewed Mar 11, 2025

View reviewed changes

...n/java/org/apache/gravitino/trino/connector/catalog/jdbc/mysql/MySQLDataTypeTransformer.java Outdated Show resolved Hide resolved

diqiu50 reviewed Mar 11, 2025

View reviewed changes

...n/java/org/apache/gravitino/trino/connector/catalog/jdbc/mysql/MySQLDataTypeTransformer.java Outdated Show resolved Hide resolved

yuqi1129 requested a review from mchades March 11, 2025 07:33

diqiu50 reviewed Mar 13, 2025

View reviewed changes

...n/java/org/apache/gravitino/trino/connector/catalog/jdbc/mysql/MySQLDataTypeTransformer.java Outdated Show resolved Hide resolved

jerryshao requested a review from diqiu50 March 20, 2025 09:26

diqiu50 reviewed Mar 20, 2025

View reviewed changes

diqiu50 reviewed Apr 21, 2025

View reviewed changes

api/src/main/java/org/apache/gravitino/rel/types/Types.java Outdated Show resolved Hide resolved

diqiu50 reviewed Apr 21, 2025

View reviewed changes

...c-mysql/src/main/java/org/apache/gravitino/catalog/mysql/operation/MysqlTableOperations.java Outdated Show resolved Hide resolved

predator4ann force-pushed the fix/support_timestamp_digits branch from e525a57 to d3e8ef5 Compare April 22, 2025 02:05

diqiu50 reviewed Apr 23, 2025

View reviewed changes

api/src/main/java/org/apache/gravitino/rel/types/Types.java Show resolved Hide resolved

diqiu50 reviewed Apr 23, 2025

View reviewed changes

...n/java/org/apache/gravitino/trino/connector/catalog/jdbc/mysql/MySQLDataTypeTransformer.java Outdated Show resolved Hide resolved

diqiu50 reviewed Apr 23, 2025

View reviewed changes

...n/java/org/apache/gravitino/trino/connector/catalog/jdbc/mysql/MySQLDataTypeTransformer.java Outdated Show resolved Hide resolved

mchades reviewed Apr 23, 2025

View reviewed changes

api/src/main/java/org/apache/gravitino/rel/types/Types.java Show resolved Hide resolved

api/src/main/java/org/apache/gravitino/rel/types/Types.java Outdated Show resolved Hide resolved

predator4ann force-pushed the fix/support_timestamp_digits branch 2 times, most recently from 664bda6 to 1359a93 Compare May 28, 2025 06:07

mchades reviewed Jun 1, 2025

View reviewed changes

predator4ann changed the title ~~[#6656 ] fix(trino-connector): support read MySQL time/datetime/timestamp columns with precision~~ [#6656] fix(catalog): support read time/datetime/timestamp columns with precision Jun 3, 2025

predator4ann mentioned this pull request Jun 4, 2025

[Enhancement] Support Time/Datetime/Timestamp Precision Across Catalogs and Connectors #7331

Open

predator4ann force-pushed the fix/support_timestamp_digits branch from 2c59e0c to 28febe9 Compare June 4, 2025 10:38

feat: support read time/datetime/timestamp columns with precision

9d76ca2

Signed-off-by: zacsun <[email protected]>

predator4ann force-pushed the fix/support_timestamp_digits branch from 28febe9 to 9d76ca2 Compare June 4, 2025 10:39

fix: update doc

dfc09ab

Signed-off-by: zacsun <[email protected]>

mchades reviewed Jun 5, 2025

View reviewed changes

[#6656] fix(catalog): support read time/datetime/timestamp columns with precision #6657

Are you sure you want to change the base?

[#6656] fix(catalog): support read time/datetime/timestamp columns with precision #6657

Uh oh!

Conversation

predator4ann commented Mar 10, 2025 • edited by mchades Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jerryshao commented Mar 18, 2025

Uh oh!

diqiu50 left a comment

Choose a reason for hiding this comment

Uh oh!

predator4ann commented Mar 24, 2025

Uh oh!

jerqi commented Apr 10, 2025

Uh oh!

predator4ann commented Apr 16, 2025

Uh oh!

predator4ann commented Apr 20, 2025

Uh oh!

Uh oh!

Uh oh!

predator4ann commented Apr 22, 2025

Uh oh!

diqiu50 commented Apr 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mchades left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mchades left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mchades left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mchades left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

predator4ann commented Mar 10, 2025 •

edited by mchades

Loading