Nanoseconds overflow for Timestamp in Gluten

[4ertus2]

Bug description

Velox set TimestampPrecision::kNanoseconds for parquet Writer by default.

velox/velox/dwio/parquet/writer/Writer.cpp

Line 345 in 6b65978

TimestampPrecision::kNanoseconds));

.

It leads to an error in Gluten in case of integer overflow.

Repro:
write TIMESTAMP '9999-01-01 00:00:00' into Iceberg table via Gluten+Velox

System information

System independent

Relevant logs

Caused by: org.apache.gluten.exception.GlutenException: Exception: VeloxUserError
Error Source: USER
Error Code: INVALID_ARGUMENT
Reason: Could not convert Timestamp(127142956800, 0) to nanoseconds, Exception: VeloxUserError
Error Source: USER
Error Code: ARITHMETIC_ERROR
Reason: integer overflow: 127142956800 * 1000000000
Retriable: False
Function: checkedMultiply
File: ./velox/common/base/CheckedArithmetic.h
Line: 51
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_14VeloxUserErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZNK8facebook5velox9Timestamp7toNanosEv
# 4  _ZN8facebook5velox12_GLOBAL__N_125gatherFromTimestampBufferERKNS0_10BaseVectorERKNS1_9SelectionE13TimestampUnitRNS0_6BufferE
# 5  _ZN8facebook5velox12_GLOBAL__N_112exportValuesERKNS0_10BaseVectorERKNS1_9SelectionERK12ArrowOptionsR10ArrowArrayPNS0_6memory10MemoryPoolERNS1_24VeloxToArrowBridgeHolderE.constprop.0
# 6  _ZN8facebook5velox12_GLOBAL__N_117exportToArrowImplERKNS0_10BaseVectorERKNS1_9SelectionERK12ArrowOptionsR10ArrowArrayPNS0_6memory10MemoryPoolE
# 7  _ZN8facebook5velox12_GLOBAL__N_117exportToArrowImplERKNS0_10BaseVectorERKNS1_9SelectionERK12ArrowOptionsR10ArrowArrayPNS0_6memory10MemoryPoolE
# 8  _ZN8facebook5velox13exportToArrowERKSt10shared_ptrINS0_10BaseVectorEER10ArrowArrayPNS0_6memory10MemoryPoolERK12ArrowOptions
# 9  _ZN8facebook5velox7parquet6Writer5writeERKSt10shared_ptrINS0_10BaseVectorEE
# 10 _ZN8facebook5velox9connector4hive12HiveDataSink5writeEmSt10shared_ptrINS0_9RowVectorEE
# 11 _ZN8facebook5velox9connector4hive12HiveDataSink10appendDataESt10shared_ptrINS0_9RowVectorEE
# 12 _ZN6gluten13IcebergWriter5writeERKNS_18VeloxColumnarBatchE
# 13 Java_org_apache_gluten_execution_IcebergWriteJniWrapper_write
# 14 0x000077cbb72245da

[rui-mo]

@4ertus2 I'll follow up this issue, thanks.

[FelixYBW]

@4ertus2 this repo is used to hold temp PRs not merged into upstream yet. The issue and pulls in the repo isn't monitored. Can you submit issues to Gluten repo in future?

[jinchengchenghh]

Yes, this issue has been noticed and under developing, could you show the link to resolve it? @PingLiuPing

[rui-mo]

I assume this issue can be fixed by setting the correct parquetWriteTimestampUnit in Gluten. I'll open a fix.

[kapilks]

Can this happen with delta format also or this is restricted to Iceberg?

@rui-mo Are you working on the fix?
Thanks

[jinchengchenghh]

Do you try the newest version? This problem should have been fixed. @kapilks This overflow is only related to Iceberg

[rui-mo]

@jinchengchenghh Would you like to share how this problem was fixed in the newer version? Thanks.