How to support Spark's TimestampNTZ type

[rui-mo]

Although Presto and Spark expose similarly named timestamp types, their semantics differ significantly. Since both engines reuse the Timestamp implementation in Velox, it is necessary to clearly understand the differences at the type level.

Presto’s TimestampType type is typically stored as a long value representing milliseconds, and it is interpreted with respect to a session time zone. In contrast, Spark’s TimestampType is stored as a long value representing microseconds, also timezone-aware. Both reuse the same underlying Velox representation Timestamp, and they differ only in precision.

Presto’s timestamp with time zone encodes a value as UTC milliseconds plus a time zone key. This allows each row to represent a fully qualified point in time. In Velox, this is supported via a dedicated Presto custom type.

Spark’s TimestampNTZType is also stored as a long value in microseconds, but it is timezone–agnostic. As such, it is not semantically equivalent to any of above timestamp types.

When it comes to storage in Apache Parquet, the differences become more apparent. Parquet defines timestamp columns using parameters such as isAdjustedToUTC and timeUnit. Spark relies on these annotations for interpretation:

• When isAdjustedToUTC = true, Spark interprets the column as TimestampType.
• When isAdjustedToUTC = false, Spark interprets it as TimestampNTZType.

For example:
Timestamp(isAdjustedToUTC = false, timeUnit = microseconds)
This means the data is stored as microseconds and will be read as TimestampNTZType in Spark.

To support this type in Gluten, we initially attempted to add a custom TimestampNTZType in Velox specifically for Spark. However, we encountered several issues:

1. We need a custom Arrow conversion: feat: Support custom type in Arrow schema import and export #16615.
2. We need a custom Parquet reader implementation: feat: Add custom Parquet read support for Spark TimestampNTZ Type #16789.
   Reading these values as Velox’s Timestamp(int64_t seconds, uint64_t nanos) and converting them into int64_t would introduce extra overhead due to the multiple conversion steps: micros(int64_t) → seconds + nanos → micros(int64_t).

Based on above, an alternative approach we are considering is extending the existing Velox Timestamp type to add a TimestampUTC type, similar to how Time and TimeUTC are handled. This would allow us to reuse most of the existing Timestamp reader implementations, reducing complexity and overhead.

Looking forward to your thoughts and insights, thank you.

cc: @Yuhta @mbasmanova @pedroerp

[rui-mo]

Hi @Yuhta @mbasmanova @pedroerp, based on the above context, what do you think about the proposal to add the TimestampUTC type?

[Yuhta]

Is isAdjustedToUTC affecting the casting and udf behaviors? kLocalTime in TIME is affecting the casting by consider local time zone, is there similar difference between Timestamp and TimestampUTC?

[rui-mo]

The two types also differ in how they handle time zones in functions and cast operations. For example, in Spark, casting a string to Timestamp requires timezone adjustment, whereas for TimestampUTC, no timezone conversion is applied.

velox/velox/functions/sparksql/specialforms/SparkCastHooks.cpp

Lines 45 to 46 in a6b99fc

	return util::fromParsedTimestampWithTimeZone(
	conversionResult.value(), sessionTimezone);

[rui-mo]

Hi all,
Support for TimestampNTZ functions and cast conversions is a high priority in Gluten, and it depends on adding the corresponding type support in Velox. I’d like to know if you have any preference for how this should be implemented in Velox.
The current approach is to implement it as a custom type for Spark, which requires a hook for custom Parquet reader support. Another option is to extend the existing Timestamp type and introduce a TimestampUTC type.
Please let me know how you would like us to proceed. Thanks!

cc: @mbasmanova @Yuhta @pedroerp

[mbasmanova]

Thanks for the detailed writeup, @rui-mo.

If I understand the proposal correctly: add TimestampUTC as a new built-in logical type backed by the Timestamp physical type, following the same pattern as TIME / TIME_MICRO_UTC.

This would give you:

• Reuse of existing Timestamp vector operations, comparison, hashing, and reader/writer code
• A way to distinguish timezone-aware vs timezone-agnostic timestamps at the type level
• Naming consistent with the existing TimeUTC convention

Note that the Parquet conversion overhead (micros ↔ seconds+nanos) remains with this approach — same as with the custom type. But that seems unavoidable given the Timestamp physical representation.

If that's what you have in mind, I'm fine with this approach.

[rui-mo]

@mbasmanova Thanks for the confirmation. Since both the custom type and Parquet use int64_t, conversion overhead can be reduced. Reading as Timestamp adds two conversions (bigint ↔ timestamp), whereas using TimestampUTC reduces it to just one (bigint → timestamp).

I’ll proceed with extending the Timestamp type, thanks!