Commit ce612c6
Add Iceberg partition transforms: year/month/day/hour (#3903)
contributes to NVIDIA/spark-rapids#13384
contributes to NVIDIA/spark-rapids#13385
contributes to NVIDIA/spark-rapids#13386
contributes to NVIDIA/spark-rapids#13387
Add Iceberg partition transforms: year/month/day/hour.
CPU code is in `iceberg org.apache.iceberg.util.DateTimeUtil`,
[link](https://github.com/apache/iceberg/blob/apache-iceberg-1.6.1/api/src/main/java/org/apache/iceberg/util/DateTimeUtil.java)
### details
We can not use `ColumnView.year`: input MUST be timestamp type instead
of date type.
This PR adds the following four APIs:
```java
/**
* Calculates the difference in years between the epoch year (1970) and the
* given date/timestamp column. E.g.: for date '1971-01-01', the result would be
* 1: (1 year after epoch year)
*
* @param input The input date/timestamp column.
* @return A column of type INT32 containing the year differences from epoch.
*/
public static ColumnVector toYears(ColumnView input)
/**
* Calculates the difference in months between the epoch month (1970-01) and the
* given date/timestamp column. E.g.: for date '1971-02-01', the result would be
* 13: (1 year and 1 month after epoch month)
*
* @param input The input date/timestamp column.
* @return A column of type INT32 containing the month differences from epoch.
*/
public static ColumnVector toMonths(ColumnView input)
/**
* Calculates the difference in days between the epoch day (1970-01-01) and the
* given date/timestamp column. E.g.: for date '1970-01-21', the result would be
* 20: (20 days after epoch day)
*
* @param input The input date/timestamp column.
* @return A column of type Date.
*/
public static ColumnVector toDays(ColumnView input)
/**
* Calculates the difference in hours between the epoch hour
* (1970-01-01T00:00:00) and the given timestamp column.
* E.g.: for timestamp '1970-01-01 01:00:00', the result would be 1
* (1 hour after epoch hour)
*
* @param timestamp The input timestamp column.
* @return A column of type INT32 containing the hour differences from epoch.
*/
public static ColumnVector toHours(ColumnView input)
```
Signed-off-by: Chong Gao <res_life@163.com>
---------
Signed-off-by: Chong Gao <res_life@163.com>
Co-authored-by: Chong Gao <res_life@163.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>1 parent aeffc02 commit ce612c6
File tree
9 files changed
+817
-1
lines changed- src
- main
- cpp
- src
- java/com/nvidia/spark/rapids/jni/iceberg
- test/java/com/nvidia/spark/rapids/jni/iceberg
9 files changed
+817
-1
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
112 | 112 | | |
113 | 113 | | |
114 | 114 | | |
| 115 | + | |
115 | 116 | | |
116 | 117 | | |
117 | 118 | | |
| |||
194 | 195 | | |
195 | 196 | | |
196 | 197 | | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
197 | 204 | | |
198 | 205 | | |
199 | 206 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
198 | 198 | | |
199 | 199 | | |
200 | 200 | | |
| 201 | + | |
201 | 202 | | |
202 | 203 | | |
203 | 204 | | |
| |||
235 | 236 | | |
236 | 237 | | |
237 | 238 | | |
| 239 | + | |
238 | 240 | | |
239 | 241 | | |
240 | 242 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
| 20 | + | |
19 | 21 | | |
20 | 22 | | |
21 | 23 | | |
| |||
230 | 232 | | |
231 | 233 | | |
232 | 234 | | |
233 | | - | |
| 235 | + | |
234 | 236 | | |
235 | 237 | | |
236 | 238 | | |
| |||
0 commit comments