Skip to content

Commit 8bd8716

Browse files
[GLUTEN-6887][VL] Daily Update Velox Version (2026_09_17) (#13043)
* [GLUTEN-6887][VL] Daily Update Velox Version (dft-2026_09_17) Upstream Velox's New Commits: fdfbc1c1a by Suryadev Sahadevan Rajesh, perf(encoding): Prune SubIntSplit boundary candidates (#19010) 447c01129 by Krishna Pai, misc!: Reconcile fbcode and GitHub source trees (#19057) 16a0305ea by Natasha Sehgal, feat: Add legacy_timestamp_with_timezone query config 5bf9a8ba8 by RindsSchei225e, refactor(nimble): Remove the raw size stats consistency check (#18898) 07fd55cb3 by Suryadev Sahadevan Rajesh, perf(encoding): Skip constant SubIntSplit sections when decoding (#19051) a597b2d39 by Maria Basmanova, feat(rpc): Look up RPC function signatures without stubs (#19052) bf41c616b by abhinavmuk04, fix(trace): Redact credentials from task trace metadata (#19053) 3c38b3161 by Amit Dutta, refactor: Remove backward compatibility macro 52986e2cb by kevinwilfong, fix(fuzzer): Parenthesize derived tables in DuckDB reference SQL (#18931) 3bcef3b0f by tanjialiang, refactor(serializer): Remove kLegacy nimble serialization format (#18872) b08de36f0 by hu hengrui, perf: Prefetch in RowContainer::listRows on x86 db3421f83 by Shruti Shivakumar, fix(cudf): Index BETWEEN column bounds correctly 1074fc019 by Shruti Shivakumar, test(cudf): Disable CPU fallback in GPU tests b83ea4c16 by Ke, feat(fuzzer): Report query-shape coverage 18bb55199 by Jimmy Lu, feat: Propagate worker count and splits through a nested fixed point (#18958) c2366f300 by Suryadev Sahadevan Rajesh, perf(encoding): Direct-indexed frequency counting for narrow SubIntSplit segments (#19009) e647cd3d1 by Suryadev Sahadevan Rajesh, perf(encoding): Trim constant bit planes before the SubIntSplit split DP (#18908) 584c4fa05 by Ke, feat(tablet): Add V2 chunk stats writer 5c21e9f47 by Andrii Rosa, build: Bump FB_OS_VERSION to v2026.09.07.00 (#18968) 032350cc7 by Krishnakanth, fix(json): Collapse duplicate keys in json_parse (keep last) (#18212) c30a0cdc5 by Suryadev Sahadevan Rajesh, feat(encoding): Zigzag-delta pre-transform for SubIntSplit (#18803) ba26f411a by Xiaoxuan, perf(nimble): Optimize nullable materialization (#19024) 5efabda9b by HighW4y2H3ll, refactor: Use string_view data()+size() instead of end() with raw pointers 43941fb64 by LingBin, fix(functions): Avoid string_view iterator-to-pointer use(libc++) b9956f766 by Wenxin Cheng, fix(wave): Fix kernel_ld_test build break from CUDA 13 cuCtxCreate_v4 signature e680a48ae by peterenescu, fix(sparksql): Support to_json template for disabling scalar check (#19030) 025d471ed by Deepthi Bose, feat(function): Update functions to support UNKNOWN type 7ff6c21c7 by Andrii Rosa, build: Migrate S3 test storage from MinIO to pgsty/silo 0e9f3babd by XiaoDu, fix(nimble): Apply layouts to predefined FlatMap keys (#18970) e79a05ee7 by Jimmy Lu, feat: Convergence criterion as a plan sequence (#18854) a917d59bb by tanjialiang, refactor(sequence): Migrate Sequence Storage callers off nimble kLegacy (#18948) ec3ce27fe by Naveen Mahadevuni, refactor(iceberg): Add folly dependency to velox_hive_iceberg_test ae7e5b4be by Maria Basmanova, fix(expression): Extract filters through widening casts ee0d6ec79 by HighW4y2H3ll, refactor: Do not assume `std::string_view::end()` is a raw pointer 0b7e4cf29 by HighW4y2H3ll, refactor: Compare `pos` against `data() + size()` instead of `std::string_view::end()` 6d4fcd76d by Ke, feat(fuzzer): Add configuration coverage framework (#19005) 342ba1a02 by tanjialiang, perf(index): Devirtualize per-stream bufferRef, hoist locals in collectStripeStreamViews (#18988) 13de20e15 by Zac, feat(nimble): Add IndexLookup::KeyCursor for full-index scans 0e156dca5 by tanjialiang, perf(tablet): Devirtualize StripeGroup::streamMajor readAt via dispatch helper (#18987) ea8f4019c by RindsSchei225e, perf(nimble): Fuse the string size scan into the min/max scan b12ed29f1 by Bradley Dice, fix(dwio): Join cache executor before shutdown e3164fdfd by PRASHANT GOLASH, perf(nimble): Optimize FSST reader validation and output materialization (#18937) 48be91e65 by tanjialiang, perf(tablet): Persist DirectDataInput load-path scratch vectors (#18986) 208c508a9 by tanjialiang, perf(tablet): Guard DirectDataInput::load per-group sort with is_sorted (#18985) ca9595f05 by tanjialiang, perf(encoding): MainlyConstant count skips Trivial<Bool> materialize (#18984) 99dbc8cca by Krishnakanth, fix(exec): Gate multi-operator plan-node stats by operator boundary 059a0d648 by macduan, perf(nimble): Add a dense bulk visitor path for ALP decoding 7ae3bfc58 by tanjialiang, perf(encoding): Reuse RLE sliceRuns scratch via ScopedVector + bufferPool (#18983) a4de5771b by tanjialiang, perf(encoding): Slice BlockBitPacking offsets via SliceEncoding delta (#18982) 27fc77e37 by Mariam AlMesfer, fix(sparksql): Make unix_timestamp/to_unix_timestamp ANSI-aware cd2891814 by Yuxuan Chen, Use `string_view::data()` instead of `begin()` for the `memcpy` source f6105f168 by Yuxuan Chen, Use `data()` instead of `std::string_view::begin()` where a raw pointer is required 38436f6fc by Kevin Tang, fix(geospatial): raise the bing tile zoom-span limit as a user error (#18993) 17fbf01d2 by Dong Wang, misc: Fix typos in FileSplitReader.h 6bd17de0b by macduan, perf(dwio): Retrieves the ScanSpec directly when creating column loaders 04eccf71e by tanjialiang, fix(encoding): Handle zero-bit-width FBW in copyPackedBits 563634c34 by tanjialiang, perf(encoding): Apply SliceEncoding value delta to sorted position streams (#18979) d76ab551e by Shrinidhi Joshi, feat(exchange): Add file durable exchange backend (#18871)
1 parent 690c836 commit 8bd8716

5 files changed

Lines changed: 47 additions & 6 deletions

File tree

ep/build-velox/src/get-velox.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,8 @@ set -exu
1818

1919
CURRENT_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd)
2020
VELOX_REPO=https://github.com/IBM/velox.git
21-
VELOX_BRANCH=dft-2026_09_14
22-
VELOX_ENHANCED_BRANCH=ibm-2026_09_14
21+
VELOX_BRANCH=dft-2026_09_17
22+
VELOX_ENHANCED_BRANCH=ibm-2026_09_17
2323
VELOX_HOME=""
2424
RUN_SETUP_SCRIPT=ON
2525
ENABLE_ENHANCED_FEATURES=OFF

gluten-ut/spark34/src/test/scala/org/apache/spark/sql/GlutenDateFunctionsSuite.scala

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@
1616
*/
1717
package org.apache.spark.sql
1818

19+
import org.apache.gluten.utils.BackendTestUtils
20+
21+
import org.apache.spark.SparkException
1922
import org.apache.spark.sql.catalyst.util.DateTimeUtils
2023
import org.apache.spark.sql.functions._
2124
import org.apache.spark.sql.internal.SQLConf
@@ -188,8 +191,16 @@ class GlutenDateFunctionsSuite extends DateFunctionsSuite with GlutenSQLTestsTra
188191
Seq(Row(secs(ts5.getTime)), Row(null)))
189192

190193
// invalid format
194+
// Velox rejects the unsupported specifier 'b' while building the Joda
195+
// formatter, as vanilla Spark does, though it reports a different
196+
// exception type. The legacy formatter returns null instead of failing.
191197
val invalid = df1.selectExpr(s"to_unix_timestamp(x, 'yyyy-MM-dd bb:HH:ss')")
192-
checkAnswer(invalid, Seq(Row(null), Row(null), Row(null), Row(null)))
198+
if (legacyParserPolicy == "legacy" || !BackendTestUtils.isVeloxBackendLoaded()) {
199+
checkAnswer(invalid, Seq(Row(null), Row(null), Row(null), Row(null)))
200+
} else {
201+
val e = intercept[SparkException](invalid.collect())
202+
assert(e.getMessage.contains("Specifier b is not supported"))
203+
}
193204
}
194205
}
195206
}

gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDateFunctionsSuite.scala

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@
1616
*/
1717
package org.apache.spark.sql
1818

19+
import org.apache.gluten.utils.BackendTestUtils
20+
21+
import org.apache.spark.SparkException
1922
import org.apache.spark.sql.catalyst.util.DateTimeUtils
2023
import org.apache.spark.sql.functions._
2124
import org.apache.spark.sql.internal.SQLConf
@@ -203,8 +206,16 @@ class GlutenDateFunctionsSuite extends DateFunctionsSuite with GlutenSQLTestsTra
203206
df.selectExpr("to_unix_timestamp(ts, 'invalid-format')"),
204207
Seq(Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
205208

209+
// Velox rejects the unsupported specifier 'b' while building the Joda
210+
// formatter, as vanilla Spark does, though it reports a different
211+
// exception type. The legacy formatter returns null instead of failing.
206212
val invalid = df1.selectExpr(s"to_unix_timestamp(x, 'yyyy-MM-dd bb:HH:ss')")
207-
checkAnswer(invalid, Seq(Row(null), Row(null), Row(null), Row(null)))
213+
if (legacyParserPolicy == "legacy" || !BackendTestUtils.isVeloxBackendLoaded()) {
214+
checkAnswer(invalid, Seq(Row(null), Row(null), Row(null), Row(null)))
215+
} else {
216+
val e = intercept[SparkException](invalid.collect())
217+
assert(e.getMessage.contains("Specifier b is not supported"))
218+
}
208219
}
209220
}
210221
}

gluten-ut/spark40/src/test/scala/org/apache/spark/sql/GlutenDateFunctionsSuite.scala

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -206,8 +206,16 @@ class GlutenDateFunctionsSuite extends DateFunctionsSuite with GlutenSQLTestsTra
206206
df.selectExpr("to_unix_timestamp(ts, 'invalid-format')"),
207207
Seq(Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
208208

209+
// Velox rejects the unsupported specifier 'b' while building the Joda
210+
// formatter, as vanilla Spark does, though it reports a different
211+
// exception type. The legacy formatter returns null instead of failing.
209212
val invalid = df1.selectExpr(s"to_unix_timestamp(x, 'yyyy-MM-dd bb:HH:ss')")
210-
checkAnswer(invalid, Seq(Row(null), Row(null), Row(null), Row(null)))
213+
if (legacyParserPolicy == "legacy" || !BackendTestUtils.isVeloxBackendLoaded()) {
214+
checkAnswer(invalid, Seq(Row(null), Row(null), Row(null), Row(null)))
215+
} else {
216+
val e = intercept[SparkException](invalid.collect())
217+
assert(e.getMessage.contains("Specifier b is not supported"))
218+
}
211219
}
212220
}
213221
}

gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenDateFunctionsSuite.scala

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@
1616
*/
1717
package org.apache.spark.sql
1818

19+
import org.apache.gluten.utils.BackendTestUtils
20+
21+
import org.apache.spark.SparkException
1922
import org.apache.spark.sql.catalyst.util.DateTimeUtils
2023
import org.apache.spark.sql.functions._
2124
import org.apache.spark.sql.internal.SQLConf
@@ -203,8 +206,16 @@ class GlutenDateFunctionsSuite extends DateFunctionsSuite with GlutenSQLTestsTra
203206
df.selectExpr("to_unix_timestamp(ts, 'invalid-format')"),
204207
Seq(Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
205208

209+
// Velox rejects the unsupported specifier 'b' while building the Joda
210+
// formatter, as vanilla Spark does, though it reports a different
211+
// exception type. The legacy formatter returns null instead of failing.
206212
val invalid = df1.selectExpr(s"to_unix_timestamp(x, 'yyyy-MM-dd bb:HH:ss')")
207-
checkAnswer(invalid, Seq(Row(null), Row(null), Row(null), Row(null)))
213+
if (legacyParserPolicy == "legacy" || !BackendTestUtils.isVeloxBackendLoaded()) {
214+
checkAnswer(invalid, Seq(Row(null), Row(null), Row(null), Row(null)))
215+
} else {
216+
val e = intercept[SparkException](invalid.collect())
217+
assert(e.getMessage.contains("Specifier b is not supported"))
218+
}
208219
}
209220
}
210221
}

0 commit comments

Comments
 (0)