Skip to content

Commit 711a7f7

Browse files
committed
Fix RowSerializationEventConverterTest in opensearch-spark-30 to handle the StructType.toString format change in Spark 3.4+.
Signed-off-by: Sotaro Hikita <bering1814@gmail.com>
1 parent 41dbf1d commit 711a7f7

2 files changed

Lines changed: 6 additions & 3 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
2020
- Fixed global refresh when using dynamic index patterns ([#686](https://github.com/opensearch-project/opensearch-hadoop/pull/686))
2121
- Fixed build failures when downloading Apache project dependencies (Hadoop, Hive, Spark) ([#595](https://github.com/opensearch-project/opensearch-hadoop/pull/595))
2222
- Fixed serverless mode SaveMode.Overwrite failing when document count exceeds scroll size ([#693](https://github.com/opensearch-project/opensearch-hadoop/pull/693))
23+
- Fixed RowSerializationEventConverterTest for Spark 3.4+ StructType.toString() format change ([#XXX](https://github.com/opensearch-project/opensearch-hadoop/pull/XXX))
2324

2425
### Security
2526

spark/sql-30/src/test/scala/org/opensearch/spark/sql/RowSerializationEventConverterTest.scala

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -58,9 +58,11 @@ class RowSerializationEventConverterTest {
5858

5959
val rawEvent = eventConverter.getRawEvent(iaeFailure)
6060

61-
// Scala 2.13 changed what toString() looks like, so can't do an exact match here:
62-
assertTrue(rawEvent.contains("(StructField(field1,StringType,true), " +
63-
"StructField(field2,StringType,true), StructField(field3,StringType,true)),[value1,value2,value3])"))
61+
// Spark 3.4+ changed StructType.toString() format (wrapped in StructType(), no spaces after commas)
62+
assertTrue(rawEvent.contains("StructField(field1,StringType,true)"))
63+
assertTrue(rawEvent.contains("StructField(field2,StringType,true)"))
64+
assertTrue(rawEvent.contains("StructField(field3,StringType,true)"))
65+
assertTrue(rawEvent.contains("[value1,value2,value3]"))
6466
val timestamp = eventConverter.getTimestamp(iaeFailure)
6567
assertTrue(StringUtils.hasText(timestamp))
6668
assertTrue(DateUtils.parseDate(timestamp).getTime.getTime > 1L)

0 commit comments

Comments
 (0)