HIVE-29375: FULL OUTER JOIN is failing with Unexpected hash table key type DATE #6239

Aggarwal-Raghav · 2025-12-17T08:46:31Z

What changes were proposed in this pull request?

Check HIVE-29375 for repro and stacktrace

Why are the changes needed?

To make DATE type support for full outer join, which is, getting converted to map join because of hive.optimize.dynamic.partition.hashjoin=true;

Does this PR introduce any user-facing change?

No

How was this patch tested?

Wrote a new q file and will see CI output

mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=vector_full_outer_join_date.q -Drat.skip -Dtest.output.overwrite -Pitests -pl itests/qtest

… type DATE

Aggarwal-Raghav · 2025-12-17T08:54:19Z

Explanation:

In Vectorizer.java the hashTableKeyTypeis getting set as DATE for DATE column type

hive/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java

Line 3455 in 5410b73

hashTableKeyType = HashTableKeyType.DATE;

But the DATE type is not present VectorMapJoinOuterGenerateResultOperator in :

hive/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinOuterGenerateResultOperator.java

Line 821 in c237510

switch (hashTableKeyType) {

For Double and Timestamp Columns, they are working without the patch as well because the default hashTableKeyType is MULTI_KEY

hive/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java

Line 3429 in 5410b73

hashTableKeyType = HashTableKeyType.MULTI_KEY;

Aggarwal-Raghav · 2025-12-17T16:27:12Z

Will address sonar issues post review comments. I',m willing to move the if-else pattern + old switch style with jdk21 'switch expressions'. Reviewers can let me know. will file separate jira to migrate especially in vectorization.

Aggarwal-Raghav · 2025-12-19T07:08:57Z

CC @zabetak , can you please help with the review?

zabetak · 2025-12-19T09:38:37Z

Hey @Aggarwal-Raghav, I am on holidays till Dec 29, with intermittent and not stable internet connection. Not sure if I will find time to check this before then.

Aggarwal-Raghav · 2025-12-19T09:41:47Z

Hey @Aggarwal-Raghav, I am on holidays till Dec 29, with intermittent and not stable internet connection. Not sure if I will find time to check this before then.

No worries. Enjoy !! 😅

mdayakar · 2025-12-19T13:44:57Z

Hi @Aggarwal-Raghav ,
I am not a vectorization feature expert but as per code changes I feel you missed below places adding DATE HashTableKeyType, please check.

Also I could see there are many UT test cases, may be you can add more test cases related to DATE HashTableKeyType. For example MapJoinTestConfig.java

Aggarwal-Raghav · 2025-12-20T17:53:00Z

Hi @Aggarwal-Raghav , I am not a vectorization feature expert but as per code changes I feel you missed below places adding DATE HashTableKeyType, please check.

CheckFastRowHashMap.java

VectorMapJoinOptimizedLongHashMap.java

VectorMapJoinOptimizedLongHashMap.java

Also I could see there are many UT test cases, may be you can add more test cases related to DATE HashTableKeyType. For example MapJoinTestConfig.java

Thanks for the thorough review @mdayakar , i checked the above places you mentioned. For [2] and [3], it is already handled in this PR. For [1] and MapJoinTestConfig.java which are test classes, I'll evaluate on adding more test cases for DATE type and update in sometime.

zabetak

Changes LGTM! I left some comments for minor improvements and potentially few unit tests more.

(The Sonar issues are not worth fixing; at least not now)

ql/src/test/queries/clientpositive/vector_full_outer_join_date.q

...g/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashMap.java

ql/src/test/queries/clientpositive/vector_full_outer_join_date.q

...test/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/TestVectorMapJoinFastRowHashMap.java

zabetak · 2026-01-02T19:01:57Z

ql/src/test/org/apache/hadoop/hive/ql/exec/vector/mapjoin/MapJoinTestConfig.java

+      case DATE:
+        hashTableKeyType = HashTableKeyType.DATE;
+        break;


Do we have unit tests exploiting this config? Do we need to add something in TestMapJoinOperator?

Will add a testcase testDate0 in TestMapJoinOperator. As testString0 makes use of the DATE type, but it does so as a Value column, not as a Join Key.

Well if its not a Join key then the tests are not directly targeting the fix so not sure how much are needed. Do we have unit tests for join keys with different types somewhere in the repo?

In TestMapJoinOperator.java, the following are the join keys for UT. The bigTableKeyColumnNums = new int[] {0}; determines the join column.

- TestLong0: long - testLong0_NoRegularKeys: long - testLong1: int - testLong2: short - testLong3: int - testLong3_NoRegularKeys: int - testLong4: int - testLong5: long - testLong6: long - testDate0: date - testMultiKey0: short, int - testMultiKey1: timestamp, short, string - testMultiKey2: long, short, string - testMultiKey3: date, byte - testString0: string - testString1: binary - testString2: string

Aggarwal-Raghav · 2026-01-03T15:17:43Z

Thanks for the review @zabetak , I'll accomodate the suggestions.

Aggarwal-Raghav · 2026-01-04T18:26:50Z

Will push changes based on your input on #6239 (comment)

ql/src/test/queries/clientpositive/vector_full_outer_join_date.q

zabetak · 2026-01-05T09:36:23Z

ql/src/test/org/apache/hadoop/hive/ql/exec/vector/mapjoin/MapJoinTestConfig.java

+      case DATE:
+        hashTableKeyType = HashTableKeyType.DATE;
+        break;


Well if its not a Join key then the tests are not directly targeting the fix so not sure how much are needed. Do we have unit tests for join keys with different types somewhere in the repo?

...g/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashMap.java

sonarqubecloud · 2026-01-05T11:22:12Z

Quality Gate passed

Issues
17 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.6% Duplication on New Code

See analysis details on SonarQube Cloud

zabetak

@Aggarwal-Raghav Can you please clarify what kind of coverage we are adding with the unit tests:

TestVectorMapJoinFastRowHashMap#testDateRowsExact
TestMapJoinOperator#testDate0

It seems that none of the above goes through the problematic code.

Aggarwal-Raghav · 2026-01-06T15:23:57Z

@Aggarwal-Raghav Can you please clarify what kind of coverage we are adding with the unit tests:

TestVectorMapJoinFastRowHashMap#testDateRowsExact

TestMapJoinOperator#testDate0

It seems that none of the above goes through the problematic code.

That's strange, it was failing for me. Let me re-check and confirm on that.

Aggarwal-Raghav · 2026-01-06T15:41:49Z

Please test with just these files from this PR:

 M ql/src/test/org/apache/hadoop/hive/ql/exec/vector/mapjoin/MapJoinTestConfig.java
 M ql/src/test/org/apache/hadoop/hive/ql/exec/vector/mapjoin/TestMapJoinOperator.java
 M ql/src/test/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/TestVectorMapJoinFastRowHashMap.java
?? ql/src/test/queries/clientpositive/vector_full_outer_join_date.q
?? ql/src/test/results/clientpositive/llap/vector_full_outer_join_date.q.out

MapJoinTestConfig changes are required otherwise the testDate0 is using MULTI_KEY as default.

For TestVectorMapJoinFastRowHashMap, ensure you have not taken CheckFastRowHashMap changes.
Both MapJoinTestConfig and CheckFastRowHashMap are supporting test classes. They don't contains test/UT as such

zabetak · 2026-01-06T16:02:04Z

Thanks for the detailed explanation @Aggarwal-Raghav! My workspace was not clean and didn't notice that I was missing MapJoinTestConfig sorry for the confusion.

zabetak

Thanks for your patience. I will merge the PR soon!

Aggarwal-Raghav · 2026-01-06T16:03:36Z

Thanks for the detailed explanation @Aggarwal-Raghav! My workspace was not clean and didn't notice that I was missing MapJoinTestConfig sorry for the confusion.

No worries. Thanks for the thorough review.

zabetak · 2026-01-07T07:42:16Z

Many thanks for the PR @Aggarwal-Raghav and @mdayakar for the review!

HIVE-29375: FULL OUTER JOIN is failing with Unexpected hash table key…

90033cf

… type DATE

asf-ci-hive added the tests pending label Dec 17, 2025

asf-ci-hive added tests passed and removed tests pending labels Dec 17, 2025

Add DATE support in test code as well

b2674e5

asf-ci-hive added tests pending tests passed and removed tests passed tests pending labels Dec 22, 2025

zabetak approved these changes Jan 2, 2026

View reviewed changes

address review comments

ce2b285

zabetak approved these changes Jan 5, 2026

View reviewed changes

asf-ci-hive added tests pending and removed tests passed labels Jan 5, 2026

asf-ci-hive added tests passed and removed tests pending labels Jan 5, 2026

zabetak reviewed Jan 6, 2026

View reviewed changes

zabetak approved these changes Jan 6, 2026

View reviewed changes

zabetak merged commit 0978d70 into apache:master Jan 7, 2026
2 checks passed

HIVE-29375: FULL OUTER JOIN is failing with Unexpected hash table key type DATE #6239

HIVE-29375: FULL OUTER JOIN is failing with Unexpected hash table key type DATE #6239

Uh oh!

Conversation

Aggarwal-Raghav commented Dec 17, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Aggarwal-Raghav commented Dec 17, 2025

Uh oh!

Aggarwal-Raghav commented Dec 17, 2025

Uh oh!

Aggarwal-Raghav commented Dec 19, 2025

Uh oh!

zabetak commented Dec 19, 2025

Uh oh!

Aggarwal-Raghav commented Dec 19, 2025

Uh oh!

mdayakar commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Aggarwal-Raghav commented Dec 20, 2025

Uh oh!

zabetak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zabetak Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

zabetak Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav commented Jan 3, 2026

Uh oh!

Aggarwal-Raghav commented Jan 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

zabetak Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sonarqubecloud bot commented Jan 5, 2026

Quality Gate passed

Uh oh!

zabetak left a comment

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav commented Jan 6, 2026

Uh oh!

Aggarwal-Raghav commented Jan 6, 2026

Uh oh!

zabetak commented Jan 6, 2026

Uh oh!

zabetak left a comment

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav commented Jan 6, 2026

Uh oh!

Uh oh!

zabetak commented Jan 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

mdayakar commented Dec 19, 2025 •

edited

Loading

Aggarwal-Raghav commented Jan 4, 2026 •

edited

Loading