Skip to content

Conversation

liyubin117
Copy link
Contributor

@liyubin117 liyubin117 commented Sep 28, 2025

Purpose

Skip processed sequence group fields to improve performance of PartialUpdateMergeFunction

  1. skip null sequence group fields
  2. skip sequence group fields have been updated at once
  3. move seqComparator != null check forward for common case optimization

Tests

  1. PartialUpdateMergeFunctionTest
  2. PartialUpdateMergeFunctionBenchmark
    new
Running benchmark: partial-update-benchmark
  Running case: updateWithSequenceGroup
Iteration 0 took 13835399 microseconds
Iteration 1 took 13355838 microseconds
Iteration 2 took 13048940 microseconds
Iteration 3 took 13016992 microseconds
Iteration 4 took 13141102 microseconds
  Stopped after 5 iterations, 66398 ms

OpenJDK 64-Bit Server VM 1.8.0_412-b08 on Mac OS X 14.4.1
Apple M1 Pro
partial-update-benchmark:                                                                            Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_partial-update-benchmark_updateWithSequenceGroup                                           13017 / 13280           3072.9            325.4       1.0X

Running benchmark: partial-update-benchmark
  Running case: retractWithSequenceGroup
Iteration 0 took 12762749 microseconds
Iteration 1 took 12572677 microseconds
Iteration 2 took 12441134 microseconds
Iteration 3 took 12490136 microseconds
Iteration 4 took 12677572 microseconds
  Stopped after 5 iterations, 62944 ms

OpenJDK 64-Bit Server VM 1.8.0_412-b08 on Mac OS X 14.4.1
Apple M1 Pro
partial-update-benchmark:                                                                            Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_partial-update-benchmark_retractWithSequenceGroup                                          12441 / 12589           3215.1            311.0       1.0X

Running benchmark: partial-update-benchmark
  Running case: updateWithEmptySequenceGroup
Iteration 0 took 10842099 microseconds
Iteration 1 took 10364416 microseconds
Iteration 2 took 10588347 microseconds
Iteration 3 took 10492397 microseconds
Iteration 4 took 10618167 microseconds
  Stopped after 5 iterations, 52905 ms

OpenJDK 64-Bit Server VM 1.8.0_412-b08 on Mac OS X 14.4.1
Apple M1 Pro
partial-update-benchmark:                                                                            Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_partial-update-benchmark_updateWithEmptySequenceGroup                                      10364 / 10581           3859.4            259.1       1.0X

old

Running benchmark: partial-update-benchmark
  Running case: updateWithSequenceGroup
Iteration 0 took 14191378 microseconds
Iteration 1 took 14021648 microseconds
Iteration 2 took 14039543 microseconds
Iteration 3 took 13882215 microseconds
Iteration 4 took 13819146 microseconds
  Stopped after 5 iterations, 69953 ms

OpenJDK 64-Bit Server VM 1.8.0_412-b08 on Mac OS X 14.4.1
Apple M1 Pro
partial-update-benchmark:                                                                            Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_partial-update-benchmark_updateWithSequenceGroup                                           13819 / 13991           2894.5            345.5       1.0X

Running benchmark: partial-update-benchmark
  Running case: retractWithSequenceGroup
Iteration 0 took 14462734 microseconds
Iteration 1 took 14674714 microseconds
Iteration 2 took 15522533 microseconds
Iteration 3 took 15111800 microseconds
Iteration 4 took 14651207 microseconds
  Stopped after 5 iterations, 74422 ms

OpenJDK 64-Bit Server VM 1.8.0_412-b08 on Mac OS X 14.4.1
Apple M1 Pro
partial-update-benchmark:                                                                            Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_partial-update-benchmark_retractWithSequenceGroup                                          14463 / 14885           2765.7            361.6       1.0X

Running benchmark: partial-update-benchmark
  Running case: updateWithEmptySequenceGroup
Iteration 0 took 10743736 microseconds
Iteration 1 took 11192192 microseconds
Iteration 2 took 12587153 microseconds
Iteration 3 took 11009585 microseconds
Iteration 4 took 10661533 microseconds
  Stopped after 5 iterations, 56194 ms

OpenJDK 64-Bit Server VM 1.8.0_412-b08 on Mac OS X 14.4.1
Apple M1 Pro
partial-update-benchmark:                                                                            Best/Avg Time(ms)    Row Rate(K/s)      Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_partial-update-benchmark_updateWithEmptySequenceGroup                                      10662 / 11239           3751.8            266.5       1.0X

@liyubin117 liyubin117 force-pushed the fasten_partial_update branch 2 times, most recently from da97bfe to b393b2a Compare September 28, 2025 06:12
@liyubin117
Copy link
Contributor Author

liyubin117 commented Sep 28, 2025

@JingsongLi All CI succeeded extra non-related hive metastore issue, PTAL, thanks :)
CI succeeded in my repo.
image

@liyubin117 liyubin117 force-pushed the fasten_partial_update branch from b393b2a to 1afe5a6 Compare September 29, 2025 02:21
@liyubin117
Copy link
Contributor Author

@JingsongLi CI succeeded

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant