Skip to content

Commit 05ae3f3

Browse files
QuafadasCopilot
Quafadas
and
Copilot
authored
Grp ops (#58)
* groupOps * testing * benching * Optimize splitAmntFast by eliminating redundant array copies in group operations (#59) * Initial plan * Optimize group operations: eliminate array copies, add in-place variants - Eliminate unnecessary array copies in applyGroupCumSumFast and applyGroupDiffFast - Add groupCumSumInPlace and groupDiffInPlace for 34-53% performance gains - Optimize group lookups by caching values to avoid redundant array access - Add isolated benchmark for group operations performance testing Performance improvements (10k elements, 100 groups): - groupCumSum: 34% faster with in-place variant (11.6μs → 7.6μs) - groupDiff: 53% faster with in-place variant (10.3μs → 4.9μs) Co-authored-by: Quafadas <24899792+Quafadas@users.noreply.github.com> * Remove ineffective group lookup optimization from code review feedback Code review correctly identified that caching group values in the while condition doesn't reduce array accesses since the array is still accessed on every iteration. Co-authored-by: Quafadas <24899792+Quafadas@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Quafadas <24899792+Quafadas@users.noreply.github.com> * Testing splitAmnt * split works * . * formatting * format * remove days * . * arg * . * seems to pass * . * . * . * . * . * . * . * . * . * benchmark * format * . * . * . * . * format * . * . --------- Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: Quafadas <24899792+Quafadas@users.noreply.github.com>
1 parent 68adc3c commit 05ae3f3

92 files changed

Lines changed: 1824 additions & 1566 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/copilot-instructions.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ Answer all questions in the style of a friendly colleague that is an expert in l
1212

1313
Mill may be found via it's wrapper script `./mill` in the root of the repository. For example `./mill vecxt.__.compile` will compile the JVM, JS and native targets.
1414

15+
If you are on windows, use `mill`, not `./mill`.
16+
1517
Each module contains it's own build definition in the package.mill file in it's module directory.
1618

1719
- BUILDS: Mill cold compilation takes 2 minutes or so. Stay patient!
@@ -33,7 +35,7 @@ vecxt/
3335
├── styleguide.md # Coding style guidelines
3436
├── benchmarks/ # Benchmarking code - not published, may be run in CI on request
3537
├── experiments/ # Not published, inlined experiments - use this as a sandbox
36-
├── vecxtensions/ # Published module with experiments / concepts which may not be suitable for main module
38+
├── vecxtensions/ # Published experimental / concepts which may not be suitable for main module
3739
│ ├── src/ # Cross-platform shared source code
3840
│ ├── src-jvm/ # JVM-specific implementations (SIMD Vector API)
3941
│ ├── src-js/ # JavaScript-specific implementations

‎.github/workflows/ci.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ jobs:
4545
cache: npm
4646

4747
- name: formatCheck
48+
if: matrix.project == 'jvm'
4849
run: ./mill mill.scalalib.scalafmt.ScalafmtModule/checkFormatAll __.sources
4950

5051
- if: matrix.project == 'js'
@@ -64,6 +65,9 @@ jobs:
6465
- name: Test
6566
run: ./mill vecxt.${{ matrix.project }}.test
6667

68+
- name: Test
69+
run: ./mill vecxtensions.${{ matrix.project }}.test
70+
6771
- name: Laws Test
6872
if: matrix.project == 'jvm'
6973
run: ./mill laws.${{ matrix.project }}.test

‎benchmark/package.mill‎

Lines changed: 28 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -9,34 +9,34 @@ object `package` extends JmhModule with ScalaModule:
99
def scalaVersion = build.vecxt.jvm.scalaVersion
1010
def jmhCoreVersion = "1.37"
1111
override def forkArgs: T[Seq[String]] = super.forkArgs() ++ build.vecIncubatorFlag
12-
override def moduleDeps: Seq[JavaModule] = Seq(build.vecxt.jvm)
12+
override def moduleDeps: Seq[JavaModule] = Seq(build.vecxt.jvm, build.vecxtensions.jvm)
1313
def enableBsp = false
1414

15-
// override def generateBenchmarkSources = T{
16-
// val dest = T.ctx().dest
17-
18-
// val forkedArgs = forkArgs().toSeq
19-
20-
// val sourcesDir = dest / "jmh_sources"
21-
// val resourcesDir = dest / "jmh_resources"
22-
23-
// os.remove.all(sourcesDir)
24-
// os.makeDir.all(sourcesDir)
25-
// os.remove.all(resourcesDir)
26-
// os.makeDir.all(resourcesDir)
27-
28-
// Jvm.runSubprocess(
29-
// "org.openjdk.jmh.generators.bytecode.JmhBytecodeGenerator",
30-
// (runClasspath() ++ generatorDeps()).map(_.path),
31-
// mainArgs = Seq(
32-
// compile().classes.path.toString,
33-
// sourcesDir.toString,
34-
// resourcesDir.toString,
35-
// "default"
36-
// ),
37-
// jvmArgs = forkedArgs
38-
// )
39-
40-
// (sourcesDir, resourcesDir)
41-
// }
15+
// override def generateBenchmarkSources = T{
16+
// val dest = T.ctx().dest
17+
18+
// val forkedArgs = forkArgs().toSeq
19+
20+
// val sourcesDir = dest / "jmh_sources"
21+
// val resourcesDir = dest / "jmh_resources"
22+
23+
// os.remove.all(sourcesDir)
24+
// os.makeDir.all(sourcesDir)
25+
// os.remove.all(resourcesDir)
26+
// os.makeDir.all(resourcesDir)
27+
28+
// Jvm.runSubprocess(
29+
// "org.openjdk.jmh.generators.bytecode.JmhBytecodeGenerator",
30+
// (runClasspath() ++ generatorDeps()).map(_.path),
31+
// mainArgs = Seq(
32+
// compile().classes.path.toString,
33+
// sourcesDir.toString,
34+
// resourcesDir.toString,
35+
// "default"
36+
// ),
37+
// jvmArgs = forkedArgs
38+
// )
39+
40+
// (sourcesDir, resourcesDir)
41+
// }
4242
end `package`

‎benchmark/src/and.scala‎

Lines changed: 0 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,3 @@
1-
/*
2-
* Copyright 2020, 2021, Ludovic Henry
3-
*
4-
* Permission is hereby granted, free of charge, to any person obtaining a copy
5-
* of this software and associated documentation files (the "Software"), to deal
6-
* in the Software without restriction, including without limitation the rights
7-
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
8-
* copies of the Software, and to permit persons to whom the Software is
9-
* furnished to do so, subject to the following conditions:
10-
*
11-
* The above copyright notice and this permission notice shall be included in
12-
* all copies or substantial portions of the Software.
13-
*
14-
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
15-
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
16-
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
17-
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
18-
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
19-
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
20-
* SOFTWARE.
21-
*
22-
* Please contact git@ludovic.dev or visit ludovic.dev if you need additional
23-
* information or have any questions.
24-
*/
25-
261
package vecxt.benchmark
272

283
import org.openjdk.jmh.annotations.*

‎benchmark/src/argmax.scala‎

Lines changed: 0 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,3 @@
1-
/*
2-
* Copyright 2020, 2021, Ludovic Henry
3-
*
4-
* Permission is hereby granted, free of charge, to any person obtaining a copy
5-
* of this software and associated documentation files (the "Software"), to deal
6-
* in the Software without restriction, including without limitation the rights
7-
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
8-
* copies of the Software, and to permit persons to whom the Software is
9-
* furnished to do so, subject to the following conditions:
10-
*
11-
* The above copyright notice and this permission notice shall be included in
12-
* all copies or substantial portions of the Software.
13-
*
14-
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
15-
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
16-
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
17-
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
18-
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
19-
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
20-
* SOFTWARE.
21-
*
22-
* Please contact git@ludovic.dev or visit ludovic.dev if you need additional
23-
* information or have any questions.
24-
*/
25-
261
package vecxt.benchmark
272

283
import org.openjdk.jmh.annotations.*

‎benchmark/src/countTrue.scala‎

Lines changed: 0 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,3 @@
1-
/*
2-
* Copyright 2020, 2021, Ludovic Henry
3-
*
4-
* Permission is hereby granted, free of charge, to any person obtaining a copy
5-
* of this software and associated documentation files (the "Software"), to deal
6-
* in the Software without restriction, including without limitation the rights
7-
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
8-
* copies of the Software, and to permit persons to whom the Software is
9-
* furnished to do so, subject to the following conditions:
10-
*
11-
* The above copyright notice and this permission notice shall be included in
12-
* all copies or substantial portions of the Software.
13-
*
14-
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
15-
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
16-
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
17-
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
18-
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
19-
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
20-
* SOFTWARE.
21-
*
22-
* Please contact git@ludovic.dev or visit ludovic.dev if you need additional
23-
* information or have any questions.
24-
*/
25-
261
package vecxt.benchmark
272

283
import org.openjdk.jmh.annotations.*

‎benchmark/src/groupOpsBenchmark.scala‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
/*
2+
* Benchmark for isolated group operations (groupCumSum, groupDiff, groupSum)
3+
* Testing performance of these hot-path operations
4+
*/
5+
6+
package vecxt.benchmark
7+
8+
import org.openjdk.jmh.annotations.*
9+
import org.openjdk.jmh.infra.Blackhole
10+
import scala.compiletime.uninitialized
11+
import vecxtensions.*
12+
import java.util.Random
13+
import java.util.concurrent.TimeUnit
14+
15+
@State(Scope.Thread)
16+
@BenchmarkMode(Array(Mode.AverageTime))
17+
@OutputTimeUnit(TimeUnit.NANOSECONDS)
18+
@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
19+
@Measurement(iterations = 10, time = 1, timeUnit = TimeUnit.SECONDS)
20+
@Fork(1)
21+
class GroupOpsBenchmark:
22+
23+
@Param(Array("100", "1000", "10000", "100000"))
24+
var size: String = uninitialized
25+
26+
@Param(Array("10", "100", "1000"))
27+
var groupSize: String = uninitialized
28+
29+
var groups: Array[Int] = uninitialized
30+
var values: Array[Double] = uninitialized
31+
var valuesCopy1: Array[Double] = uninitialized
32+
var valuesCopy2: Array[Double] = uninitialized
33+
34+
@Setup(Level.Invocation)
35+
def setup: Unit =
36+
val n = size.toInt
37+
val avgGroupSize = groupSize.toInt
38+
val numGroups = math.max(1, n / avgGroupSize)
39+
40+
val random = new Random(42) // Fixed seed for reproducibility
41+
42+
// Generate sorted groups array with realistic group sizes
43+
groups = new Array[Int](n)
44+
var currentGroup = 0
45+
var i = 0
46+
while i < n do
47+
val remainingElements = n - i
48+
val remainingGroups = numGroups - currentGroup
49+
val targetGroupSize =
50+
if remainingGroups > 0 then remainingElements / remainingGroups
51+
else remainingElements
52+
53+
val actualGroupSize = math.max(1, targetGroupSize + random.nextInt(avgGroupSize / 2 + 1) - avgGroupSize / 4)
54+
val groupEnd = math.min(i + actualGroupSize, n)
55+
56+
while i < groupEnd do
57+
groups(i) = currentGroup
58+
i += 1
59+
end while
60+
61+
currentGroup += 1
62+
end while
63+
64+
// Generate random values
65+
values = Array.fill(n)(random.nextDouble() * 100.0)
66+
valuesCopy1 = values.clone()
67+
valuesCopy2 = values.clone()
68+
end setup
69+
70+
@Benchmark
71+
def benchGroupCumSum(bh: Blackhole): Unit =
72+
val result = groupCumSum(groups, values)
73+
bh.consume(result)
74+
end benchGroupCumSum
75+
76+
@Benchmark
77+
def benchGroupCumSumInPlace(bh: Blackhole): Unit =
78+
groupCumSumInPlace(groups, valuesCopy1)
79+
bh.consume(valuesCopy1)
80+
end benchGroupCumSumInPlace
81+
82+
@Benchmark
83+
def benchGroupDiff(bh: Blackhole): Unit =
84+
val result = groupDiff(groups, values)
85+
bh.consume(result)
86+
end benchGroupDiff
87+
88+
@Benchmark
89+
def benchGroupDiffInPlace(bh: Blackhole): Unit =
90+
groupDiffInPlace(groups, valuesCopy2)
91+
bh.consume(valuesCopy2)
92+
end benchGroupDiffInPlace
93+
94+
@Benchmark
95+
def benchGroupSum(bh: Blackhole): Unit =
96+
val (uniqueGroups, sums) = groupSum(groups, values)
97+
bh.consume(uniqueGroups)
98+
bh.consume(sums)
99+
end benchGroupSum
100+
101+
end GroupOpsBenchmark

‎benchmark/src/increments.scala‎

Lines changed: 0 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,3 @@
1-
/*
2-
* Copyright 2020, 2021, Ludovic Henry
3-
*
4-
* Permission is hereby granted, free of charge, to any person obtaining a copy
5-
* of this software and associated documentation files (the "Software"), to deal
6-
* in the Software without restriction, including without limitation the rights
7-
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
8-
* copies of the Software, and to permit persons to whom the Software is
9-
* furnished to do so, subject to the following conditions:
10-
*
11-
* The above copyright notice and this permission notice shall be included in
12-
* all copies or substantial portions of the Software.
13-
*
14-
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
15-
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
16-
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
17-
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
18-
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
19-
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
20-
* SOFTWARE.
21-
*
22-
* Please contact git@ludovic.dev or visit ludovic.dev if you need additional
23-
* information or have any questions.
24-
*/
25-
261
package vecxt.benchmark
272

283
import org.openjdk.jmh.annotations.*

‎benchmark/src/incrementsInt.scala‎

Lines changed: 0 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,3 @@
1-
/*
2-
* Copyright 2020, 2021, Ludovic Henry
3-
*
4-
* Permission is hereby granted, free of charge, to any person obtaining a copy
5-
* of this software and associated documentation files (the "Software"), to deal
6-
* in the Software without restriction, including without limitation the rights
7-
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
8-
* copies of the Software, and to permit persons to whom the Software is
9-
* furnished to do so, subject to the following conditions:
10-
*
11-
* The above copyright notice and this permission notice shall be included in
12-
* all copies or substantial portions of the Software.
13-
*
14-
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
15-
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
16-
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
17-
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
18-
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
19-
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
20-
* SOFTWARE.
21-
*
22-
* Please contact git@ludovic.dev or visit ludovic.dev if you need additional
23-
* information or have any questions.
24-
*/
25-
261
package vecxt.benchmark
272

283
import org.openjdk.jmh.annotations.*

‎benchmark/src/logical.scala‎

Lines changed: 0 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,3 @@
1-
/*
2-
* Copyright 2020, 2021, Ludovic Henry
3-
*
4-
* Permission is hereby granted, free of charge, to any person obtaining a copy
5-
* of this software and associated documentation files (the "Software"), to deal
6-
* in the Software without restriction, including without limitation the rights
7-
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
8-
* copies of the Software, and to permit persons to whom the Software is
9-
* furnished to do so, subject to the following conditions:
10-
*
11-
* The above copyright notice and this permission notice shall be included in
12-
* all copies or substantial portions of the Software.
13-
*
14-
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
15-
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
16-
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
17-
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
18-
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
19-
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
20-
* SOFTWARE.
21-
*
22-
* Please contact git@ludovic.dev or visit ludovic.dev if you need additional
23-
* information or have any questions.
24-
*/
25-
261
package vecxt.benchmark
272

283
import org.openjdk.jmh.annotations.*

0 commit comments

Comments
 (0)