Weekly benchmarks by AnshulPatil2005 · Pull Request #1758 · elalish/manifold

AnshulPatil2005 · 2026-06-16T06:59:10Z

Adds weekly benchmark workflow and reporting structure.

AnshulPatil2005 · 2026-06-16T07:01:53Z

I am currently working on running the Ember benchmarks in the workflow first. Once that is done, I will move on to adding the other tests.

The structure is similar to the previous benchmark guard CI, with a few changes for the Ember cases.

codecov · 2026-06-16T07:26:23Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.62%. Comparing base (895bfc2) to head (75ece90).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1758   +/-   ##
=======================================
  Coverage   94.62%   94.62%           
=======================================
  Files          44       44           
  Lines        9441     9441           
=======================================
  Hits         8934     8934           
  Misses        507      507

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

pca006132 · 2026-06-16T07:43:11Z

did you try it on your fork? say, once a day, to see if it works?

AnshulPatil2005 · 2026-06-16T09:23:09Z

did you try it on your fork? say, once a day, to see if it works?

I tested it manually on my fork and the workflow passed. I also added a temporary schedule on the fork default branch to test the scheduled trigger I set it at a very short time(10 mins), but it did not emit any runs so i increased the time so see if it would work that way.

i checked regarding this and i found on github's docs that scheduled workflows are best-effort and can be delayed or dropped under load, I will leave it enabled for longer and confirm once the scheduled run appears.

but this seems to be very unreliable, what if it dropped the weekly run ?

AnshulPatil2005 · 2026-06-17T07:20:49Z

Update: the scheduled workflow is working on my fork now. I changed the temporary schedule to a 12-hour interval, and it has produced 3 scheduled Ember benchmark runs so far.

AnshulPatil2005 · 2026-06-23T10:18:06Z

I have run the workflow several times on my fork now(using schedule).

The weekly benchmark currently lands on different runners, so the absolute timings vary between runs. I will compile the results from these runs to estimate how much difference this is causing.

AnshulPatil2005 · 2026-07-03T15:32:16Z

xlx_processed.xlsx

MacOS(named "arm" here) seems to be more noisier due to the OS scheduler moving the benchmark thread to an E-core(efficiency core rather than performance core) for even one repeat, that repeat is 3–4× slower.

@pca006132 @elalish

elalish · 2026-07-03T18:09:33Z

Nice plots, though the labels could use a little work. If "arm" is MacOS, maybe just call it that? And what are EPYC####? And what are C###? Are those particular test models?

AnshulPatil2005 · 2026-07-03T23:34:28Z

Nice plots, though the labels could use a little work. If "arm" is MacOS, maybe just call it that? And what are EPYC####? And what are C###? Are those particular test models?

Yes, I should have mentioned the label meanings. This was discussed in the previous meeting, so I missed adding the context here.

EPYC#### refers to the GitHub-hosted Linux runner CPU models that were assigned during CI runs. arm is the macOS 15 ARM runner; I should rename that label to macOS 15 to make it clearer.

I am also avoiding macos-latest for the weekly benchmark because the macOS version can change over time and add extra noise.

And yes, C667 means Ember benchmark case 667.

Let me know any improvements or confusion is there, I will change it accordingly

pca006132 · 2026-07-05T19:42:04Z

@AnshulPatil2005 what about the macos timing anomaly for the largest sphere in perfTest? Any idea about that?

AnshulPatil2005 · 2026-07-05T20:45:50Z

@AnshulPatil2005 what about the macos timing anomaly for the largest sphere in perfTest? Any idea about that?

Root cause is the macOS-15 runners run on Apple Silicon (M1/M2/M3). The OS moves threads from P-cores (~3.5GHz) to E-cores (~2.0GHz) at any point during execution. A 33-second computation gives the scheduler many opportunities to do this so a mid-run migration to E-cores can cost 10–15 seconds.

I think we could try pinning the process to P-cores only if thats possible in macOS

pca006132 · 2026-07-05T20:59:48Z

Interesting. How did you check that?

AnshulPatil2005 · 2026-07-05T21:57:13Z

Interesting. How did you check that?

ok so, i found this
P-cores run at ~4,512 MHz; E-cores run at ~1,050 MHz for background tasks which is a 4.3× frequency ratio

and the Observed worst-case ratio for nTri=512: 4.60ms ÷ 1.10ms = 4.16× which is very similar to that ratio

For nTri=8388608: Observed ratio: 44.16s ÷ 20.17s = 2.19× this doesnt match the 4.3x because The likely reason: a 30+ second computation is long enough that the OS migrates the thread back to a P-core mid-run thus we are noticing less effect for larger size.

But the effect is clear for the smallest sphere

pca006132 · 2026-07-08T10:53:15Z

oh btw, maybe you can convert those STLs to binary STLs, so it will not mess up the line count.

- merge stderr into stdout in run_ember_case so man_bench diagnostic output is not silently dropped - remove Assembly/Triangulation/Simplification/Sorting from INDEPENDENT_PHASES; their Timer::Print calls were removed in PR elalish#1745 and they always produced 0.0; only Intersect12 and Winding03 phases are still reported - update build_ember_summary table to match the 4 remaining phases Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

AnshulPatil2005 · 2026-07-13T11:28:10Z

@pca006132 @elalish Would it make sense to also run the benchmark workflow on release publish events?(version-wise data)

The idea is to reuse the same benchmark runner, but store release results separately in the same benchmark-data branch, for example under releases/<tag>/<run_id>/, while keeping weekly runs under weekly/.

then plot the data on the dashboard
rust has done similar thing in their perf dashboard

pca006132 · 2026-07-14T11:41:43Z

I think we can mark the release in the UI, but the release triggered run should still add to the same dataset. E.g., weekly data but tag when we have releases.

AnshulPatil2005 · 2026-07-14T18:29:03Z

I think we can mark the release in the UI, but the release triggered run should still add to the same dataset. E.g., weekly data but tag when we have releases.

So when a release is published, we trigger the same benchmark workflow and store the result in the same weekly/ dataset, just like a scheduled run, but add release metadata such as release_tag: "v3.5.1".

Then the dashboard can mark that run as a release point in the same timeline.

Am I understanding this correctly?

pca006132 · 2026-07-15T09:11:33Z

Yes, this is what I am thinking.

AnshulPatil2005 · 2026-07-15T09:13:07Z

Yes, this is what I am thinking.

Makes sense, I'll do that

Not for merge - revert before this PR lands. Only relaxing the if: gate so a manual run can validate the folded run: > block and sanitizer arg passing reviewers asked about, without waiting for a scheduled/release run.

…for testing" This reverts commit e00c5f5.

- Move build_sanitizer/test_sanitizer in weekly_benchmark.yml from macos-15 to ubuntu-24.04, matching the existing PR-triggered sanitizer lane in manifold.yml, since nothing about ASan/UBSan requires macOS here. Drops the compiler overrides and macOS-only setup steps. - Download sanitizer build artifacts into build/ instead of the workspace root, so manifold_test's baked-in RPATH to libsamples can actually resolve. It was crashing on load with "Library not loaded: @rpath/libsamples.3.dylib" because the artifact download flattened the build/ directory away. - Capture the sanitizer build's own cmake configure log and derive SANITIZER_SUMMARY_COMPILER from it at runtime instead of hardcoding the string, reusing system_metadata.py's existing helpers. - Trim set -euo pipefail to set -u in the data-publish steps, since GitHub Actions' default bash invocation already applies -e/pipefail. - Remove the duplicate weeklybenchmark-test job from manifold.yml in favor of testing weekly_benchmark.yml directly via workflow_dispatch.

…+ simplify perf summary formatting + clean up .gitignore

pca006132 · 2026-07-22T07:47:03Z

+      if: github.event_name == 'release' || github.ref == format('refs/heads/{0}', github.event.repository.default_branch)
+      shell: bash
+      run: |
+        set -u


set -u makes the script error out immediately if it references a variable that was never set. It was added more for future typo avoidance rather than catching a current error.

I am personally not particularly worried about this kind of typo, since it will show up in the diff. But well, it is fine to keep it as is. I just don't like the idea of putting set -u everywhere.

pca006132 · 2026-07-22T07:47:27Z

@@ -1,6 +1,10 @@
 #!/usr/bin/env bash
 set -euo pipefail


btw normally you can just set it in the shebang.

wouldnt shebang line just becomes a plain comment to bash and never gets executed as flags ?

if you invoke it as ./script.sh then it will invoke the shebang.

AnshulPatil2005 mentioned this pull request Jun 18, 2026

Issues in Benchmark guard CI #1761

Open

pca006132 reviewed Jun 24, 2026

View reviewed changes

pca006132 mentioned this pull request Jul 8, 2026

Fix benchmark guard and add memory guard #1762

Merged

AnshulPatil2005 added 12 commits July 11, 2026 18:30

support weekly benchmarks

ea12169

updated for perftest

6b995a6

rem clipper

f2c414e

other tests

496a06a

parse cmake output + use ember-benchmark-cases.json

863ff36

add only required STl files for ember tests

a87ffbd

remove ember tests+fix corrupted stl file

27735fe

shell script to py + minor changes

f7777ef

mac-os

a36c870

more info of machine with linux fallback

d31419d

storage for weekly benchmarks

23a8374

rebase

65bd5ed

AnshulPatil2005 force-pushed the weekly-benchmark branch from 8a57683 to 65bd5ed Compare July 11, 2026 14:15

AnshulPatil2005 and others added 5 commits July 11, 2026 19:54

binary stl

6f13f8f

clean

2fd22ac

.

7ccf728

revert runner

c7da249

AnshulPatil2005 added 2 commits July 17, 2026 20:07

add frontend + trigger on publish + san extended test

6c831a4

fix formatting

4276bda

pca006132 reviewed Jul 21, 2026

View reviewed changes

AnshulPatil2005 added 4 commits July 21, 2026 23:16

Revert "TEMP: allow workflow_dispatch to exercise data-publish steps …

276d1bc

…for testing" This reverts commit e00c5f5.

Merge dashboard pages + remove dead JS fallbacks + gtest JSON output …

48e3e30

…+ simplify perf summary formatting + clean up .gitignore

pca006132 reviewed Jul 22, 2026

View reviewed changes

Comment thread .github/workflows/weekly_benchmark.yml

pca006132 reviewed Jul 22, 2026

View reviewed changes

Stop forcing clang-18 for sanitizer builds, use runner default (gcc)

75ece90

Conversation

AnshulPatil2005 commented Jun 16, 2026

Uh oh!

AnshulPatil2005 commented Jun 16, 2026

Uh oh!

codecov Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pca006132 commented Jun 16, 2026

Uh oh!

AnshulPatil2005 commented Jun 16, 2026

Uh oh!

AnshulPatil2005 commented Jun 17, 2026

Uh oh!

AnshulPatil2005 commented Jun 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AnshulPatil2005 commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elalish commented Jul 3, 2026

Uh oh!

AnshulPatil2005 commented Jul 3, 2026

Uh oh!

pca006132 commented Jul 5, 2026

Uh oh!

AnshulPatil2005 commented Jul 5, 2026

Uh oh!

pca006132 commented Jul 5, 2026

Uh oh!

AnshulPatil2005 commented Jul 5, 2026

Uh oh!

pca006132 commented Jul 8, 2026

Uh oh!

AnshulPatil2005 commented Jul 13, 2026

Uh oh!

pca006132 commented Jul 14, 2026

Uh oh!

AnshulPatil2005 commented Jul 14, 2026

Uh oh!

pca006132 commented Jul 15, 2026

Uh oh!

AnshulPatil2005 commented Jul 15, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pca006132 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

AnshulPatil2005 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

pca006132 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

pca006132 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

AnshulPatil2005 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Jun 16, 2026 •

edited

Loading

AnshulPatil2005 commented Jul 3, 2026 •

edited

Loading