Skip to content

Add Benchmarks for RRT#67

Merged
eholum merged 9 commits into
mainfrom
benchmarking-integration-tests
Sep 30, 2025
Merged

Add Benchmarks for RRT#67
eholum merged 9 commits into
mainfrom
benchmarking-integration-tests

Conversation

@eholum

@eholum eholum commented Sep 16, 2025

Copy link
Copy Markdown
Collaborator

Not really what I was supposed to be doing but at least it would catch if things are broken. Plus I think they would be useful...

-------------------------------------------------------------------------------------------- benchmark: 6 tests --------------------------------------------------------------------------------------------
Name (time in ms)                           Min                 Max                Mean             StdDev              Median                IQR            Outliers      OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_rrt_connect[kinova]      22.7227 (1.0)       64.8647 (1.10)      38.9695 (1.0)      10.8353 (3.33)      37.7636 (1.0)      14.6174 (3.28)          6;0  25.6611 (1.0)          23           1
test_benchmark_rrt[kinova]              46.5273 (2.05)      58.7023 (1.0)       52.2198 (1.34)      3.2497 (1.0)       52.3314 (1.39)      4.4541 (1.0)           3;0  19.1498 (0.75)         12           1
test_benchmark_rrt_connect[ur5]         93.4375 (4.11)     187.5150 (3.19)     129.8622 (3.33)     33.1143 (10.19)    117.8829 (3.12)     50.7555 (11.40)         3;0   7.7005 (0.30)         10           1
test_benchmark_rrt[ur5]                119.8362 (5.27)     134.7720 (2.30)     127.2674 (3.27)      5.1344 (1.58)     127.2968 (3.37)      7.5349 (1.69)          3;0   7.8575 (0.31)          8           1
test_benchmark_rrt_connect[franka]     192.6760 (8.48)     359.8098 (6.13)     254.2296 (6.52)     66.5204 (20.47)    239.0451 (6.33)     94.2356 (21.16)         1;0   3.9335 (0.15)          6           1
test_benchmark_rrt[franka]             227.0057 (9.99)     248.0522 (4.23)     240.8397 (6.18)      8.1034 (2.49)     243.9057 (6.46)      7.2561 (1.63)          1;0   4.1521 (0.16)          5           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

@eholum eholum self-assigned this Sep 16, 2025
Comment thread benchmarks/test_benchmark_rrt.py Outdated


def solve_many(scene: Scene, rrt: RRT, iterations: int = 10):
for i in range(7):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be range(iterations) yeah?

@sea-bass sea-bass left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like I want this to land before the continuous joint expanding PR to know how much worse it's going to make the results (if even noticeable)

@eholum
eholum force-pushed the benchmarking-integration-tests branch from 4f698e4 to 16ae8c3 Compare September 29, 2025 15:00
Comment on lines +123 to +126
# TODO: Decide on a method for automated benchmark performance testing.
- name: Run benchmarks
run: |
pixi run test_benchmarks

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've played around with, but haven't actually connected, two options here for continuous benchmark monitoring (assuming we want this)...

  1. https://github.com/benchmark-action/github-action-benchmark, which can store benchmarking results in gh-pages. It'll be a little annoying to configure so if we go this route it's probably easier to just wait until the repo is public. On the plus side we wouldn't need a separate app and it supports more benchmarking suites out of the box. On the down side it's more to configure.

  2. https://github.com/CodSpeedHQ/action, third part app to which benchmarking results are uploaded and are viewable on their dashboard. It's pretty easy to setup and use but requires granting access to our repo.

Thoughts? Or do we just want the benchmarks in place and then can come back to this later and rely on checking manually?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sample output here.

https://github.com/open-planning/roboplan/actions/runs/18103021233/job/51510609862?pr=67#step:6:27

If this is enough for now then we can call it enough for now...

@sea-bass sea-bass Sep 29, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think waiting for the repo to be public is good for me. We can just make this a non-required check for now.

As for 1 or 2, I think whichever is easier tbh

@eholum
eholum marked this pull request as ready for review September 30, 2025 00:26
@eholum eholum changed the title [WIP] Add Benchmarks for RRT Add Benchmarks for RRT Sep 30, 2025
@eholum
eholum merged commit efb96fd into main Sep 30, 2025
10 of 16 checks passed
@eholum
eholum deleted the benchmarking-integration-tests branch September 30, 2025 00:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants