Skip to content

Add delta report for benchmarks - #91

Merged
gabotechs merged 2 commits into
mainfrom
gabrielmusat/add-delta-benchmark-report
Aug 13, 2025
Merged

Add delta report for benchmarks#91
gabotechs merged 2 commits into
mainfrom
gabrielmusat/add-delta-benchmark-report

Conversation

@gabotechs

@gabotechs gabotechs commented Aug 12, 2025

Copy link
Copy Markdown
Collaborator

Adds a delta report that compares the last run benchmarks with the current ones, something like this:

==== Comparison with the previous benchmark from 2025-08-12 14:38:33 UTC ====
 Query 1: prev=  74 ms, new=  70 ms, diff=1.06 faster ✔
 Query 2: prev=  25 ms, new=  25 ms, diff=1.00 slower ✖
 Query 3: prev=  37 ms, new=  36 ms, diff=1.03 faster ✔
 Query 4: prev=  25 ms, new=  28 ms, diff=1.12 slower ✖
 Query 5: prev=  46 ms, new=  54 ms, diff=1.17 slower ✖
 Query 6: prev=  22 ms, new=  21 ms, diff=1.05 faster ✔
 Query 7: prev=  64 ms, new=  61 ms, diff=1.05 faster ✔
 Query 8: prev=  47 ms, new=  43 ms, diff=1.09 faster ✔
 Query 9: prev=  63 ms, new=  53 ms, diff=1.19 faster ✔
Query 10: prev=  66 ms, new=  53 ms, diff=1.25 faster ✅
Query 11: prev=  34 ms, new=  21 ms, diff=1.62 faster ✅
Query 12: prev=  58 ms, new=  41 ms, diff=1.41 faster ✅
Query 13: prev=  34 ms, new=  32 ms, diff=1.06 faster ✔
Query 14: prev=  35 ms, new=  30 ms, diff=1.17 faster ✔
Query 15: prev=  53 ms, new=  48 ms, diff=1.10 faster ✔
Query 16: prev=  23 ms, new=  19 ms, diff=1.21 faster ✅
Query 17: prev=  77 ms, new=  70 ms, diff=1.10 faster ✔
Query 18: prev= 103 ms, new=  92 ms, diff=1.12 faster ✔
Query 19: prev=  58 ms, new=  51 ms, diff=1.14 faster ✔
Query 20: prev=  52 ms, new=  41 ms, diff=1.27 faster ✅
Query 21: prev= 112 ms, new=  89 ms, diff=1.26 faster ✅
Query 22: prev=  22 ms, new=  17 ms, diff=1.29 faster ✅

The first time the benchmarks are run, no report will be printed to stdout, the second time, a report will be produced comparing it with the first run, the third time will be compared to the second one, etc...

This can be tested running this command twice:

cargo run -p datafusion-distributed-benchmarks --release -- tpch --path benchmarks/data/tpch_sf1 

Comment thread benchmarks/src/tpch/run.rs Outdated
Comment thread benchmarks/src/tpch/run.rs Outdated
Comment thread benchmarks/Cargo.toml

@robtandy robtandy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, but it will have conflicts for port number if #89 is merged first. I suggest we do it in that order to accommodate our staggered working hours.

@gabotechs
gabotechs force-pushed the gabrielmusat/add-delta-benchmark-report branch from c1be7cc to 47e0b92 Compare August 13, 2025 10:48
@gabotechs
gabotechs merged commit bae85ca into main Aug 13, 2025
3 checks passed
@gabotechs
gabotechs deleted the gabrielmusat/add-delta-benchmark-report branch August 13, 2025 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants