Skip to content

[Helion Benchmark] Use geomean for speedup #7126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
oulgen merged 1 commit into from
Sep 8, 2025
Merged

[Helion Benchmark] Use geomean for speedup #7126

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions torchci/clickhouse_queries/oss_ci_benchmark_llms/query.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,10 @@ WITH benchmarks AS (
o.model.'backend' AS backend,
o.model.'origins' AS origins,
o.metric.'name' AS metric,
-- Arithmetic mean
floor(arrayAvg(o.metric.'benchmark_values'), 2) AS actual,
-- Geometric mean
floor(exp(arrayAvg(arrayMap(x -> log(x), o.metric.'benchmark_values'))), 2) AS actual_geomean,
floor(toFloat64(o.metric.'target_value'), 2) AS target,
o.benchmark.'mode' AS mode,
o.benchmark.'dtype' AS dtype,
Expand Down Expand Up @@ -141,6 +144,7 @@ SELECT DISTINCT
origins,
metric,
actual,
actual_geomean,
target,
mode,
dtype,
Expand Down
3 changes: 3 additions & 0 deletions torchci/components/benchmark/llms/components/LLMsSummaryPanel.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -384,6 +384,9 @@ export default function LLMsSummaryPanel({
l = l === 1 ? "Pass" : "Fail";
r = r === 1 ? "Pass" : "Fail";
} else if (metric.includes("speedup")) {
l = v.l.actual_geomean;
r = v.r.actual_geomean;

const accuracy = metric.replace(/_speedup$/, "_accuracy");
const accuracy_v = params.row[accuracy];
if (accuracy_v.l.actual !== 1) {
Expand Down
7 changes: 4 additions & 3 deletions torchci/lib/benchmark/llms/common.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,11 @@ export const METRIC_DISPLAY_HEADERS: { [k: string]: string } = {
p99_ttft_ms: "p99 TTFT (ms)",
requests_per_second: "Requests/s",
tokens_per_second: "Tokens/s",
triton_speedup: "Triton Speedup",
triton_speedup: "Triton Speedup (Geomean)",
triton_accuracy: "Triton Accuracy",
torch_compile_speedup: "Torch Compile Speedup",
torch_compile_speedup: "Torch Compile Speedup (Geomean)",
torch_compile_accuracy: "Torch Compile Accuracy",
helion_speedup: "Helion Speedup",
helion_speedup: "Helion Speedup (Geomean)",
helion_accuracy: "Helion Accuracy",
};
// The variable name is a bit dumb, but it tells if a higher metric value
Expand Down Expand Up @@ -127,6 +127,7 @@ export interface LLMsBenchmarkData {
job_id: number;
metric: string;
actual: number;
actual_geomean: number;
target: number;
mode?: string;
dtype: string;
Expand Down
9 changes: 9 additions & 0 deletions torchci/lib/benchmark/llms/utils/llmUtils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,7 @@ export function computeGeomean(data: LLMsBenchmarkData[], metricName: string) {
job_id: Number(jobId),
metric: `${metric} (geomean)`,
actual: Number(gm),
actual_geomean: Number(gm),
target: 0,
dtype: dtype,
device: device,
Expand Down Expand Up @@ -339,19 +340,23 @@ const toRowData = (
l: hasL
? {
actual: Number.MAX_SAFE_INTEGER, // indicate the failure on left side
actual_geomean: Number.MAX_SAFE_INTEGER, // indicate the failure on left side
target: 0,
}
: {
actual: 0,
actual_geomean: 0,
target: 0,
},
r: hasR
? {
actual: Number.MAX_SAFE_INTEGER, // indicate the failure on right side
actual_geomean: Number.MAX_SAFE_INTEGER, // indicate the failure on right side
target: 0,
}
: {
actual: 0,
actual_geomean: 0,
target: 0,
},
highlight: hasL && hasR,
Expand All @@ -361,19 +366,23 @@ const toRowData = (
l: hasL
? {
actual: record["l"].actual,
actual_geomean: record["l"].actual_geomean,
target: record["l"].target,
}
: {
actual: 0,
actual_geomean: 0,
target: 0,
},
r: hasR
? {
actual: record["r"].actual,
actual_geomean: record["r"].actual_geomean,
target: record["r"].target,
}
: {
actual: 0,
actual_geomean: 0,
target: 0,
},
highlight: hasL && hasR,
Expand Down