Skip to content

Commit 3fd48e2

Browse files
jonathanpwangshuklaayushGolovanov399arayikhalatyannyunyunyunyu
authored
perf: execution and tracegen rewrite (#1567)
**To be squash merged** ## What's Changed See changelog for details. The STARK backend `MultiStarkVerifyingKey`s for all existing VM configs remains **unchanged** between v1.3.0 and this PR, with the exception of the `RootVerifierProvingKey` noted below. However the `AppVerifyingKey` has a difference in binary serialization due to the removal of an `as_offset` field from `MemoryDimensions`. [This workflow](https://github.com/openvm-org/openvm/actions/runs/17061122733/job/48368023655) shows that after converting to account for this change: - all `AppVerifyingKey`s remain unchanged - all STARK aggregation vkeys except the Root Verifier vkey remain unchanged - the Root Verifier vkey's trace height constraints were updated to fix a missing permutation. This changes the pre-vkey hash of the Root Verifier vkey, which affects the Halo2 static verifier's initial Fiat-Shamir transcript. However this change does not impact the security of existing Root Verifier vkeys generated through the SDK because Root Verifier proofs have fixed trace heights and these trace heights have been checked to satisfy all trace height constraints statically., - the Halo2Verifier.sol Solidity verifier contract has changed between v1.3 and this PR change the `Halo2Config::default()` now uses `verifier_k = 23` instead of `verifier_k = 24` for a smaller default halo2 circuit. The initial Fiat-Shamir transcript state also changed due to the Root Verifier vkey change above. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Ayush Shukla <[email protected]> Co-authored-by: Alexander Golovanov <[email protected]> Co-authored-by: Arayi Khalatyan <[email protected]> Co-authored-by: Ayush Shukla <[email protected]> Co-authored-by: Xinding Wei <[email protected]> Co-authored-by: Lun-Kai Hsu <[email protected]> Co-authored-by: Arayi <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: crStiv <[email protected]> Co-authored-by: HrikB <[email protected]> Co-authored-by: Yi Sun <[email protected]> Co-authored-by: Avaneesh-axiom <[email protected]> Co-authored-by: stephenh-axiom-xyz <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Teo Kitanovski <[email protected]> Co-authored-by: Valery Cherepanov <[email protected]> Co-authored-by: Peyman Jabbarzade <[email protected]> Co-authored-by: Valery <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: matejav <[email protected]> Co-authored-by: PangZhi <[email protected]> Co-authored-by: Ubuntu <[email protected]>
1 parent c033e18 commit 3fd48e2

File tree

577 files changed

+56546
-32799
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

577 files changed

+56546
-32799
lines changed

‎.config/nextest.toml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Nextest configuration for OpenVM project
2+
3+
# Define test groups with different weights
4+
[[profile.default.overrides]]
5+
# Match all tests with "persistent" in their name
6+
filter = 'test(~persistent)'
7+
# Give these tests 5x the default weight because they use more memory
8+
threads-required = 16
9+
10+
# custom profile for heavy tests
11+
[profile.heavy]
12+
# Run fewer tests in parallel for heavy workloads
13+
test-threads = 2

‎.github/workflows/benchmark-call.yml‎

Lines changed: 16 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ on:
4949
features:
5050
type: string
5151
required: false
52-
description: Host features, comma separated (aggregation,profiling)
52+
description: Host features, comma separated (aggregation,perf-metrics)
5353
workflow_call:
5454
inputs:
5555
benchmark_name:
@@ -102,12 +102,12 @@ on:
102102
features:
103103
type: string
104104
required: false
105-
description: Host features, comma separated (aggregation,profiling)
105+
description: Host features, comma separated (aggregation,perf-metrics)
106106

107107
env:
108108
S3_METRICS_PATH: s3://openvm-public-data-sandbox-us-east-1/benchmark/github/metrics
109109
S3_FLAMEGRAPHS_PATH: s3://openvm-public-data-sandbox-us-east-1/benchmark/github/flamegraphs
110-
FEATURE_FLAGS: "bench-metrics,parallel,nightly-features"
110+
FEATURE_FLAGS: "metrics,parallel,nightly-features"
111111
INPUT_ARGS: ""
112112
CARGO_NET_GIT_FETCH_WITH_CLI: "true"
113113

@@ -170,7 +170,7 @@ jobs:
170170
ROOT_ARG="--root_log_blowup ${{ inputs.root_log_blowup }}"
171171
INTERNAL_ARG="--internal_log_blowup ${{ inputs.internal_log_blowup }}"
172172
bash ./extensions/native/recursion/trusted_setup_s3.sh
173-
PARAMS_DIR=$(pwd)/params
173+
PARAMS_DIR=$HOME/.openvm/params
174174
PARAMS_ARG="--kzg-params-dir $PARAMS_DIR"
175175
echo "INPUT_ARGS=${ROOT_ARG} ${INTERNAL_ARG} ${PARAMS_ARG} ${INPUT_ARGS}" >> $GITHUB_ENV
176176
fi
@@ -230,11 +230,11 @@ jobs:
230230
s5cmd cp $METRIC_PATH ${{ env.S3_METRICS_PATH }}/${METRIC_NAME}-${current_sha}.json
231231
232232
- name: Install inferno-flamegraph
233-
if: ${{ contains(env.FEATURE_FLAGS, 'profiling') }}
233+
if: ${{ contains(env.FEATURE_FLAGS, 'perf-metrics') }}
234234
run: cargo install inferno
235235

236236
- name: Generate flamegraphs
237-
if: ${{ contains(env.FEATURE_FLAGS, 'profiling') }}
237+
if: ${{ contains(env.FEATURE_FLAGS, 'perf-metrics') }}
238238
run: |
239239
if [[ -f $METRIC_PATH ]]; then
240240
GUEST_SYMBOLS_PATH="${METRIC_PATH%.json}.syms"
@@ -250,9 +250,15 @@ jobs:
250250
fi
251251
252252
##########################################################################
253-
# Update s3 for latest main metrics upon a push event #
253+
# Update s3 for latest branch metrics upon a push event #
254254
##########################################################################
255-
- name: Update latest main result in s3
256-
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
255+
- name: Update latest branch result in s3
256+
if: github.event_name == 'push'
257257
run: |
258-
s5cmd cp $METRIC_PATH "${{ env.S3_METRICS_PATH }}/main-${METRIC_NAME}.json"
258+
if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
259+
# for backwards compatibility
260+
REF_HASH="main"
261+
else
262+
REF_HASH=$(echo "${{ github.ref }}" | sha256sum | cut -d' ' -f1)
263+
fi
264+
s5cmd cp $METRIC_PATH "${{ env.S3_METRICS_PATH }}/${REF_HASH}-${METRIC_NAME}.json"

‎.github/workflows/benchmarks-execute.yml‎

Lines changed: 88 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
1-
name: "benchmarks-execute"
1+
name: "Execution benchmarks"
22

33
on:
44
push:
5-
branches: ["main"]
5+
# TODO(ayush): remove after feat/new-execution is merged
6+
branches: ["main", "feat/new-execution"]
67
pull_request:
78
types: [opened, synchronize, reopened, labeled]
89
branches: ["**"]
@@ -18,95 +19,101 @@ on:
1819
- ".github/workflows/benchmarks-execute.yml"
1920
workflow_dispatch:
2021

22+
concurrency:
23+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
24+
cancel-in-progress: true
25+
2126
env:
2227
CARGO_TERM_COLOR: always
28+
S3_FIXTURES_PATH: s3://openvm-public-data-sandbox-us-east-1/benchmark/fixtures
29+
JEMALLOC_SYS_WITH_MALLOC_CONF: "retain:true,background_thread:true,metadata_thp:always,thp:always,dirty_decay_ms:10000,muzzy_decay_ms:10000,abort_conf:true"
2330

2431
jobs:
25-
execute-benchmarks:
32+
codspeed-walltime-benchmarks:
33+
name: Run codspeed walltime benchmarks
2634
runs-on:
2735
- runs-on=${{ github.run_id }}
28-
- runner=8cpu-linux-x64
36+
- family=m5a.xlarge # 2.5Ghz clock speed
37+
- image=ubuntu24-full-x64
38+
- extras=s3-cache
39+
40+
env:
41+
CODSPEED_RUNNER_MODE: walltime
42+
2943
steps:
44+
- uses: runs-on/action@v1
3045
- uses: actions/checkout@v4
31-
32-
- name: Set up Rust
33-
uses: actions-rs/toolchain@v1
46+
- uses: dtolnay/rust-toolchain@stable
47+
- uses: Swatinem/rust-cache@v2
3448
with:
35-
profile: minimal
36-
toolchain: stable
37-
override: true
49+
cache-on-failure: true
3850

39-
- name: Run execution benchmarks
40-
working-directory: benchmarks/execute
41-
run: cargo run | tee benchmark_output.log
51+
- name: Install architecture specific tools
52+
run: |
53+
source ci/scripts/utils.sh
54+
install_s5cmd
4255
43-
- name: Parse benchmark results
56+
- name: Pull fixtures from S3
57+
run: |
58+
mkdir -p benchmarks/fixtures
59+
s5cmd cp "${{ env.S3_FIXTURES_PATH }}/*" benchmarks/fixtures/ || echo "No fixtures found in S3"
60+
61+
- name: Install cargo-binstall
62+
uses: cargo-bins/cargo-binstall@main
63+
- name: Install codspeed
64+
run: cargo binstall --no-confirm --force cargo-codspeed
65+
66+
- name: Build benchmarks
4467
working-directory: benchmarks/execute
68+
run: cargo codspeed build --profile maxperf
69+
- name: Run benchmarks
70+
uses: CodSpeedHQ/action@v3
71+
with:
72+
working-directory: benchmarks/execute
73+
run: cargo codspeed run
74+
token: ${{ secrets.CODSPEED_TOKEN }}
75+
76+
codspeed-instrumentation-benchmarks:
77+
name: Run codspeed instrumentation benchmarks
78+
runs-on:
79+
- runs-on=${{ github.run_id }}
80+
- family=m5a.xlarge
81+
- image=ubuntu24-full-x64
82+
- extras=s3-cache
83+
if: github.event_name != 'pull_request'
84+
85+
env:
86+
CODSPEED_RUNNER_MODE: instrumentation
87+
88+
steps:
89+
- uses: runs-on/action@v1
90+
- uses: actions/checkout@v4
91+
- uses: dtolnay/rust-toolchain@stable
92+
- uses: Swatinem/rust-cache@v2
93+
with:
94+
cache-on-failure: true
95+
96+
- name: Install architecture specific tools
4597
run: |
46-
# Determine if running in GitHub Actions environment
47-
if [ -n "$GITHUB_STEP_SUMMARY" ]; then
48-
SUMMARY_FILE="$GITHUB_STEP_SUMMARY"
49-
echo "### Benchmark Results Summary" >> "$SUMMARY_FILE"
50-
else
51-
SUMMARY_FILE="benchmark_summary.md"
52-
echo "### Benchmark Results Summary" > "$SUMMARY_FILE"
53-
echo "Saving summary to $SUMMARY_FILE"
54-
fi
55-
56-
# Set up summary table header
57-
echo "| Program | Total Time (ms) |" >> "$SUMMARY_FILE"
58-
echo "| ------- | --------------- |" >> "$SUMMARY_FILE"
59-
60-
# Variables to track current program and total time
61-
current_program=""
62-
total_time=0
63-
64-
# Process the output file line by line
65-
while IFS= read -r line; do
66-
# Check if line contains "Running program" message
67-
if [[ $line =~ ｉ\ \[info\]:\ Running\ program:\ ([a-zA-Z0-9_-]+) ]]; then
68-
# If we were processing a program, output its results
69-
if [[ -n "$current_program" ]]; then
70-
echo "| $current_program | $total_time |" >> "$SUMMARY_FILE"
71-
fi
72-
73-
# Start tracking new program
74-
current_program="${BASH_REMATCH[1]}"
75-
total_time=0
76-
fi
77-
78-
# Check for program completion to catch programs that might have no execution segments
79-
if [[ $line =~ ｉ\ \[info\]:\ Completed\ program:\ ([a-zA-Z0-9_-]+) ]]; then
80-
completed_program="${BASH_REMATCH[1]}"
81-
# If no segments were found for this program, ensure it's still in the output
82-
if [[ "$current_program" == "$completed_program" && $total_time == 0 ]]; then
83-
echo "| $current_program | 0 |" >> "$SUMMARY_FILE"
84-
current_program=""
85-
fi
86-
fi
87-
88-
# Check if line contains execution time (looking for the format with ms or s)
89-
if [[ $line =~ execute_segment\ \[\ ([0-9.]+)(ms|s)\ \|\ [0-9.]+%\ \]\ segment ]]; then
90-
segment_time="${BASH_REMATCH[1]}"
91-
unit="${BASH_REMATCH[2]}"
92-
93-
# Convert to milliseconds if in seconds
94-
if [[ "$unit" == "s" ]]; then
95-
segment_time=$(echo "scale=6; $segment_time * 1000" | bc)
96-
fi
97-
98-
# Add segment time to total
99-
total_time=$(echo "scale=6; $total_time + $segment_time" | bc)
100-
fi
101-
done < benchmark_output.log
102-
103-
# Output the last program result if there was one
104-
if [[ -n "$current_program" ]]; then
105-
echo "| $current_program | $total_time |" >> "$SUMMARY_FILE"
106-
fi
107-
108-
# If not in GitHub Actions, print the summary to the terminal
109-
if [ -z "$GITHUB_STEP_SUMMARY" ]; then
110-
echo -e "\nBenchmark Summary:"
111-
cat "$SUMMARY_FILE"
112-
fi
98+
source ci/scripts/utils.sh
99+
install_s5cmd
100+
101+
- name: Pull fixtures from S3
102+
run: |
103+
mkdir -p benchmarks/fixtures
104+
s5cmd cp "${{ env.S3_FIXTURES_PATH }}/*" benchmarks/fixtures/ || echo "No fixtures found in S3"
105+
106+
- name: Install cargo-binstall
107+
uses: cargo-bins/cargo-binstall@main
108+
- name: Install codspeed
109+
run: cargo binstall --no-confirm --force cargo-codspeed
110+
111+
- name: Build benchmarks
112+
working-directory: benchmarks/execute
113+
run: cargo codspeed build
114+
- name: Run benchmarks
115+
uses: CodSpeedHQ/action@v3
116+
with:
117+
working-directory: benchmarks/execute
118+
run: cargo codspeed run
119+
token: ${{ secrets.CODSPEED_TOKEN }}

‎.github/workflows/benchmarks-upload-fixtures.yml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: "Upload benchmark fixtures"
2+
3+
on:
4+
workflow_dispatch:
5+
6+
env:
7+
CARGO_TERM_COLOR: always
8+
S3_FIXTURES_PATH: s3://openvm-public-data-sandbox-us-east-1/benchmark/fixtures
9+
10+
jobs:
11+
generate-fixtures:
12+
name: Generate and upload benchmark fixtures
13+
runs-on:
14+
- runs-on=${{ github.run_id }}
15+
- runner=64cpu-linux-arm64
16+
- family=m7
17+
- extras=s3-cache
18+
19+
steps:
20+
- uses: runs-on/action@v1
21+
- uses: actions/checkout@v4
22+
- uses: dtolnay/rust-toolchain@stable
23+
- uses: Swatinem/rust-cache@v2
24+
with:
25+
cache-on-failure: true
26+
27+
- name: Install architecture specific tools
28+
run: |
29+
source ci/scripts/utils.sh
30+
install_s5cmd
31+
32+
- name: Generate fixtures
33+
run: cargo r -r --bin generate-fixtures --features generate-fixtures
34+
35+
- name: Upload fixtures to S3
36+
run: |
37+
if [ -d "benchmarks/fixtures" ]; then
38+
s5cmd cp benchmarks/fixtures/ ${{ env.S3_FIXTURES_PATH }}/
39+
else
40+
echo "No fixtures directory found"
41+
exit 1
42+
fi

‎.github/workflows/benchmarks.yml‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ name: "OpenVM Benchmarks: Coordinate Runner & Reporting"
22

33
on:
44
push:
5-
branches: ["main"]
5+
branches: ["main", "feat/new-execution"]
66
pull_request:
77
types: [opened, synchronize, reopened, labeled]
88
branches: ["**"]
@@ -89,7 +89,7 @@ jobs:
8989
FEATURE_FLAGS="aggregation,${FEATURE_FLAGS}"
9090
fi
9191
if [[ "${{ github.event.inputs.flamegraphs }}" == "true" ]]; then
92-
FEATURE_FLAGS="profiling,${FEATURE_FLAGS}"
92+
FEATURE_FLAGS="perf-metrics,${FEATURE_FLAGS}"
9393
fi
9494
9595
matrix=$(jq -c --argjson run_e2e $RUN_E2E --arg features "$FEATURE_FLAGS" '
@@ -211,9 +211,21 @@ jobs:
211211
json_file_list=$(echo -n "$json_files" | paste -sd "," -)
212212
echo $json_file_list
213213
214-
prev_json_files=$(echo $matrix | jq -r '
214+
# For PRs, get the latest commit from the target branch
215+
if [[ "${{ github.event_name }}" == "pull_request" ]]; then
216+
if [[ "${{ github.base_ref }}" == "main" ]]; then
217+
REF_HASH="main"
218+
else
219+
REF_HASH=$(echo "refs/heads/${{ github.base_ref }}" | sha256sum | cut -d' ' -f1)
220+
fi
221+
echo "Target branch REF_HASH: $REF_HASH"
222+
else
223+
REF_HASH="main"
224+
fi
225+
226+
prev_json_files=$(echo $matrix | jq -r --arg target "$REF_HASH" '
215227
.[] |
216-
"main-\(.id).json"')
228+
"\($target)-\(.id).json"')
217229
prev_json_file_list=$(echo -n "$prev_json_files" | paste -sd "," -)
218230
echo $prev_json_file_list
219231

‎.github/workflows/build.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ jobs:
1616
runs-on:
1717
- runs-on=${{ github.run_id }}
1818
- runner=64cpu-linux-arm64
19+
- image=ubuntu24-full-arm64
1920
- extras=s3-cache
2021
steps:
2122
- uses: runs-on/action@v1

0 commit comments

Comments
 (0)