Skip to content

Commit 6b03190

Browse files
MRiabovqwencoder
andcommitted
(refactor) Rename plan templates and add renderer experiment
- Rename benchmark_generator/plan.md to benchmark_plan.md - Rename engineer/plan.md to engineering_plan.md - Update todo.md references to match new plan filenames - Update benchmark_plan_evidence_script.py templates for drafting mode - Add renderer-parallel-modalities performance experiment Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
1 parent bc232c3 commit 6b03190

10 files changed

Lines changed: 646 additions & 8 deletions

File tree

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# Renderer Parallel Modalities Experiment
2+
3+
This experiment measures preview modality rendering on the current headless
4+
renderer backend. Parallel RGB/depth/segmentation fan-out is only supported in
5+
smoke-test mode; non-smoke runs stay sequential and fail closed if parallel mode
6+
is requested.
7+
8+
It exercises the same `render_preview_scene_bundle(...)` code path that the
9+
renderer worker uses internally, but it runs on a synthetic scene so the result
10+
is not dominated by worker HTTP admission, retry loops, or seed bundle staging.
11+
12+
The benchmark compares sequential rendering, and in smoke-test mode can also
13+
exercise the parallel branch.
14+
15+
The default scene uses 24 views, all three modalities, axes and edges enabled,
16+
and a payload-path overlay, which matches the expensive part of the current
17+
preview workload.
18+
19+
## Run
20+
21+
From repo root:
22+
23+
```bash
24+
uv run python scripts/experiments/performance/renderer-parallel-modalities/benchmark_parallel_modalities.py
25+
```
26+
27+
Quick smoke test with a single view:
28+
29+
```bash
30+
uv run python scripts/experiments/performance/renderer-parallel-modalities/benchmark_parallel_modalities.py \
31+
--smoke-test-mode true \
32+
--repetitions 1 \
33+
--warmup-runs 0
34+
```
35+
36+
Write to a custom output file:
37+
38+
```bash
39+
uv run python scripts/experiments/performance/renderer-parallel-modalities/benchmark_parallel_modalities.py \
40+
--output scripts/experiments/performance/renderer-parallel-modalities/results-custom.json
41+
```
42+
43+
## Output
44+
45+
The script writes a timestamped JSON result in this folder by default.
46+
47+
Important fields:
48+
49+
- `summary.parallel.mean_sec`
50+
- `summary.sequential.mean_sec`
51+
- `summary.parallel_minus_sequential_sec`
52+
- `summary.parallel_to_sequential_ratio`
53+
- `rows`
54+
55+
How to read the result:
56+
57+
- `parallel_minus_sequential_sec > 0` means parallel modality rendering was
58+
slower on average.
59+
- `parallel_to_sequential_ratio < 1` means parallel modality rendering was
60+
faster on average.
61+
- `rows[*].warmup=true` are intentionally excluded from the summary.

0 commit comments

Comments
 (0)