Skip to content

Commit de19605

Browse files
committed
refactor: consolidate rust benchmark tooling
Signed-off-by: lucarlig <luca.carlig@ibm.com>
1 parent 8c6326d commit de19605

31 files changed

+619
-85
lines changed

Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7277,7 +7277,7 @@ profile-compare:
72777277
.PHONY: benchmark
72787278
benchmark:
72797279
@echo "Starting benchmark console (first run may compile; wait for TUI)..."
7280-
cargo run --manifest-path tools_rust/benchmark_console/Cargo.toml --
7280+
cargo run --manifest-path tools_rust/contextforge_benchmark/benchmark_console/Cargo.toml --
72817281

72827282
.PHONY: async-validate
72837283
async-validate:

benchmarks/__init__.py

Lines changed: 0 additions & 1 deletion
This file was deleted.

docs/docs/testing/benchmark-suite.md

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -28,25 +28,25 @@ Each scenario runs against the real containerized testing stack:
2828
The runner is:
2929

3030
```bash
31-
cargo run --manifest-path tools_rust/benchmark_runner/Cargo.toml --
31+
cargo run --manifest-path tools_rust/contextforge_benchmark/benchmark_runner/Cargo.toml --
3232
```
3333

3434
Committed scenarios now live in:
3535

3636
```bash
37-
benchmarks/contextforge/scenarios/
37+
tools_rust/contextforge_benchmark/assets/scenarios/
3838
```
3939

4040
The benchmark launcher now lives in:
4141

4242
```bash
43-
tools_rust/benchmark_console/
43+
tools_rust/contextforge_benchmark/benchmark_console/
4444
```
4545

4646
The Goose driver now lives in:
4747

4848
```bash
49-
tools_rust/contextforge_goose/
49+
tools_rust/contextforge_benchmark/contextforge_goose/
5050
```
5151

5252
## Quick Start
@@ -69,36 +69,36 @@ Inside the launcher you can choose:
6969
- `Generate`
7070

7171
The `Generate` action opens a template builder that saves a new scenario file
72-
under `benchmarks/contextforge/scenarios/`. It fills in the important fields in
72+
under `tools_rust/contextforge_benchmark/assets/scenarios/`. It fills in the important fields in
7373
the UI and writes a full TOML template containing all supported sections and
7474
keys, including commented optional settings for advanced tuning.
7575

7676
Build the benchmark image expected by scenarios with `rebuild_policy = "never"`:
7777

7878
```bash
79-
make container-build CONTAINER_FILE=benchmarks/contextforge/Containerfile ENABLE_RUST_BUILD=1 ENABLE_PROFILING_BUILD=1 CONTAINER_RUNTIME=podman
79+
make container-build CONTAINER_FILE=tools_rust/contextforge_benchmark/assets/Containerfile ENABLE_RUST_BUILD=1 ENABLE_PROFILING_BUILD=1 CONTAINER_RUNTIME=podman
8080
```
8181

8282
Validate the suite:
8383

8484
```bash
85-
cargo run --manifest-path tools_rust/benchmark_runner/Cargo.toml -- validate --scenario modular-design-300
85+
cargo run --manifest-path tools_rust/contextforge_benchmark/benchmark_runner/Cargo.toml -- validate --scenario rust-mcp-runtime-300
8686
```
8787

8888
Run the smoke suite:
8989

9090
```bash
91-
cargo run --manifest-path tools_rust/benchmark_runner/Cargo.toml -- run --scenario a2a-invoke-300 --smoke
91+
cargo run --manifest-path tools_rust/contextforge_benchmark/benchmark_runner/Cargo.toml -- run --scenario a2a-invoke-300 --smoke
9292
```
9393

9494
Run the suite:
9595

9696
```bash
97-
cargo run --manifest-path tools_rust/benchmark_runner/Cargo.toml -- run --scenario modular-design-300
97+
cargo run --manifest-path tools_rust/contextforge_benchmark/benchmark_runner/Cargo.toml -- run --scenario rust-mcp-runtime-300
9898
```
9999

100100
The TUI discovers committed scenarios automatically from
101-
`benchmarks/contextforge/scenarios/`.
101+
`tools_rust/contextforge_benchmark/assets/scenarios/`.
102102

103103
## Scenario Contract
104104

@@ -146,15 +146,15 @@ Unsupported keys now fail validation instead of being silently accepted.
146146

147147
## Request Mixes
148148

149-
The Rust Goose driver in `tools_rust/contextforge_goose/` uses real request
149+
The Rust Goose driver in `tools_rust/contextforge_benchmark/contextforge_goose/` uses real request
150150
families:
151151

152152
- health checks
153153
- admin plugin UI
154154
- REST discovery (`/tools`, `/resources`, `/prompts`)
155155
- JSON-RPC discovery (`tools/list`, `resources/list`, `prompts/list`)
156156
- JSON-RPC prompt/resource/tool calls from payload fixtures in
157-
`benchmarks/contextforge/payloads/`
157+
`tools_rust/contextforge_benchmark/assets/payloads/`
158158

159159
This means plugin-heavy profiles now hit real prompt/resource/tool code paths,
160160
not just health or admin endpoints.
@@ -193,11 +193,11 @@ Key reporting behaviors:
193193
Re-render a saved run:
194194

195195
```bash
196-
cargo run --manifest-path tools_rust/benchmark_runner/Cargo.toml -- regenerate-report --run-dir reports/benchmarks/<run-dir>
196+
cargo run --manifest-path tools_rust/contextforge_benchmark/benchmark_runner/Cargo.toml -- regenerate-report --run-dir reports/benchmarks/<run-dir>
197197
```
198198

199199
Rebuild comparisons for a saved run:
200200

201201
```bash
202-
cargo run --manifest-path tools_rust/benchmark_runner/Cargo.toml -- compare-run --run-dir reports/benchmarks/<run-dir>
202+
cargo run --manifest-path tools_rust/contextforge_benchmark/benchmark_runner/Cargo.toml -- compare-run --run-dir reports/benchmarks/<run-dir>
203203
```

benchmarks/contextforge/Containerfile renamed to tools_rust/contextforge_benchmark/assets/Containerfile

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -87,12 +87,12 @@ COPY run-gunicorn.sh run-granian.sh run.sh /app/
8787
COPY gunicorn.config.py /app/
8888
COPY mcpgateway/ /app/mcpgateway/
8989
COPY plugins/ /app/plugins/
90-
COPY benchmarks/contextforge/ /app/benchmarks/contextforge/
90+
COPY tools_rust/contextforge_benchmark/assets/ /app/tools_rust/contextforge_benchmark/assets/
9191
COPY mcp-catalog.yml /app/
9292

93-
RUN cp /app/benchmarks/contextforge/docker-entrypoint.sh /app/docker-entrypoint.sh \
94-
&& chmod +x /app/run-gunicorn.sh /app/run-granian.sh /app/docker-entrypoint.sh /app/run.sh /app/benchmarks/contextforge/run-uvicorn.sh \
95-
&& python3 -OO -m compileall -q /app/.venv /app/mcpgateway /app/plugins /app/benchmarks/contextforge 2>/dev/null || true \
93+
RUN cp /app/tools_rust/contextforge_benchmark/assets/docker-entrypoint.sh /app/docker-entrypoint.sh \
94+
&& chmod +x /app/run-gunicorn.sh /app/run-granian.sh /app/docker-entrypoint.sh /app/run.sh /app/tools_rust/contextforge_benchmark/assets/run-uvicorn.sh \
95+
&& python3 -OO -m compileall -q /app/.venv /app/mcpgateway /app/plugins /app/tools_rust/contextforge_benchmark/assets 2>/dev/null || true \
9696
&& find /app -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true \
9797
&& chown -R 1001:0 /app \
9898
&& chmod -R g=u /app

benchmarks/contextforge/docker-entrypoint.sh renamed to tools_rust/contextforge_benchmark/assets/docker-entrypoint.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ case "${HTTP_SERVER}" in
1515
;;
1616
uvicorn)
1717
echo "Starting ContextForge benchmark image with Uvicorn..."
18-
exec ./benchmarks/contextforge/run-uvicorn.sh "$@"
18+
exec ./tools_rust/contextforge_benchmark/assets/run-uvicorn.sh "$@"
1919
;;
2020
*)
2121
echo "ERROR: Unknown HTTP_SERVER value: ${HTTP_SERVER}"

benchmarks/contextforge/payloads/prompts/get_compare_timezones.json renamed to tools_rust/contextforge_benchmark/assets/payloads/prompts/get_compare_timezones.json

File renamed without changes.

benchmarks/contextforge/payloads/prompts/get_customer_greeting.json renamed to tools_rust/contextforge_benchmark/assets/payloads/prompts/get_customer_greeting.json

File renamed without changes.

benchmarks/contextforge/payloads/prompts/list_prompts.json renamed to tools_rust/contextforge_benchmark/assets/payloads/prompts/list_prompts.json

File renamed without changes.

benchmarks/contextforge/payloads/resources/list_resources.json renamed to tools_rust/contextforge_benchmark/assets/payloads/resources/list_resources.json

File renamed without changes.

benchmarks/contextforge/payloads/resources/read_timezone_info.json renamed to tools_rust/contextforge_benchmark/assets/payloads/resources/read_timezone_info.json

File renamed without changes.

0 commit comments

Comments
 (0)