Skip to content

Commit f16a12f

Browse files
lukekimlukekim
authored
Merge branch 'trunk' into lukim/databricks
2 parents b59a034 + fb16774 commit f16a12f

41 files changed

Lines changed: 2494 additions & 128 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/skills/system-adapter-builder/SKILL.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
---
2+
name: system-adapter-builder
3+
description: Build or update a SpiceBench system adapter with JSON-RPC over stdio and HTTP, including setup/query_method/teardown/metrics support and template validation.
4+
---
5+
6+
# SpiceBench System Adapter Builder
7+
8+
Use this skill when implementing a new system adapter or customizing one of the templates in `system-adapters/templates`.
9+
10+
## What this skill builds
11+
12+
A JSON-RPC 2.0 adapter that supports both transports:
13+
14+
- stdio transport (line-delimited JSON-RPC messages)
15+
- HTTP transport (`POST /jsonrpc` by default)
16+
17+
Required methods:
18+
19+
- `setup(run_id, datasets)`
20+
- `query_method(run_id)`
21+
- `teardown(run_id)`
22+
- `metrics(run_id)`
23+
- `rpc.methods`
24+
25+
## Inputs to collect first
26+
27+
- Target language (`python`, `nodejs`, `rust`, `go`, or `java`)
28+
- SUT provisioning flow for `setup` and `teardown`
29+
- How to resolve query endpoint and credentials for `query_method`
30+
- Where to source metrics (cloud APIs, DB telemetry, host exporters)
31+
- Runtime/toolchain target (latest/LTS channel used by repository workflows)
32+
33+
## Build steps
34+
35+
1. Copy the nearest template from `system-adapters/templates/<language>`.
36+
2. Keep request/response envelopes JSON-RPC 2.0 compliant (`jsonrpc`, `id`, `method`, `params`).
37+
3. Implement `setup` and `teardown` with run-scoped resources keyed by `run_id`.
38+
4. Implement `query_method` to return:
39+
- `driver`: typically `flightsql` or `databricks`
40+
- `db_kwargs`: real endpoint + auth kwargs for the SUT
41+
5. Implement `metrics` to return both objects:
42+
- `resource`: CPU, memory, disk bytes, disk IOPS
43+
- `ingestion`: rows, bytes, rows/s, active connections
44+
6. Keep stdio and HTTP using the same dispatcher so behavior is identical.
45+
7. Return JSON-RPC errors with standard codes:
46+
- `-32700` parse error
47+
- `-32600` invalid request
48+
- `-32601` method not found
49+
- `-32602` invalid params
50+
- `-32603` internal error
51+
52+
## Metrics mapping checklist
53+
54+
Map live SUT metrics into these fields:
55+
56+
- `resource.cpu_usage_percent`
57+
- `resource.memory_usage_bytes`
58+
- `resource.disk_read_bytes`
59+
- `resource.disk_write_bytes`
60+
- `resource.disk_read_iops`
61+
- `resource.disk_write_iops`
62+
- `ingestion.rows_ingested`
63+
- `ingestion.bytes_ingested`
64+
- `ingestion.rows_per_sec`
65+
- `ingestion.active_connections`
66+
67+
If any metric is unavailable, return `0`/`0.0` and document why.
68+
69+
## Validation checklist
70+
71+
- Adapter responds to all required methods over stdio and HTTP.
72+
- `rpc.methods` includes every exposed method.
73+
- `query_method` returns a valid `driver` and complete `db_kwargs`.
74+
- `metrics` returns both `resource` and `ingestion` objects.
75+
- Language build/syntax checks pass:
76+
- Python: `python -m py_compile`
77+
- Node.js: `node --check`
78+
- Rust: `cargo build`
79+
- Go: `go build ./...`
80+
- Java: `mvn compile`
81+
82+
## Done criteria
83+
84+
The adapter is considered complete when:
85+
86+
- both transports work,
87+
- required methods are implemented,
88+
- metric fields are populated from real telemetry or documented stubs,
89+
- and template validation workflow passes in CI.

‎.github/workflows/pr.yml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,24 @@ jobs:
5555
- name: cargo test
5656
run: make test
5757

58+
fmt:
59+
name: cargo fmt check
60+
runs-on: spiceai-macos
61+
timeout-minutes: 60
62+
needs: changes
63+
if: needs.changes.outputs.rust == 'true'
64+
steps:
65+
- uses: actions/checkout@v6
66+
67+
- name: Setup Rust toolchain
68+
uses: actions-rust-lang/setup-rust-toolchain@v1
69+
with:
70+
toolchain: 1.91
71+
cache: false # Using GHA cache is slower than re-installing
72+
73+
- name: cargo fmt check
74+
run: make fmt-check
75+
5876
clippy:
5977
name: cargo clippy
6078
runs-on: spiceai-macos

‎.github/workflows/validate_system_adapter_templates.yml‎

Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
---
2+
name: validate-system-adapter-templates
3+
4+
on:
5+
merge_group:
6+
types: [checks_requested]
7+
pull_request:
8+
branches:
9+
- trunk
10+
workflow_dispatch:
11+
12+
jobs:
13+
changes:
14+
name: Detect template changes
15+
runs-on: ubuntu-latest
16+
outputs:
17+
templates: ${{ steps.filter.outputs.templates }}
18+
steps:
19+
- uses: actions/checkout@v6
20+
- uses: dorny/paths-filter@v3
21+
id: filter
22+
with:
23+
filters: |
24+
templates:
25+
- 'system-adapters/templates/**'
26+
- '.github/workflows/validate_system_adapter_templates.yml'
27+
28+
python-template:
29+
name: Python template check
30+
runs-on: ubuntu-latest
31+
needs: changes
32+
if: needs.changes.outputs.templates == 'true' || github.event_name == 'workflow_dispatch'
33+
steps:
34+
- uses: actions/checkout@v6
35+
- uses: actions/setup-python@v5
36+
with:
37+
python-version: '3.x'
38+
- name: Validate Python template syntax
39+
run: python -m py_compile system-adapters/templates/python/adapter.py
40+
- name: Smoke test Python template (HTTP setup)
41+
run: |
42+
set -euo pipefail
43+
python system-adapters/templates/python/adapter.py --transport http --host 127.0.0.1 --port 18080 --path /jsonrpc > /tmp/python-template.log 2>&1 &
44+
pid=$!
45+
trap 'kill "$pid" 2>/dev/null || true' EXIT
46+
47+
for _ in {1..30}; do
48+
if curl -sS -o /tmp/python-resp.json -H 'Content-Type: application/json' \
49+
-d '{"jsonrpc":"2.0","id":1,"method":"setup","params":{"run_id":"00000000-0000-0000-0000-000000000000","datasets":{}}}' \
50+
http://127.0.0.1:18080/jsonrpc; then
51+
break
52+
fi
53+
sleep 1
54+
done
55+
56+
grep -q '"jsonrpc":"2.0"' /tmp/python-resp.json
57+
grep -q '"ok":true' /tmp/python-resp.json
58+
curl -sS -o /tmp/python-methods.json -H 'Content-Type: application/json' \
59+
-d '{"jsonrpc":"2.0","id":2,"method":"rpc.methods","params":{}}' \
60+
http://127.0.0.1:18080/jsonrpc
61+
grep -q '"methods"' /tmp/python-methods.json
62+
grep -q '"setup"' /tmp/python-methods.json
63+
grep -q '"metrics"' /tmp/python-methods.json
64+
65+
nodejs-template:
66+
name: Node.js template check
67+
runs-on: ubuntu-latest
68+
needs: changes
69+
if: needs.changes.outputs.templates == 'true' || github.event_name == 'workflow_dispatch'
70+
steps:
71+
- uses: actions/checkout@v6
72+
- uses: actions/setup-node@v4
73+
with:
74+
node-version: 'lts/*'
75+
- name: Validate Node.js template syntax
76+
run: node --check system-adapters/templates/nodejs/adapter.js
77+
- name: Smoke test Node.js template (HTTP setup)
78+
run: |
79+
set -euo pipefail
80+
node system-adapters/templates/nodejs/adapter.js --transport http --host 127.0.0.1 --port 18081 --path /jsonrpc > /tmp/node-template.log 2>&1 &
81+
pid=$!
82+
trap 'kill "$pid" 2>/dev/null || true' EXIT
83+
84+
for _ in {1..30}; do
85+
if curl -sS -o /tmp/node-resp.json -H 'Content-Type: application/json' \
86+
-d '{"jsonrpc":"2.0","id":1,"method":"setup","params":{"run_id":"00000000-0000-0000-0000-000000000000","datasets":{}}}' \
87+
http://127.0.0.1:18081/jsonrpc; then
88+
break
89+
fi
90+
sleep 1
91+
done
92+
93+
grep -q '"jsonrpc":"2.0"' /tmp/node-resp.json
94+
grep -q '"ok":true' /tmp/node-resp.json
95+
curl -sS -o /tmp/node-methods.json -H 'Content-Type: application/json' \
96+
-d '{"jsonrpc":"2.0","id":2,"method":"rpc.methods","params":{}}' \
97+
http://127.0.0.1:18081/jsonrpc
98+
grep -q '"methods"' /tmp/node-methods.json
99+
grep -q '"setup"' /tmp/node-methods.json
100+
grep -q '"metrics"' /tmp/node-methods.json
101+
102+
rust-template:
103+
name: Rust template build
104+
runs-on: ubuntu-latest
105+
needs: changes
106+
if: needs.changes.outputs.templates == 'true' || github.event_name == 'workflow_dispatch'
107+
steps:
108+
- uses: actions/checkout@v6
109+
- name: Setup Rust toolchain
110+
uses: actions-rust-lang/setup-rust-toolchain@v1
111+
with:
112+
toolchain: 1.91
113+
cache: false
114+
- name: Build Rust template
115+
run: cargo build --manifest-path system-adapters/templates/rust/Cargo.toml
116+
- name: Smoke test Rust template (HTTP setup)
117+
run: |
118+
set -euo pipefail
119+
cargo run --manifest-path system-adapters/templates/rust/Cargo.toml -- --transport http --host 127.0.0.1 --port 18082 --path /jsonrpc > /tmp/rust-template.log 2>&1 &
120+
pid=$!
121+
trap 'kill "$pid" 2>/dev/null || true' EXIT
122+
123+
for _ in {1..30}; do
124+
if curl -sS -o /tmp/rust-resp.json -H 'Content-Type: application/json' \
125+
-d '{"jsonrpc":"2.0","id":1,"method":"setup","params":{"run_id":"00000000-0000-0000-0000-000000000000","datasets":{}}}' \
126+
http://127.0.0.1:18082/jsonrpc; then
127+
break
128+
fi
129+
sleep 1
130+
done
131+
132+
grep -q '"jsonrpc":"2.0"' /tmp/rust-resp.json
133+
grep -q '"ok":true' /tmp/rust-resp.json
134+
curl -sS -o /tmp/rust-methods.json -H 'Content-Type: application/json' \
135+
-d '{"jsonrpc":"2.0","id":2,"method":"rpc.methods","params":{}}' \
136+
http://127.0.0.1:18082/jsonrpc
137+
grep -q '"methods"' /tmp/rust-methods.json
138+
grep -q '"setup"' /tmp/rust-methods.json
139+
grep -q '"metrics"' /tmp/rust-methods.json
140+
141+
go-template:
142+
name: Go template build
143+
runs-on: ubuntu-latest
144+
needs: changes
145+
if: needs.changes.outputs.templates == 'true' || github.event_name == 'workflow_dispatch'
146+
steps:
147+
- uses: actions/checkout@v6
148+
- uses: actions/setup-go@v5
149+
with:
150+
go-version-file: 'system-adapters/templates/go/go.mod'
151+
- name: Build Go template
152+
working-directory: system-adapters/templates/go
153+
run: go build ./...
154+
- name: Smoke test Go template (HTTP setup)
155+
run: |
156+
set -euo pipefail
157+
go run ./system-adapters/templates/go --transport http --host 127.0.0.1 --port 18083 --path /jsonrpc > /tmp/go-template.log 2>&1 &
158+
pid=$!
159+
trap 'kill "$pid" 2>/dev/null || true' EXIT
160+
161+
for _ in {1..30}; do
162+
if curl -sS -o /tmp/go-resp.json -H 'Content-Type: application/json' \
163+
-d '{"jsonrpc":"2.0","id":1,"method":"setup","params":{"run_id":"00000000-0000-0000-0000-000000000000","datasets":{}}}' \
164+
http://127.0.0.1:18083/jsonrpc; then
165+
break
166+
fi
167+
sleep 1
168+
done
169+
170+
grep -q '"jsonrpc":"2.0"' /tmp/go-resp.json
171+
grep -q '"ok":true' /tmp/go-resp.json
172+
curl -sS -o /tmp/go-methods.json -H 'Content-Type: application/json' \
173+
-d '{"jsonrpc":"2.0","id":2,"method":"rpc.methods","params":{}}' \
174+
http://127.0.0.1:18083/jsonrpc
175+
grep -q '"methods"' /tmp/go-methods.json
176+
grep -q '"setup"' /tmp/go-methods.json
177+
grep -q '"metrics"' /tmp/go-methods.json
178+
179+
java-template:
180+
name: Java template build
181+
runs-on: ubuntu-latest
182+
needs: changes
183+
if: needs.changes.outputs.templates == 'true' || github.event_name == 'workflow_dispatch'
184+
steps:
185+
- uses: actions/checkout@v6
186+
- uses: actions/setup-java@v4
187+
with:
188+
distribution: temurin
189+
java-version: '25'
190+
- name: Build Java template
191+
run: mvn -f system-adapters/templates/java/pom.xml -B -q -DskipTests compile
192+
- name: Smoke test Java template (HTTP setup)
193+
run: |
194+
set -euo pipefail
195+
mvn -f system-adapters/templates/java/pom.xml -B -q compile exec:java -Dexec.mainClass=com.spicebench.template.AdapterServer -Dexec.args="--transport http --host 127.0.0.1 --port 18084 --path /jsonrpc" > /tmp/java-template.log 2>&1 &
196+
pid=$!
197+
trap 'kill "$pid" 2>/dev/null || true' EXIT
198+
199+
for _ in {1..30}; do
200+
if curl -sS -o /tmp/java-resp.json -H 'Content-Type: application/json' \
201+
-d '{"jsonrpc":"2.0","id":1,"method":"setup","params":{"run_id":"00000000-0000-0000-0000-000000000000","datasets":{}}}' \
202+
http://127.0.0.1:18084/jsonrpc; then
203+
break
204+
fi
205+
sleep 1
206+
done
207+
208+
grep -q '"jsonrpc":"2.0"' /tmp/java-resp.json
209+
grep -q '"ok":true' /tmp/java-resp.json
210+
curl -sS -o /tmp/java-methods.json -H 'Content-Type: application/json' \
211+
-d '{"jsonrpc":"2.0","id":2,"method":"rpc.methods","params":{}}' \
212+
http://127.0.0.1:18084/jsonrpc
213+
grep -q '"methods"' /tmp/java-methods.json
214+
grep -q '"setup"' /tmp/java-methods.json
215+
grep -q '"metrics"' /tmp/java-methods.json

‎METRICS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ This document tracks every benchmark metric listed in the README, its OTel instr
55
## Pipeline Overview
66

77
```
8-
spicebench (OTel instruments)
8+
SpiceBench (OTel instruments)
99
├─ Query Driver ──► per-query gauges ──► Telemetry.emit() ──► telemetry.spiceai.io
1010
├─ StreamingOtlpExporter ──► real-time histograms/counters ──► --otlp-endpoint
1111
└─ SUT Adapter (JSON-RPC `metrics`) ──► scraped gauges ──► Telemetry.emit() ──► telemetry.spiceai.io
@@ -43,7 +43,7 @@ When `--otlp-endpoint` is configured, the following are exported every 5 seconds
4343

4444
## SUT Adapter `metrics` JSON-RPC Method
4545

46-
The system adapter protocol now includes a `metrics` JSON-RPC method that spicebench scrapes periodically (every 5s) when `--scrape-sut-metrics` is enabled.
46+
The system adapter protocol now includes a `metrics` JSON-RPC method that SpiceBench scrapes periodically (every 5s) when `--scrape-sut-metrics` is enabled.
4747

4848
### Request
4949

0 commit comments

Comments
 (0)