Skip to content
This repository was archived by the owner on Jun 7, 2026. It is now read-only.

Commit 62964e3

Browse files
tomusdrwclaude
andauthored
Sync README: dashboard link, block counts, target packaging (#44)
- Fix broken #dashboard anchor; point at the real Pages URL. - Correct demo block counts (tiny: 5000, full: 500) in the intro and Status section, and add num_blocks: 500 to the demo-full template. - Make standard target packaging the canonical onboarding path; mark docker_cmd as a legacy compat shim and drop it from the templates. - Backfill missing items in Project structure (ci.yml, deploy-dashboard.yml, scripts/, agents.md, dashboard/ submodule, tests/config.test.ts). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 9388732 commit 62964e3

1 file changed

Lines changed: 58 additions & 36 deletions

File tree

README.md

Lines changed: 58 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -13,25 +13,28 @@ and the suite runs three stages against it:
1313
- **[Picofuzz](./picofuzz/)** — runs the same four STF suites (`fallback`,
1414
`safrole`, `storage`, `storage_light`) but does not check responses. Its
1515
only purpose is to measure block import performance (timings are displayed
16-
on the [dashboard](#dashboard)).
16+
on the [dashboard](https://fluffylabs.github.io/jam-testing/)).
1717
- **Fuzz testing** — one implementation (the "source") generates random
1818
blocks and another (the "target") must process them without crashing.
1919
Currently [graymatter](https://github.com/jambrains/graymatter) is
2020
available as a fuzz source, run against both the JAM `tiny` and `full`
21-
specs. Every team gets two demo fuzz jobs (5 000 blocks each on a shared
22-
runner — one per spec); dedicated long-running runs cover both specs in
23-
a single matrix.
21+
specs. Every team gets two demo fuzz jobs on a shared runner — one per
22+
spec — sized so each fits the runner's budget (5 000 blocks for `tiny`,
23+
500 for `full`); dedicated long-running runs cover both specs in a
24+
single matrix.
2425

2526
## Status
2627

2728
The **Performance** column covers minifuzz (conformance gate) + picofuzz
28-
(timing). **Demo (tiny)** and **Demo (full)** are short fuzz runs (5 000
29-
blocks each on a shared runner) executing the JAM `tiny` and `full` specs
30-
respectively. **Long-run** is a dedicated, multi-hour fuzz run that
31-
exercises both specs in a matrix (single badge — red if either spec
32-
fails). Targets pick which spec to run from the `JAM_FUZZ_SPEC` environment
33-
variable; the matching `--spec <value>` is also passed to the graymatter
34-
source command by the workflow.
29+
(timing). **Demo (tiny)** and **Demo (full)** are short fuzz runs on a
30+
shared runner executing the JAM `tiny` and `full` specs respectively
31+
(5 000 blocks for `tiny`, 500 for `full` — full-spec blocks are heavier,
32+
so the count is scaled down to fit a comparable wall-time budget).
33+
**Long-run** is a dedicated, multi-hour fuzz run that exercises both specs
34+
in a matrix (single badge — red if either spec fails). Targets pick which
35+
spec to run from the `JAM_FUZZ_SPEC` environment variable; the matching
36+
`--spec <value>` is also passed to the graymatter source command by the
37+
workflow.
3538

3639
| Team | Performance | Demo (tiny) | Demo (full) | Long-run |
3740
|------|-------------|-------------|-------------|----------|
@@ -99,25 +102,40 @@ connections on the Unix socket. Two modes are supported:
99102

100103
## Adding your team
101104

102-
1. **Provide a Docker image** that accepts a Unix socket path and speaks the
103-
[JAM Fuzz protocol](https://github.com/davxy/jam-conformance/tree/main/fuzz-proto).
105+
1. **Provide a Docker image** that speaks the
106+
[JAM Fuzz protocol](https://github.com/davxy/jam-conformance/tree/main/fuzz-proto)
107+
and follows the
108+
[standard target packaging](https://github.com/davxy/jam-conformance/tree/main/fuzz-proto#standard-target-packaging).
104109
The image must be publicly pullable (or accessible to the runner).
105110

106-
The harness sets the
107-
[standard target packaging](https://github.com/davxy/jam-conformance/tree/main/fuzz-proto#standard-target-packaging)
108-
env vars on every target container: `JAM_FUZZ=1`, `JAM_FUZZ_SPEC=<tiny|full>`,
109-
`JAM_FUZZ_DATA_PATH=/shared/data`, `JAM_FUZZ_SOCK_PATH=/shared/jam_target.sock`,
110-
`JAM_FUZZ_LOG_LEVEL=debug`. `JAM_FUZZ_SPEC` is set per-workflow; your image
111-
must support both `tiny` and `full` and pick the right one from the env var.
112-
New targets should read the socket path from `JAM_FUZZ_SOCK_PATH` and can
113-
be launched with their image's default `CMD` — leave `docker_cmd` unset in
114-
that case. For backwards compatibility, when `docker_cmd` is provided the
115-
legacy `{TARGET_SOCK}` placeholder is substituted with the same socket
116-
path, so existing targets keep working unchanged. **Full-spec workflows
117-
must omit `docker_cmd`** — full-spec runs are env-only. Anything in `docker_env`
118-
is appended after the standard vars and can override them.
119-
`JAM_FUZZ_DATA_PATH` is wiped between sequential fuzz-source runs to match
120-
official testing's fresh-init behavior.
111+
The harness sets the standard env vars on every target container, and
112+
your image must read its configuration from them rather than from CLI
113+
args:
114+
115+
| Env var | Value |
116+
|---|---|
117+
| `JAM_FUZZ` | `1` |
118+
| `JAM_FUZZ_SPEC` | `tiny` or `full` (per-workflow) |
119+
| `JAM_FUZZ_DATA_PATH` | `/shared/data` |
120+
| `JAM_FUZZ_SOCK_PATH` | `/shared/jam_target.sock` |
121+
| `JAM_FUZZ_LOG_LEVEL` | `debug` |
122+
123+
Concretely:
124+
125+
- Your image must support **both** `tiny` and `full` and pick the right
126+
one from `JAM_FUZZ_SPEC`.
127+
- Your image must read the socket path from `JAM_FUZZ_SOCK_PATH` and be
128+
launchable with its own default `CMD`**do not set `docker_cmd`**
129+
in any of the workflow files below. New teams should leave it unset.
130+
- Anything you put in `docker_env` is appended after the standard vars
131+
and can override them; use it for impl-specific tuning only.
132+
- `JAM_FUZZ_DATA_PATH` is wiped between sequential fuzz-source runs to
133+
match official testing's fresh-init behavior.
134+
135+
`docker_cmd` exists only as a backwards-compatibility shim for a few
136+
already-onboarded targets that predate standard packaging — when set,
137+
the legacy `{TARGET_SOCK}` placeholder is substituted with
138+
`JAM_FUZZ_SOCK_PATH`. Full-spec workflows must omit it unconditionally.
121139

122140
2. **Create the performance workflow** at `.github/workflows/<team>-performance.yml`:
123141

@@ -135,17 +153,16 @@ connections on the Unix socket. Two modes are supported:
135153
with:
136154
target_name: myteam
137155
docker_image: 'ghcr.io/myorg/myimage:latest'
138-
# Optional overrides:
139-
# docker_cmd: 'fuzz --socket {TARGET_SOCK}' # only if your image needs args; new targets read JAM_FUZZ_SOCK_PATH
156+
# Optional overrides (do NOT set docker_cmd — see step 1):
140157
# docker_env: 'MY_VAR=value'
141158
# docker_memory: '512m'
142159
# docker_platform: 'linux/amd64'
143160
# readiness_pattern: 'Server ready'
144161
```
145162

146163
3. **Create the two demo fuzz workflows.** One for `tiny`, one for `full`.
147-
Tiny may pass `docker_cmd` (legacy `{TARGET_SOCK}` substitution allowed);
148-
full **must not** pass `docker_cmd` (env-only invocation):
164+
Both rely on standard target packaging (env-only invocation); neither
165+
should set `docker_cmd`:
149166

150167
```yaml
151168
# .github/workflows/myteam-demo-tiny.yml
@@ -170,15 +187,13 @@ connections on the Unix socket. Two modes are supported:
170187
with:
171188
target_name: myteam
172189
docker_image: 'ghcr.io/myorg/myimage:latest'
173-
docker_cmd: 'fuzz --socket {TARGET_SOCK}' # tiny: legacy placeholder OK
174190
spec: tiny
175191
mention: yourgithub
176192
```
177193
178194
```yaml
179195
# .github/workflows/myteam-demo-full.yml
180-
# Identical to demo-tiny except: spec: full and no docker_cmd.
181-
# Full-spec runs are env-only — your target must read JAM_FUZZ_SOCK_PATH.
196+
# Identical to demo-tiny except: spec: full and num_blocks scaled down.
182197
name: "Demo (full): myteam"
183198

184199
on:
@@ -201,6 +216,7 @@ connections on the Unix socket. Two modes are supported:
201216
target_name: myteam
202217
docker_image: 'ghcr.io/myorg/myimage:latest'
203218
spec: full
219+
num_blocks: 500 # full-spec blocks are heavier; tiny uses the 5000 default
204220
mention: yourgithub
205221
```
206222
@@ -221,7 +237,7 @@ connections on the Unix socket. Two modes are supported:
221237
|---|---|---|---|
222238
| `target_name` | yes | — | Your implementation name |
223239
| `docker_image` | yes | — | Full image reference |
224-
| `docker_cmd` | no | `""` | Override container command. `{TARGET_SOCK}` is substituted with the socket path. Leave empty for targets that read `JAM_FUZZ_SOCK_PATH` from env. |
240+
| `docker_cmd` | no | `""` | **Legacy compat shim — new teams should leave this unset.** Override container command for pre-standard-packaging targets; `{TARGET_SOCK}` is substituted with `JAM_FUZZ_SOCK_PATH`. Must be empty in any full-spec workflow. |
225241
| `docker_env` | no | `""` | Space-separated `KEY=VALUE` pairs passed as `-e` flags |
226242
| `docker_memory` | no | `"512m"` | Container memory limit |
227243
| `docker_platform` | no | `"linux/amd64"` | Platform for `docker pull` |
@@ -284,6 +300,8 @@ up-to-date.
284300

285301
```
286302
.github/workflows/
303+
ci.yml # Unit tests for the suite itself
304+
deploy-dashboard.yml # Builds and deploys the dashboard to GitHub Pages
287305
reusable-picofuzz.yml # Core reusable workflow (minifuzz + picofuzz)
288306
demo-source.yml # Reusable demo fuzz source workflow (tiny|full)
289307
graymatter-fuzz-source.yml # Reusable long-running fuzz source workflow
@@ -292,14 +310,17 @@ up-to-date.
292310
<team>-demo-full.yml # Per-team demo fuzz against the full spec
293311
<team>-fuzz.yml # Per-team long-running fuzz (matrix over [tiny, full])
294312
# — only for teams with dedicated runners
313+
agents.md # Contributor / AI-agent guide for this repo
295314
minifuzz/ # Minifuzz Docker image (Python fuzz example runner)
296315
minifuzz-traces/ # Captured request-response pairs from typeberry
297316
populate.sh # Script to regenerate traces
298317
{suite}/ # Per-suite trace files (fallback, safrole, etc.)
299318
picofuzz/ # Picofuzz tool (fuzz protocol client + capture mode)
319+
scripts/ # Helpers used by deploy-dashboard.yml (CSV → JSON, history)
300320
tests/
301321
common.ts # Target startup & shared helpers
302322
external-process.ts # Docker process management
323+
config.test.ts # Unit tests over the workflow config files
303324
minifuzz/
304325
common.ts # Minifuzz test harness
305326
*.test.ts # Minifuzz suites (forks, no_forks, fallback, safrole, ...)
@@ -310,6 +331,7 @@ tests/
310331
common.ts # Fuzz source test harness
311332
fuzz.test.ts # Fuzz source test entry point
312333
teams/<team>/ # Team-specific scripts & data
334+
dashboard/ # Git submodule: jam-conformance-dashboard (Next.js app)
313335
picofuzz-stf-data/ # Git submodule: STF test traces
314336
picofuzz-conformance-data/ # Git submodule: jam-conformance (minifuzz examples)
315337
```

0 commit comments

Comments
 (0)