@@ -13,25 +13,28 @@ and the suite runs three stages against it:
1313- ** [ Picofuzz] ( ./picofuzz/ ) ** — runs the same four STF suites (` fallback ` ,
1414 ` safrole ` , ` storage ` , ` storage_light ` ) but does not check responses. Its
1515 only purpose is to measure block import performance (timings are displayed
16- on the [ dashboard] ( #dashboard ) ).
16+ on the [ dashboard] ( https://fluffylabs.github.io/jam-testing/ ) ).
1717- ** Fuzz testing** — one implementation (the "source") generates random
1818 blocks and another (the "target") must process them without crashing.
1919 Currently [ graymatter] ( https://github.com/jambrains/graymatter ) is
2020 available as a fuzz source, run against both the JAM ` tiny ` and ` full `
21- specs. Every team gets two demo fuzz jobs (5 000 blocks each on a shared
22- runner — one per spec); dedicated long-running runs cover both specs in
23- a single matrix.
21+ specs. Every team gets two demo fuzz jobs on a shared runner — one per
22+ spec — sized so each fits the runner's budget (5 000 blocks for ` tiny ` ,
23+ 500 for ` full ` ); dedicated long-running runs cover both specs in a
24+ single matrix.
2425
2526## Status
2627
2728The ** Performance** column covers minifuzz (conformance gate) + picofuzz
28- (timing). ** Demo (tiny)** and ** Demo (full)** are short fuzz runs (5 000
29- blocks each on a shared runner) executing the JAM ` tiny ` and ` full ` specs
30- respectively. ** Long-run** is a dedicated, multi-hour fuzz run that
31- exercises both specs in a matrix (single badge — red if either spec
32- fails). Targets pick which spec to run from the ` JAM_FUZZ_SPEC ` environment
33- variable; the matching ` --spec <value> ` is also passed to the graymatter
34- source command by the workflow.
29+ (timing). ** Demo (tiny)** and ** Demo (full)** are short fuzz runs on a
30+ shared runner executing the JAM ` tiny ` and ` full ` specs respectively
31+ (5 000 blocks for ` tiny ` , 500 for ` full ` — full-spec blocks are heavier,
32+ so the count is scaled down to fit a comparable wall-time budget).
33+ ** Long-run** is a dedicated, multi-hour fuzz run that exercises both specs
34+ in a matrix (single badge — red if either spec fails). Targets pick which
35+ spec to run from the ` JAM_FUZZ_SPEC ` environment variable; the matching
36+ ` --spec <value> ` is also passed to the graymatter source command by the
37+ workflow.
3538
3639| Team | Performance | Demo (tiny) | Demo (full) | Long-run |
3740| ------| -------------| -------------| -------------| ----------|
@@ -99,25 +102,40 @@ connections on the Unix socket. Two modes are supported:
99102
100103## Adding your team
101104
102- 1 . ** Provide a Docker image** that accepts a Unix socket path and speaks the
103- [ JAM Fuzz protocol] ( https://github.com/davxy/jam-conformance/tree/main/fuzz-proto ) .
105+ 1 . ** Provide a Docker image** that speaks the
106+ [ JAM Fuzz protocol] ( https://github.com/davxy/jam-conformance/tree/main/fuzz-proto )
107+ and follows the
108+ [ standard target packaging] ( https://github.com/davxy/jam-conformance/tree/main/fuzz-proto#standard-target-packaging ) .
104109 The image must be publicly pullable (or accessible to the runner).
105110
106- The harness sets the
107- [ standard target packaging] ( https://github.com/davxy/jam-conformance/tree/main/fuzz-proto#standard-target-packaging )
108- env vars on every target container: ` JAM_FUZZ=1 ` , ` JAM_FUZZ_SPEC=<tiny|full> ` ,
109- ` JAM_FUZZ_DATA_PATH=/shared/data ` , ` JAM_FUZZ_SOCK_PATH=/shared/jam_target.sock ` ,
110- ` JAM_FUZZ_LOG_LEVEL=debug ` . ` JAM_FUZZ_SPEC ` is set per-workflow; your image
111- must support both ` tiny ` and ` full ` and pick the right one from the env var.
112- New targets should read the socket path from ` JAM_FUZZ_SOCK_PATH ` and can
113- be launched with their image's default ` CMD ` — leave ` docker_cmd ` unset in
114- that case. For backwards compatibility, when ` docker_cmd ` is provided the
115- legacy ` {TARGET_SOCK} ` placeholder is substituted with the same socket
116- path, so existing targets keep working unchanged. ** Full-spec workflows
117- must omit ` docker_cmd ` ** — full-spec runs are env-only. Anything in ` docker_env `
118- is appended after the standard vars and can override them.
119- ` JAM_FUZZ_DATA_PATH ` is wiped between sequential fuzz-source runs to match
120- official testing's fresh-init behavior.
111+ The harness sets the standard env vars on every target container, and
112+ your image must read its configuration from them rather than from CLI
113+ args:
114+
115+ | Env var | Value |
116+ | ---| ---|
117+ | ` JAM_FUZZ ` | ` 1 ` |
118+ | ` JAM_FUZZ_SPEC ` | ` tiny ` or ` full ` (per-workflow) |
119+ | ` JAM_FUZZ_DATA_PATH ` | ` /shared/data ` |
120+ | ` JAM_FUZZ_SOCK_PATH ` | ` /shared/jam_target.sock ` |
121+ | ` JAM_FUZZ_LOG_LEVEL ` | ` debug ` |
122+
123+ Concretely:
124+
125+ - Your image must support ** both** ` tiny ` and ` full ` and pick the right
126+ one from ` JAM_FUZZ_SPEC ` .
127+ - Your image must read the socket path from ` JAM_FUZZ_SOCK_PATH ` and be
128+ launchable with its own default ` CMD ` — ** do not set ` docker_cmd ` **
129+ in any of the workflow files below. New teams should leave it unset.
130+ - Anything you put in ` docker_env ` is appended after the standard vars
131+ and can override them; use it for impl-specific tuning only.
132+ - ` JAM_FUZZ_DATA_PATH ` is wiped between sequential fuzz-source runs to
133+ match official testing's fresh-init behavior.
134+
135+ ` docker_cmd ` exists only as a backwards-compatibility shim for a few
136+ already-onboarded targets that predate standard packaging — when set,
137+ the legacy ` {TARGET_SOCK} ` placeholder is substituted with
138+ ` JAM_FUZZ_SOCK_PATH ` . Full-spec workflows must omit it unconditionally.
121139
1221402 . ** Create the performance workflow** at ` .github/workflows/<team>-performance.yml ` :
123141
@@ -135,17 +153,16 @@ connections on the Unix socket. Two modes are supported:
135153 with :
136154 target_name : myteam
137155 docker_image : ' ghcr.io/myorg/myimage:latest'
138- # Optional overrides:
139- # docker_cmd: 'fuzz --socket {TARGET_SOCK}' # only if your image needs args; new targets read JAM_FUZZ_SOCK_PATH
156+ # Optional overrides (do NOT set docker_cmd — see step 1):
140157 # docker_env: 'MY_VAR=value'
141158 # docker_memory: '512m'
142159 # docker_platform: 'linux/amd64'
143160 # readiness_pattern: 'Server ready'
144161 ```
145162
1461633 . ** Create the two demo fuzz workflows.** One for ` tiny ` , one for ` full ` .
147- Tiny may pass ` docker_cmd ` (legacy ` {TARGET_SOCK} ` substitution allowed);
148- full ** must not ** pass ` docker_cmd ` (env-only invocation) :
164+ Both rely on standard target packaging (env-only invocation); neither
165+ should set ` docker_cmd ` :
149166
150167 ``` yaml
151168 # .github/workflows/myteam-demo-tiny.yml
@@ -170,15 +187,13 @@ connections on the Unix socket. Two modes are supported:
170187 with :
171188 target_name : myteam
172189 docker_image : ' ghcr.io/myorg/myimage:latest'
173- docker_cmd : ' fuzz --socket {TARGET_SOCK}' # tiny: legacy placeholder OK
174190 spec : tiny
175191 mention : yourgithub
176192 ` ` `
177193
178194 ` ` ` yaml
179195 # .github/workflows/myteam-demo-full.yml
180- # Identical to demo-tiny except: spec: full and no docker_cmd.
181- # Full-spec runs are env-only — your target must read JAM_FUZZ_SOCK_PATH.
196+ # Identical to demo-tiny except: spec: full and num_blocks scaled down.
182197 name : " Demo (full): myteam"
183198
184199 on :
@@ -201,6 +216,7 @@ connections on the Unix socket. Two modes are supported:
201216 target_name : myteam
202217 docker_image : ' ghcr.io/myorg/myimage:latest'
203218 spec : full
219+ num_blocks : 500 # full-spec blocks are heavier; tiny uses the 5000 default
204220 mention : yourgithub
205221 ` ` `
206222
@@ -221,7 +237,7 @@ connections on the Unix socket. Two modes are supported:
221237|---|---|---|---|
222238| `target_name` | yes | — | Your implementation name |
223239| `docker_image` | yes | — | Full image reference |
224- | `docker_cmd` | no | `""` | Override container command. `{TARGET_SOCK}` is substituted with the socket path. Leave empty for targets that read `JAM_FUZZ_SOCK_PATH` from env . |
240+ | `docker_cmd` | no | `""` | **Legacy compat shim — new teams should leave this unset.** Override container command for pre-standard-packaging targets; `{TARGET_SOCK}` is substituted with `JAM_FUZZ_SOCK_PATH`. Must be empty in any full-spec workflow . |
225241| `docker_env` | no | `""` | Space-separated `KEY=VALUE` pairs passed as `-e` flags |
226242| `docker_memory` | no | `"512m"` | Container memory limit |
227243| `docker_platform` | no | `"linux/amd64"` | Platform for `docker pull` |
@@ -284,6 +300,8 @@ up-to-date.
284300
285301```
286302.github/workflows/
303+ ci.yml # Unit tests for the suite itself
304+ deploy-dashboard.yml # Builds and deploys the dashboard to GitHub Pages
287305 reusable-picofuzz.yml # Core reusable workflow (minifuzz + picofuzz)
288306 demo-source.yml # Reusable demo fuzz source workflow (tiny|full)
289307 graymatter-fuzz-source.yml # Reusable long-running fuzz source workflow
@@ -292,14 +310,17 @@ up-to-date.
292310 <team >-demo-full.yml # Per-team demo fuzz against the full spec
293311 <team >-fuzz.yml # Per-team long-running fuzz (matrix over [ tiny, full] )
294312 # — only for teams with dedicated runners
313+ agents.md # Contributor / AI-agent guide for this repo
295314minifuzz/ # Minifuzz Docker image (Python fuzz example runner)
296315minifuzz-traces/ # Captured request-response pairs from typeberry
297316 populate.sh # Script to regenerate traces
298317 {suite}/ # Per-suite trace files (fallback, safrole, etc.)
299318picofuzz/ # Picofuzz tool (fuzz protocol client + capture mode)
319+ scripts/ # Helpers used by deploy-dashboard.yml (CSV → JSON, history)
300320tests/
301321 common.ts # Target startup & shared helpers
302322 external-process.ts # Docker process management
323+ config.test.ts # Unit tests over the workflow config files
303324 minifuzz/
304325 common.ts # Minifuzz test harness
305326 * .test.ts # Minifuzz suites (forks, no_forks, fallback, safrole, ...)
@@ -310,6 +331,7 @@ tests/
310331 common.ts # Fuzz source test harness
311332 fuzz.test.ts # Fuzz source test entry point
312333teams/<team >/ # Team-specific scripts & data
334+ dashboard/ # Git submodule: jam-conformance-dashboard (Next.js app)
313335picofuzz-stf-data/ # Git submodule: STF test traces
314336picofuzz-conformance-data/ # Git submodule: jam-conformance (minifuzz examples)
315337```
0 commit comments