Skip to content

Commit 5299ee1

Browse files
committed
docs: finalize launch metrics and multi-channel launch copy
1 parent ab9714e commit 5299ee1

5 files changed

Lines changed: 187 additions & 3 deletions

File tree

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ First-run onboarding:
3737

3838
- Release notes: [docs/RELEASE_NOTES_v1.0.0.md](docs/RELEASE_NOTES_v1.0.0.md)
3939
- Launch thread copy: [docs/LAUNCH_THREAD_COPY_v1.0.0.md](docs/LAUNCH_THREAD_COPY_v1.0.0.md)
40+
- Launch day checklist: [docs/LAUNCH_DAY_CHECKLIST_2026-03-30.md](docs/LAUNCH_DAY_CHECKLIST_2026-03-30.md)
4041
- 8k validation protocol: [docs/VALIDATION_PROTOCOL_8K.md](docs/VALIDATION_PROTOCOL_8K.md)
4142
- Batch validation report: [reports/launch_8k_batch_validation_2026-03-28.md](reports/launch_8k_batch_validation_2026-03-28.md)
4243
- Device/model guide: [docs/DEVICE_MODEL_GUIDE_2026-03-30.md](docs/DEVICE_MODEL_GUIDE_2026-03-30.md)
Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# Launch Day Checklist (WebLLM Bench + Qwen2.5 ctx8192)
2+
3+
## 1) Final links
4+
5+
- Live app: `https://ar5en1c.github.io/webllm-bench/bench.html`
6+
- WebLLM Bench repo: `https://github.com/Ar5en1c/webllm-bench`
7+
- 8k workflow repo: `https://github.com/Ar5en1c/qwen2.5-1.5b-ctx8192-mlc`
8+
- HF hosted 8k model: `https://huggingface.co/Ar5en1c/Qwen2.5-1.5B-Instruct-q4f16_1-MLC-ctx8192`
9+
10+
## 2) Claims to use (and only these)
11+
12+
- Fixed-profile parity (`1024/128/10`): median deltas stay around parity band.
13+
- Long-output parity (`1024/512/5`):
14+
- Throughput delta (8k vs 4k): `-1.96%`
15+
- Latency delta: `+1.94%`
16+
- Decode delta: `-1.98%`
17+
- TTFT delta: `+0.63%`
18+
- Functional gate:
19+
- 8k handles >4k retrieval prompts
20+
- 4k fails at 5813 prompt tokens (ctx 4096)
21+
- Limitation:
22+
- Browser WebGPU does not expose exact live GPU VRAM telemetry.
23+
24+
## 3) Post order (same day)
25+
26+
1. GitHub release (both repos)
27+
2. X thread
28+
3. LinkedIn post
29+
4. r/LocalLLaMA post
30+
5. WebLLM/MLC Discord post
31+
6. Hacker News post
32+
33+
## 4) Channel-ready short copy
34+
35+
### Reddit (r/LocalLLaMA) title
36+
37+
`[OSS] WebLLM Bench v1.0.0 + Qwen2.5-1.5B ctx8192 hosted preset (parity-tested vs official 4k build)`
38+
39+
### Reddit body
40+
41+
`Open-sourced two repos:
42+
43+
- WebLLM Bench (browser-native benchmark/chat/compare): https://github.com/Ar5en1c/webllm-bench
44+
- Qwen2.5-1.5B ctx8192 workflow package: https://github.com/Ar5en1c/qwen2.5-1.5b-ctx8192-mlc
45+
- Live app: https://ar5en1c.github.io/webllm-bench/bench.html
46+
- HF hosted model: https://huggingface.co/Ar5en1c/Qwen2.5-1.5B-Instruct-q4f16_1-MLC-ctx8192
47+
48+
Measured on fixed profile (1024/128/10): parity band vs official 4k baseline.
49+
Latest long-output stress run (1024/512/5): throughput delta -1.96%, latency +1.94%, decode -1.98%, TTFT +0.63%.
50+
51+
Functional gate: 8k handles >4k retrieval prompts; 4k baseline overflows at 5813 tokens.
52+
53+
Limitation: browser WebGPU doesn’t expose exact live GPU VRAM telemetry, so VRAM values are model metadata/proxy signals.`
54+
55+
### Discord short post
56+
57+
`Released: WebLLM Bench v1.0.0 + hosted Qwen2.5-1.5B ctx8192 preset.
58+
Live: https://ar5en1c.github.io/webllm-bench/bench.html
59+
Repo: https://github.com/Ar5en1c/webllm-bench
60+
8k model: https://huggingface.co/Ar5en1c/Qwen2.5-1.5B-Instruct-q4f16_1-MLC-ctx8192
61+
Result: 8k remains in ~2% parity band vs official 4k baseline on stress profile (1024/512/5).`
62+
63+
### Hacker News title
64+
65+
`Show HN: WebLLM Bench – benchmark/chat/compare local browser LLMs (+ hosted Qwen2.5-1.5B 8k preset)`
66+
67+
### Hacker News post text
68+
69+
`I open-sourced WebLLM Bench (browser-native local LLM benchmark/chat/compare toolkit) and a reproducible workflow package for a hosted Qwen2.5-1.5B ctx8192 build.
70+
71+
Live app:
72+
https://ar5en1c.github.io/webllm-bench/bench.html
73+
74+
Repos:
75+
https://github.com/Ar5en1c/webllm-bench
76+
https://github.com/Ar5en1c/qwen2.5-1.5b-ctx8192-mlc
77+
78+
HF model:
79+
https://huggingface.co/Ar5en1c/Qwen2.5-1.5B-Instruct-q4f16_1-MLC-ctx8192
80+
81+
Measured parity summary:
82+
- Fixed profile (1024/128/10): median deltas in parity band vs official 4k baseline
83+
- Stress profile (1024/512/5): throughput delta -1.96%, latency +1.94%
84+
85+
Functional check:
86+
- 8k passes >4k retrieval prompts
87+
- 4k overflows at 5813 prompt tokens
88+
89+
Limitations are explicitly documented (browser WebGPU VRAM telemetry is not exact).`
90+
91+
## 5) First 24h response loop
92+
93+
1. Reply to every issue/comment within 12 hours.
94+
2. Ask reporters to include:
95+
- Browser/version
96+
- Model ID
97+
- Prompt/max tokens/iterations
98+
- Export JSON
99+
3. For reproducibility disputes, point to:
100+
- `docs/VALIDATION_PROTOCOL_8K.md`
101+
- `reports/launch_8k_batch_validation_2026-03-28.md`

docs/LAUNCH_THREAD_COPY_v1.0.0.md

Lines changed: 48 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
Date: 2026-03-30
44
Source metrics: `reports/launch_8k_batch_validation_2026-03-28.md`
5+
Latest stress parity run: 2026-03-30 (1024/512/5)
56

67
## 1) GitHub Release
78

@@ -31,10 +32,18 @@ What ships:
3132
- Decode delta: -0.53% .. +1.58%
3233
- Latency delta: -1.33% .. +0.48%
3334
35+
Latest long-output stress parity (1024/512/5):
36+
- 4k baseline: latency 32248 ms, throughput 15.85 tok/s, decode 17.21 tok/s, TTFT 2393 ms
37+
- 8k custom: latency 32873 ms, throughput 15.54 tok/s, decode 16.87 tok/s, TTFT 2408 ms
38+
- Delta (8k vs 4k): latency +1.94%, throughput -1.96%, decode -1.98%, TTFT +0.63%
39+
3440
Functional context gate:
3541
- 8k build passes >4k retrieval prompt checks
3642
- 4k baseline overflows at 5813 prompt tokens (context window size 4096)
3743
44+
Stability note:
45+
- In long-output stress mode, one exit(1) runtime event was observed and auto-recovered by unload/reload retry; run completed.
46+
3847
Important limitation:
3948
- Browser WebGPU does not provide exact live GPU VRAM counters. VRAM numbers are model metadata/proxy signals.
4049
@@ -48,6 +57,8 @@ Report:
4857
4958
Hosted preset artifacts:
5059
- https://huggingface.co/Ar5en1c/Qwen2.5-1.5B-Instruct-q4f16_1-MLC-ctx8192
60+
Live app:
61+
- https://ar5en1c.github.io/webllm-bench/bench.html
5162
```
5263

5364
## 2) X Thread (Exact Copy)
@@ -70,7 +81,7 @@ New in this release: one-click hosted 8k preset in the UI.
7081

7182
### Post 2
7283
```
73-
8k validation on fixed profile (1024 prompt / 128 output / 10 iterations), using 8 exported runs:
84+
8k validation on fixed profile (1024/128/10), using 8 exported runs:
7485
7586
Median delta (8k custom vs 4k baseline):
7687
- Decode TPS: +0.11%
@@ -90,22 +101,34 @@ Interpretation: parity band, no material regression on this profile.
90101

91102
### Post 4
92103
```
104+
Latest long-output stress parity (1024/512/5):
105+
- 4k throughput: 15.85 tok/s
106+
- 8k throughput: 15.54 tok/s
107+
- delta: -1.96%
108+
109+
Other deltas:
110+
- latency: +1.94%
111+
- decode: -1.98%
112+
```
113+
114+
### Post 5
115+
```
93116
Functional gate result:
94117
- ctx8192 build handles >4k retrieval prompts
95118
- ctx4096 baseline fails at 5813 prompt tokens with context overflow
96119
97120
So this is not only a config change; it is validated long-context behavior.
98121
```
99122

100-
### Post 5
123+
### Post 6
101124
```
102125
Limitation (explicit):
103126
Browser WebGPU does not expose exact live VRAM usage counters.
104127
105128
Published VRAM numbers are model metadata / proxy signals, not hardware telemetry.
106129
```
107130

108-
### Post 6
131+
### Post 7
109132
```
110133
Everything is reproducible from repo artifacts:
111134
- benchmark JSON exports
@@ -118,6 +141,15 @@ npm run report:8k:batch
118141
npm run launch:draft
119142
```
120143

144+
### Post 8
145+
```
146+
Links:
147+
- Live: https://ar5en1c.github.io/webllm-bench/bench.html
148+
- Repo: https://github.com/Ar5en1c/webllm-bench
149+
- 8k model: https://huggingface.co/Ar5en1c/Qwen2.5-1.5B-Instruct-q4f16_1-MLC-ctx8192
150+
- 8k workflow package: https://github.com/Ar5en1c/qwen2.5-1.5b-ctx8192-mlc
151+
```
152+
121153
## 3) LinkedIn Post (Exact Copy)
122154

123155
```
@@ -137,10 +169,23 @@ Range across runs:
137169
- Decode delta: -0.53% to +1.58%
138170
- Latency delta: -1.33% to +0.48%
139171
172+
Latest long-output stress parity run (1024 prompt, 512 output, 5 iterations):
173+
- 4k baseline throughput: 15.85 tok/s
174+
- 8k custom throughput: 15.54 tok/s
175+
- Delta: -1.96%
176+
177+
Other stress deltas:
178+
- Latency: +1.94%
179+
- Decode: -1.98%
180+
- TTFT: +0.63%
181+
140182
Functional context check also passed:
141183
- ctx8192 build handled prompts beyond 4k context
142184
- ctx4096 baseline overflowed at 5813 prompt tokens
143185
186+
Stress stability note:
187+
- One exit(1) runtime event occurred and was auto-recovered by unload/reload retry; run completed.
188+
144189
One limitation to state clearly: browser WebGPU does not expose exact live VRAM counters, so VRAM values are metadata/proxy signals.
145190
146191
All claims are backed by exported JSON reports and a reproducible validation script in the repository.

docs/PUBLISH_BOTH_OSS_PLAYBOOK.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,7 @@ git push -u origin main
7373

7474
Use:
7575
- `docs/LAUNCH_THREAD_COPY_v1.0.0.md`
76+
- `docs/LAUNCH_DAY_CHECKLIST_2026-03-30.md`
7677

7778
Post channels (same day):
7879
- X thread

docs/RELEASE_NOTES_v1.0.0.md

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,9 +41,45 @@ Functional context gate:
4141
- 8k model handles a >4k retrieval prompt.
4242
- 4k baseline overflows at `5813` prompt tokens (`context window size: 4096`).
4343

44+
## Long-output stress parity (latest run, 2026-03-30)
45+
46+
Profile:
47+
- `promptTokens=1024`
48+
- `maxTokens=512`
49+
- `iterations=5`
50+
- `forceFullLength=true`
51+
52+
4k baseline (official):
53+
- Avg latency: `32248 ms`
54+
- Throughput: `15.85 tok/s`
55+
- Decode TPS: `17.21 tok/s`
56+
- Prefill TPS: `338.11 tok/s`
57+
- TTFT: `2393 ms`
58+
- ms/token: `63.1`
59+
60+
8k custom (hosted):
61+
- Avg latency: `32873 ms`
62+
- Throughput: `15.54 tok/s`
63+
- Decode TPS: `16.87 tok/s`
64+
- Prefill TPS: `335.96 tok/s`
65+
- TTFT: `2408 ms`
66+
- ms/token: `64.3`
67+
68+
Delta (8k vs 4k):
69+
- Latency: `+1.94%`
70+
- Throughput: `-1.96%`
71+
- Decode TPS: `-1.98%`
72+
- Prefill TPS: `-0.64%`
73+
- TTFT: `+0.63%`
74+
- ms/token: `+1.90%`
75+
76+
Runtime stability note:
77+
- One `exit(1)` event occurred in stress mode and was auto-recovered by unload/reload retry; benchmark completed.
78+
4479
## Claim-safe summary
4580

4681
- [TESTED] Custom ctx8192 model is stable and remains in parity band vs the official 4k baseline on the fixed benchmark profile above.
82+
- [TESTED] Long-output stress profile (`1024/512/5`) also remains in parity band (about `~2%` delta).
4783
- [TESTED] The ctx8192 model passes functional >4k prompt handling where 4k fails by context limit.
4884
- [LIMITATION] Browser WebGPU does not expose exact live GPU VRAM usage counters; VRAM values are model metadata and JS heap proxies.
4985

0 commit comments

Comments
 (0)