22
33![ yzma in browser] ( ./images/yzma-in-browser.png )
44
5- A chat page where the whole language model runs in the tab. No server, no API key, nothing sent anywhere.
5+ A chat page that runs the language model in the browser tab. There is no server,
6+ no API key, and no data goes anywhere.
67
78** < https://hybridgroup.github.io/yzma-wasm-example/ > **
89
910## How it works
1011
1112[ ![ yzma logo] ( https://raw.githubusercontent.com/hybridgroup/yzma/refs/heads/main/images/yzma-logo-full-color-small.png )] ( https://github.com/hybridgroup/yzma )
1213
13- The code is written in Go and compiled by TinyGo. Using the [ yzma] ( https://github.com/hybridgroup/yzma ) package, the web page runs [ llama.cpp] ( https://github.com/ggml-org/llama.cpp ) , which has been compiled into a WebAssembly module. It all runs in a local Web Worker for best browser performance.
14+ The code is written in Go and compiled by TinyGo. The page uses the
15+ [ yzma] ( https://github.com/hybridgroup/yzma ) package to run
16+ [ llama.cpp] ( https://github.com/ggml-org/llama.cpp ) , which is compiled into a
17+ WebAssembly module. All of it runs in a local Web Worker for the best browser
18+ performance.
1419
1520```
1621 index.html
@@ -21,91 +26,92 @@ The code is written in Go and compiled by TinyGo. Using the [yzma](https://githu
2126 (Go, TinyGo) -> (llama.cpp, Emscripten)
2227```
2328
24- The page stores no record of the conversation. The conversation data is only in the TinyGo WASM module, and each turn puts the whole conversation back through the model.
29+ The page keeps no record of the conversation. The conversation stays in the
30+ TinyGo WASM module. Each turn sends the full conversation through the model
31+ again.
2532
2633## Build and run
2734
28- Needs [ TinyGo] ( https://tinygo.org/getting-started/install/ ) 0.41 or later, Go
35+ You need [ TinyGo] ( https://tinygo.org/getting-started/install/ ) 0.41 or later, Go
29361.26, ` jq ` , and ` node ` for the test.
3037
3138```
3239make build
3340make serve
3441```
3542
36- Then open < http://localhost:8080 > , press ** Load** , and wait for the model to come
37- down . The default is
38- [ Qwen2.5-0.5B-Instruct Q4_K_M] ( https://huggingface.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF ) ,
39- about 400 MB, which the browser caches. Any GGUF URL works as long as the host
40- sends CORS headers, which Hugging Face does .
43+ Open < http://localhost:8080 > , push ** Load** , and wait for the download of the
44+ model . The default model is
45+ [ Qwen2.5-0.5B-Instruct Q4_K_M] ( https://huggingface.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF ) .
46+ It is approximately 400 MB, and the browser caches it. You can use any GGUF URL
47+ if the host sends CORS headers. Hugging Face sends them .
4148
42- ` make build ` downloads about 14 MB of llama.cpp into ` build/ ` , compiles the Go
43- program, and copies the page. Nothing binary lives in the repo .
49+ ` make build ` downloads approximately 14 MB of llama.cpp into ` build/ ` , compiles
50+ the Go program, and copies the page. No binary files are in the repository .
4451
4552## The test
4653
47- A two turn conversation, in Node, with no browser. The second question only
48- makes sense if the first one is still in the prompt, so a sensible answer means
49- the chat template came out right:
54+ The test holds a two turn conversation in Node, with no browser. The second
55+ question makes sense only if the first question is still in the prompt. Thus a
56+ sensible answer shows that the chat template is correct.
5057
5158```
5259make test MODEL=~/models/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf
5360```
5461
55- ## Threads, and why there is a service worker
62+ ## Threads and the service worker
5663
57- llama.cpp comes in three WebAssembly builds, and ` yzma-loader.js ` takes the best
58- one the browser can run:
64+ llama.cpp has three WebAssembly builds. ` yzma-loader.js ` selects the best build
65+ that the browser can run.
5966
6067| Build | What it needs |
6168| --- | --- |
6269| ` yzma_wasm_webgpu ` | WebGPU with f16 shaders, and JSPI. Chrome and Edge 137 and later. |
63- | ` yzma_wasm_mt ` | ` SharedArrayBuffer ` , so a page with the COOP and COEP headers. |
64- | ` yzma_wasm ` | Nothing. It works everywhere. |
70+ | ` yzma_wasm_mt ` | ` SharedArrayBuffer ` , thus a page with the COOP and COEP headers. |
71+ | ` yzma_wasm ` | Nothing. It runs everywhere. |
6572
66- GitHub Pages cannot send the COOP and COEP headers, so without help a browser
67- gives the page no ` SharedArrayBuffer ` and llama.cpp runs on one thread. In Node
73+ GitHub Pages cannot send the COOP and COEP headers. Without help, the browser
74+ gives the page no ` SharedArrayBuffer ` , and llama.cpp runs on one thread. In Node
6875that is the difference between 0.9 and 9.6 tokens a second on Qwen2.5 0.5B.
6976
70- So the page loads
71- [ ` coi-serviceworker.js ` ] ( https://github.com/gzuidhof/coi-serviceworker ) first.
72- It registers a service worker that adds the two headers and reloads the page
73- once, and from then on the page is cross- origin isolated and the loader takes
74- the build with every thread. It works on localhost too, so a plain static server
75- is enough for development. ` make serve ` sets the headers as well , which makes no
76- difference but does no harm .
77+ Thus the page loads
78+ [ ` coi-serviceworker.js ` ] ( https://github.com/gzuidhof/coi-serviceworker ) first. It
79+ registers a service worker that adds the two headers and reloads the page once.
80+ After that the page is cross origin isolated, and the loader selects the build
81+ with all of the threads. This also operates on localhost, thus a usual static
82+ server is sufficient for development. ` make serve ` sets the headers too , which
83+ makes no difference but does no damage .
7784
78- Cross- origin isolation does mean a model has to come from a host that sends CORS
79- headers. Hugging Face does .
85+ Cross origin isolation makes it necessary for the model to come from a host that
86+ sends CORS headers. Hugging Face sends them .
8087
81- The line at the top right of the page says which build won. Force one with
82- ` ?mode=cpu ` or ` ?mode=webgpu ` on the URL.
88+ The line at the top right of the page shows the selected build. To force a
89+ build, add ` ?mode=cpu ` or ` ?mode=webgpu ` to the URL.
8390
8491## Notes
8592
86- - The system prompt box tells the model how to answer. The page sends it with
87- each question, so a change applies to the next answer. An empty box gives the
88- default prompt back .
89- - The model has to be smaller than 2 GB. One JavaScript ArrayBuffer holds no
90- more, so anything larger has to be in GGUF splits.
91- - Use a model with a chat template. A base model has none, and the page says so,
92- but the answers wander .
93- - ` ChatApplyTemplate ` formats one message at a time, so ` prompt ` in ` main.go `
94- puts the conversation together one message after another. That is exactly
95- right for a chatml model such as Qwen. A model whose template puts something
96- once at the top of a conversation, such as Gemma folding the system message
97- into the first user turn, comes out slightly off .
98- - A discrete NVIDIA card does not give f16 shaders in Chrome, so such a machine
99- falls back to the CPU unless Chrome starts with
93+ - The system prompt box tells the model how to answer. The page sends the prompt
94+ with each question, thus a change applies to the next answer. An empty box
95+ gives the default prompt again .
96+ - The model must be smaller than 2 GB. One JavaScript ArrayBuffer holds no more,
97+ thus a larger model must be in GGUF splits.
98+ - Use a model with a chat template. A base model has no template. The page shows
99+ a message, but the answers are poor .
100+ - ` ChatApplyTemplate ` formats one message at a time. Thus ` prompt ` in ` main.go `
101+ puts the conversation together one message after the other. This is correct
102+ for a chatml model such as Qwen. It is not fully correct for a model whose
103+ template puts something one time at the top of a conversation, such as Gemma,
104+ which folds the system message into the first user turn .
105+ - A discrete NVIDIA card does not give f16 shaders in Chrome. Such a machine
106+ falls back to the CPU unless you start Chrome with
100107 ` --enable-dawn-features=vulkan_enable_f16_on_nvidia ` .
101108
102109## Deploying
103110
104- ` .github/workflows/pages.yml ` builds and deploys on every push to ` main ` . Set
105- ** Settings → Pages → Source** to ** GitHub Actions** once, and that is all.
111+ ` .github/workflows/pages.yml ` builds and deploys on each push to ` main ` . Set
112+ ** Settings → Pages → Source** to ** GitHub Actions** one time. That is all.
106113
107114## License
108115
109- Apache 2.0, the same as yzma. ` web/min.css `
110- ([ min] ( https://mincss.com ) ) and ` web/coi-serviceworker.js ` are MIT, and keep
111- their own notices.
116+ Apache 2.0, the same as yzma. ` web/min.css ` ([ min] ( https://mincss.com ) ) and
117+ ` web/coi-serviceworker.js ` are MIT, and they keep their own notices.
0 commit comments