webgpu not used in webgpu-chat

[CharlesJu1]

System Info

Macos 15.3.2, with Intel UHD Graphics 630 1536 MB
chrome v135.

Environment/Platform

• Website/web-app
• Browser extension
• Server-side (e.g., Node.js, Deno, Bun)
• Desktop app (e.g., Electron)
• Other (e.g., VSCode extension)

Description

First I changed @xenova/transformers to @huggingface/transformers in worker.js file since @xenova/transformers gives an error complaining entry point not found.

After npm install, I run the program with npm run dev. When I open the browser at port 5173, I see https://cdn.jsdelivr.net/npm/@huggingface/transformers@3.5.0/dist/ort-wasm-simd-threaded.jsep.mjs and https://cdn.jsdelivr.net/npm/@huggingface/transformers@3.5.0/dist/ort-wasm-simd-threaded.jsep.wasm are downloaded. But there is not webgpu related files downloaded.

Does it mean webgpu is not used somehow?

I have turned on chrome #enable-unsafe-webgpu. And the version of chrome is 135.

Reproduction

source code: https://github.com/huggingface/transformers.js/tree/main/examples/webgpu-chat
replace @xenova/transformers with @huggingface/transformers in worker.js
npm install
npm run dev

[xenova]

After npm install, I run the program with npm run dev. When I open the browser at port 5173, I see https://cdn.jsdelivr.net/npm/@huggingface/transformers@3.5.0/dist/ort-wasm-simd-threaded.jsep.mjs and https://cdn.jsdelivr.net/npm/@huggingface/transformers@3.5.0/dist/ort-wasm-simd-threaded.jsep.wasm are downloaded. But there is not webgpu related files downloaded.

These are the correct files! 👍

You can test if this is working correctly in your browser with our demo app: https://huggingface.co/spaces/webml-community/phi-3.5-webgpu

Also, note that we have moved the example source code to https://github.com/huggingface/transformers.js-examples/tree/main/phi-3.5-webgpu. We will be removing the examples from the main repo soon.

[CharlesJu1]

Does webgpu also use ort-wasm-simd-threaded.jsep.wasm? Where to check that GPU is indeed used in inference instead of CPU?

[xenova]

Does webgpu also use ort-wasm-simd-threaded.jsep.wasm?

Yes, that is correct as of the latest version of Transformers.js (v3.5). You can perhaps check your system's task manager / activity monitor to confirm that it is indeed using the GPU.

[CharlesJu1]

I am using "@huggingface/transformers": "^3.5.0", and here is for the pipeline setup:

    if (!translator) {
      try {
        console.debug('Initializing translator...');
        initStartTime = performance.now();
        translator = await pipeline('translation', 'Xenova/nllb-200-distilled-600M', {
          quantized: true,
          progress_callback: (progress) => {
            console.debug('Loading progress:', progress);
          },
          dtype: 'q8',  // Add explicit dtype configuration
          cache_dir: null,
          local_files_only: false,
          revision: 'main',
          device: 'webgpu'
        });
        initEndTime = performance.now();
        initDuration = initEndTime - initStartTime;
        console.debug('Translator initialized successfully');
      } catch (initError) {
        console.error('Failed to initialize translator:', initError);
        throw initError;
      }
    }

without device: 'webgpu', it works fine, taking about 3 seconds to translate a word like "hello" to Chinese.

With device: 'webgpu', it does not work. It hangs in the await pipeline() call. The devtools console gives an error:

2025-04-27 16:37:59.828499 [W:onnxruntime:, session_state.cc:1280 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
Bc	@	71aed1cb-946e-4fea-b46b-9bcb2d01cbc7:100
$func11828	@	ort-wasm-simd-thread….jsep.wasm:0xf9915f
$func1841	@	ort-wasm-simd-thread….jsep.wasm:0x22df9c
$func810	@	ort-wasm-simd-threaded.jsep.wasm:0xb9f34
$func11762	@	ort-wasm-simd-thread….jsep.wasm:0xf92915
$func697	@	ort-wasm-simd-threaded.jsep.wasm:0x9f9bb
$func88	@	ort-wasm-simd-threaded.jsep.wasm:0xdb66
$func14346	@	ort-wasm-simd-thread…jsep.wasm:0x12d4ecb
$func119	@	ort-wasm-simd-threaded.jsep.wasm:0x15503
$func1384	@	ort-wasm-simd-thread….jsep.wasm:0x177a32
$func11276	@	ort-wasm-simd-thread….jsep.wasm:0xf4d91f
$Ma	@	ort-wasm-simd-thread….jsep.wasm:0xae2f36
Hd.b.<computed>	@	71aed1cb-946e-4fea-b46b-9bcb2d01cbc7:77
e._OrtCreateSession	@	71aed1cb-946e-4fea-b46b-9bcb2d01cbc7:104
(anonymous)	@	71aed1cb-946e-4fea-b46b-9bcb2d01cbc7:14
xr	@	wasm-core-impl.ts:366
await in xr		
uc	@	proxy-wrapper.ts:206
loadModel	@	session-handler-inference.ts:87
createInferenceSessionHandler	@	backend-wasm.ts:91
create	@	inference-session-impl.ts:209
await in create		
createInferenceSession	@	onnx.js:156
(anonymous)	@	models.js:351
await in (anonymous)		
constructSessions	@	models.js:349
from_pretrained	@	models.js:1143
await in from_pretrained		
from_pretrained	@	models.js:7483
await in from_pretrained		
loadItems	@	pipelines.js:3514
pipeline	@	pipelines.js:3452
translate	@	App.vue:122
callWithErrorHandling	@	runtime-core.esm-bundler.js:199
callWithAsyncErrorHandling	@	runtime-core.esm-bundler.js:206
invoker	@	runtime-dom.esm-bundler.js:729