Multi-backend refactor: Alpha release ( INTEL ONLY )

[Titus-von-Koeller]

This space is intended to receive feedback from users that are willing to help us by alpha testing the current implementation of the Intel backends (Intel CPU and XPU).

Issues to discuss could be

• bugs
• installation
• needed docs
• performance
• ease of use
• ... you name it ...

Thanks everyone for you kind support and please remember to have a constructive tone 🤗

[BBC-Esq]

I'd be interested in trying it out. How can I do that?

[epage480]

I'm interested as well!

[notsyncing]

Hello, I'm wondering is the intel backend ready to use? My attempts are all failed with

Unable to get JIT kernel for brgemm. 

for either load_in_4bit or load_in_8bit

or

NotImplementedError: Could not run 'torch_ipex::woq_linear.xpu' with arguments from the 'CPU' backend.

despite the model and inputs are all on the cpu.

on torch 2.5.1+cxx11.abi and intel_extension_for_pytorch 2.5.10+xpu and bitsandbytes 0.45.0.dev0 from https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_multi-backend-refactor/bitsandbytes-0.45.0.dev0-py3-none-manylinux_2_24_x86_64.whl

[notsyncing]

I found a similar error when googling: https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-AWQ/discussions/2, which is also trying to run a model on cpu.
Looks like it's not a bitsandbytes issue now.

[jiqing-feng]

Sure, please let me know if you found it's a BNB issue, thanks

[notsyncing]

@jiqing-feng Hello, now I'm running that script above on an A770 16G (added a model = model.to("xpu")), it now outputs the following error:

Traceback (most recent call last):
  File "/var/home/pods/test/test.py", line 32, in <module>
    print(generator("def helloWorld() {"))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/transformers/pipelines/text_generation.py", line 272, in __call__
    return super().__call__(text_inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/transformers/pipelines/base.py", line 1302, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/transformers/pipelines/base.py", line 1309, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/transformers/pipelines/base.py", line 1209, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/transformers/pipelines/text_generation.py", line 370, in _forward
    generated_sequence = self.model.generate(input_ids=input_ids, attention_mask=attention_mask, **generate_kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2215, in generate
    result = self._sample(
             ^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 3206, in _sample
    outputs = self(**model_inputs, return_dict=True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 1164, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 895, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 623, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 502, in forward
    key_states = self.k_proj(hidden_states)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/bitsandbytes/nn/modules.py", line 518, in forward
    out = bnb.matmul_4bit(x, weight, bias=bias, quant_state=self.weight.quant_state)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/pods/test/venv/lib/python3.12/site-packages/bitsandbytes/autograd/_functions.py", line 593, in matmul_4bit
    out += bias
RuntimeError: The size of tensor a (1536) must match the size of tensor b (256) at non-singleton dimension 2

I guess this could be a BNB issue. Could you take a look? Thanks!

[jiqing-feng]

Currently the XPU requires our internal package and docker, I suppose your error will be fixed after torch2.6 release.

[notsyncing]

Currently the XPU requires our internal package and docker, I suppose your error will be fixed after torch2.6 release.

okay, thanks. I'll wait for 2.6 release.

[steve-marmalade]

The following code crashes with RuntimeError: The size of tensor a (4320) must match the size of tensor b (4304) at non-singleton dimension 2 when bnb_4bit_quant_type="nf4" but not when that kwarg is removed or set to bnb_4bit_quant_type="fp4"

import torch
from transformers import AutoTokenizer, BitsAndBytesConfig, SiglipTextModel

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
)
model = SiglipTextModel.from_pretrained(
    "google/siglip-so400m-patch14-384",
    quantization_config=quantization_config,
)
tokenizer = AutoTokenizer.from_pretrained(
    "google/siglip-so400m-patch14-384"
)

with torch.no_grad():
    text = tokenizer(inputs, padding="max_length", return_tensors="pt")
    text_features = model(**text).pooler_output.tolist()
return text_features

Full stack trace:

File "/pkg/modal/_runtime/container_io_manager.py", line 741, in handle_input_exception
    yield
  File "/pkg/modal/_container_entrypoint.py", line 226, in run_input_async
    value = await res
            ^^^^^^^^^
  File "/root/aws-vit-so400m-14-siglip-384-328-cpu.py", line 99, in remote_generate
    text_features = self.model(**text).pooler_output.cpu().tolist()
                    ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/models/siglip/modeling_siglip.py", line 1046, in forward
    return self.text_model(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/models/siglip/modeling_siglip.py", line 970, in forward
    encoder_outputs = self.encoder(
                      ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/models/siglip/modeling_siglip.py", line 902, in forward
    layer_outputs = encoder_layer(
                    ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/models/siglip/modeling_siglip.py", line 652, in forward
    hidden_states = self.mlp(hidden_states)
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/models/siglip/modeling_siglip.py", line 608, in forward
    hidden_states = self.fc1(hidden_states)
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/bitsandbytes/nn/modules.py", line 512, in forward
    out = bnb.matmul_4bit(x, weight, bias=bias, quant_state=self.weight.quant_state)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/bitsandbytes/autograd/_functions.py", line 585, in matmul_4bit
    out += bias
RuntimeError: The size of tensor a (4320) must match the size of tensor b (4304) at non-singleton dimension 2

[steve-marmalade]

For reference, details about the environment:

Base image: python:3.11-slim

Python dependencies:

pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cpu

pip install huggingface-hub[hf-transfer]==0.28.0 transformers==4.48.1 sentencepiece==0.2.0 accelerate==1.3.0 optimum==1.23.3

pip install https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_multi-backend-refactor/bitsandbytes-0.44.1.dev0-py3-none-manylinux_2_24_x86_64.whl

pip install intel-extension-for-pytorch==2.5.0

pip install oneccl_bind_pt==2.5.0 --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/cpu/us/

[steve-marmalade]

FWIW I also tried base image python:3.12.8-slim-bookworm and building bitsandbytes from source, but got the same error. Stack trace is the same, except for some line number changes in bitsandbytes. Including here for completeness:

File "/pkg/modal/_runtime/container_io_manager.py", line 741, in handle_input_exception
    yield
  File "/pkg/modal/_container_entrypoint.py", line 226, in run_input_async
    value = await res
            ^^^^^^^^^
  File "/root/aws-vit-so400m-14-siglip-384-328-cpu.py", line 104, in remote_generate
    text_features = self.model(**text).pooler_output.cpu().tolist()
                    ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/transformers/models/siglip/modeling_siglip.py", line 1046, in forward
    return self.text_model(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/transformers/models/siglip/modeling_siglip.py", line 970, in forward
    encoder_outputs = self.encoder(
                      ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/transformers/models/siglip/modeling_siglip.py", line 902, in forward
    layer_outputs = encoder_layer(
                    ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/transformers/models/siglip/modeling_siglip.py", line 652, in forward
    hidden_states = self.mlp(hidden_states)
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/transformers/models/siglip/modeling_siglip.py", line 608, in forward
    hidden_states = self.fc1(hidden_states)
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/bitsandbytes/nn/modules.py", line 519, in forward
    out = bnb.matmul_4bit(x, weight, bias=bias, quant_state=self.weight.quant_state)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/bitsandbytes/autograd/_functions.py", line 618, in matmul_4bit
    out += bias
RuntimeError: The size of tensor a (4320) must match the size of tensor b (4304) at non-singleton dimension 2

[jiqing-feng]

Thanks for your report, I have reproduced it and found it's an ipex op bug. We are looking at it.

[jiqing-feng]

This is exactly an ipex bug, we have fixed it but it needs time to release the new version so you can use it. Will let you know when it's ready. Thanks again!

[BBC-Esq]

What model are you trying to run?

Also, does it happen if you try "fp4"?

[steve-marmalade]

What model are you trying to run?

In the code snippet I specify google/siglip-so400m-patch14-384, what other details can I provide?

Also, does it happen if you try "fp4"?

No, it doesn't crash on fp4 (this is the default value). I switched to nf4 because of the warning /usr/local/lib/python3.11/site-packages/bitsandbytes/backends/cpu_xpu_common.py:283: UserWarning: fp4 quantization is currently slow on CPU/XPU. Please Use nf4 instead for better performance. and the Intel CPU benchmarks showing that it's much faster.

[rskasturi]

Hi I've been testing it on Intel GPU Intel(R) Data Center GPU Max 1100) and had no issues setting up from the instructions given here

python version: 3.10
torch version: 2.6.0+xpu
bnb version: bitsandbytes-0.44.1.dev0

Model: DeepSeek-R1-Distill-Qwen-1.5B
Code: snippet

bnb_config = BitsAndBytesConfig(
    load_in_8bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

Observation 1: load_in_8bit everything runs out of the box, but it's 9x slower when compared without quantization, am I missing anything here?

Observation 2: with load_in_4bit throws an error:

ImportError: cannot import name 'ipex_cpu_only' from 'bitsandbytes.backends.cpu_xpu_common' (/home/user/.conda/envs/pt2.6/lib/python3.10/site-packages/bitsandbytes/backends/cpu_xpu_common.py)

is this error expected with load_in_4bit?

[jiqing-feng]

For Ob1, we will not treat it as an issue if you run the same code on cuda and find the same performance.

For Ob2, did you install intel_extension_for_pytorch?

[jiqing-feng]

I will check this issue as soon as I get back to work. We indeed need an alpha release for INTEL-ONLY.

[Xiangyi1996]

Hi, I am running Openvla quantization on Intel Arc 770. While runing on

   quantization_config = BitsAndBytesConfig(
        load_in_8bit=True,
        bnb_4bit_quant_type="nf4",
    )
    vla = AutoModelForVision2Seq.from_pretrained(
       'openvla/openvla-7b,
        torch_dtype=torch.bfloat16,
        low_cpu_mem_usage=True,
        trust_remote_code=True,
        quantization_config=quantization_config,
    ).to('xpu') 

The error report
[1] 849548 bus error (core dumped) python test/test_quantize.py

Running figure:

Reproduce

Environment

Name                         Version              Build                  Channel                                      
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  Deprecated                   1.2.18               pypi_0                 pypi                                         
  _libgcc_mutex                0.1                  conda_forge            conda-forge                                  
  _openmp_mutex                4.5                  2_kmp_llvm             conda-forge                                  
  annotated-types              0.7.0                pyhd8ed1ab_1           conda-forge                                  
  aom                          3.9.1                hac33072_0             conda-forge                                  
  beautifulsoup4               4.13.3               pyha770c72_0           conda-forge                                  
  bitsandbytes                 0.44.1.dev0+9315692  pypi_0                 pypi                                         
  brotli-python                1.1.0                py310hf71b8c6_2        conda-forge                                  
  bzip2                        1.0.8                h4bc722e_7             conda-forge                                  
  ca-certificates              2025.1.31            hbcca054_0             conda-forge                                  
  certifi                      2025.1.31            pyhd8ed1ab_0           conda-forge                                  
  cffi                         1.17.1               py310h8deb56e_0        conda-forge                                  
  charset-normalizer           3.4.1                pyhd8ed1ab_0           conda-forge                                  
  colorama                     0.4.6                pyhd8ed1ab_1           conda-forge                                  
  compiler_shared              2025.0.4             intel_1519             https://software.repos.intel.com/python/conda
  cpython                      3.10.16              py310hd8ed1ab_1        conda-forge                                  
  dav1d                        1.2.1                hd590300_0             conda-forge                                  
  dpcpp-cpp-rt                 2025.0.4             intel_1519             https://software.repos.intel.com/python/conda
  dpcpp_cpp_rt                 2025.0.4             intel_1519             https://software.repos.intel.com/python/conda
  dpcpp_impl_linux-64          2025.0.4             intel_1519             https://software.repos.intel.com/python/conda
  dpctl                        0.18.3               py310ha998128_0        https://software.repos.intel.com/python/conda
  edgeml                       0.2.1                pypi_0                 pypi                                         
  filelock                     3.17.0               pyhd8ed1ab_0           conda-forge                                  
  freetype                     2.12.1               h267a509_2             conda-forge                                  
  fsspec                       2025.2.0             pyhd8ed1ab_0           conda-forge                                  
  funcsigs                     1.0.2                pypi_0                 pypi                                         
  gdown                        5.2.0                pyhd8ed1ab_1           conda-forge                                  
  giflib                       5.2.2                hd590300_0             conda-forge                                  
  gmp                          6.3.0                hac33072_2             conda-forge                                  
  gmpy2                        2.1.5                py310he8512ff_3        conda-forge                                  
  h2                           4.2.0                pyhd8ed1ab_0           conda-forge                                  
  hpack                        4.1.0                pyhd8ed1ab_0           conda-forge                                  
  hyperframe                   6.1.0                pyhd8ed1ab_0           conda-forge                                  
  idna                         3.10                 pyhd8ed1ab_1           conda-forge                                  
  imageio                      2.37.0               pypi_0                 pypi                                         
  impi-devel                   2021.14.0            intel_790              https://software.repos.intel.com/python/conda
  impi_rt                      2021.14.0            intel_790              https://software.repos.intel.com/python/conda
  intel-cmplr-lib-rt           2025.0.4             intel_1519             https://software.repos.intel.com/python/conda
  intel-cmplr-lib-ur           2025.0.4             intel_1519             https://software.repos.intel.com/python/conda
  intel-cmplr-lic-rt           2025.0.4             intel_1519             https://software.repos.intel.com/python/conda
  intel-extension-for-pytorch  2.5.10               py310_xpu_0            https://software.repos.intel.com/python/conda
  intel-opencl-rt              2025.0.4             intel_1519             https://software.repos.intel.com/python/conda
  intel-openmp                 2025.0.4             intel_1519             https://software.repos.intel.com/python/conda
  intel-pti                    0.10.0               intel_284              https://software.repos.intel.com/python/conda
  intel-sycl-rt                2025.0.4             intel_1519             https://software.repos.intel.com/python/conda
  ipdb                         0.13.13              pypi_0                 pypi                                         
  jinja2                       3.1.5                pyhd8ed1ab_0           conda-forge                                  
  lcms2                        2.17                 h717163a_0             conda-forge                                  
  ld_impl_linux-64             2.43                 h712a8e2_2             conda-forge                                  
  lerc                         4.0.0                h27087fc_0             conda-forge                                  
  libabseil                    20240722.0           cxx17_hbbce691_4       conda-forge                                  
  libavif16                    1.1.1                h1909e37_2             conda-forge                                  
  libblas                      3.9.0                28_h59b9bed_openblas   conda-forge                                  
  libcblas                     3.9.0                28_he106b2a_openblas   conda-forge                                  
  libde265                     1.0.15               h00ab1b0_0             conda-forge                                  
  libdeflate                   1.23                 h4ddbbb0_0             conda-forge                                  
  libffi                       3.4.2                h7f98852_5             conda-forge                                  
  libgcc                       14.2.0               h77fa898_1             conda-forge                                  
  libgcc-ng                    14.2.0               h69a702a_1             conda-forge                                  
  libgfortran                  14.2.0               h69a702a_1             conda-forge                                  
  libgfortran5                 14.2.0               hd5240d6_1             conda-forge                                  
  libgomp                      14.2.0               h77fa898_1             conda-forge                                  
  libheif                      1.19.5               gpl_hc21c24c_100       conda-forge                                  
  libhwloc                     2.11.2               default_h0d58e46_1001  conda-forge                                  
  libiconv                     1.17                 hd590300_2             conda-forge                                  
  libjpeg-turbo                3.0.0                hd590300_1             conda-forge                                  
  liblapack                    3.9.0                28_h7ac8fdf_openblas   conda-forge                                  
  liblzma                      5.6.4                hb9d3cd8_0             conda-forge                                  
  libnsl                       2.0.1                hd590300_0             conda-forge                                  
  libopenblas                  0.3.28               pthreads_h94d23a6_1    conda-forge                                  
  libpng                       1.6.46               h943b412_0             conda-forge                                  
  libprotobuf                  5.28.3               h6128344_1             conda-forge                                  
  libsqlite                    3.48.0               hee588c1_1             conda-forge                                  
  libstdcxx                    14.2.0               hc0a3c3a_1             conda-forge                                  
  libstdcxx-ng                 14.2.0               h4852527_1             conda-forge                                  
  libtiff                      4.7.0                hd9ff511_3             conda-forge                                  
  libuuid                      2.38.1               h0b41bf4_0             conda-forge                                  
  libuv                        1.50.0               hb9d3cd8_0             conda-forge                                  
  libwebp-base                 1.5.0                h851e524_0             conda-forge                                  
  libxcb                       1.17.0               h8a09558_0             conda-forge                                  
  libxcrypt                    4.4.36               hd590300_1             conda-forge                                  
  libxml2                      2.13.5               h0d44e9d_1             conda-forge                                  
  libzlib                      1.3.1                hb9d3cd8_2             conda-forge                                  
  llvm-openmp                  19.1.7               h024ca30_0             conda-forge                                  
  markupsafe                   3.0.2                py310h89163eb_1        conda-forge                                  
  mkl                          2025.0.1             intel_14               https://software.repos.intel.com/python/conda
  mkl-dpcpp                    2025.0.1             intel_14               https://software.repos.intel.com/python/conda
  mkl-include                  2025.0.1             intel_14               https://software.repos.intel.com/python/conda
  mkl-static                   2025.0.1             intel_14               https://software.repos.intel.com/python/conda
  mpc                          1.3.1                h24ddda3_1             conda-forge                                  
  mpfr                         4.2.1                h90cbb55_3             conda-forge                                  
  mpmath                       1.3.0                pyhd8ed1ab_1           conda-forge                                  
  ncurses                      6.5                  h2d0b736_3             conda-forge                                  
  networkx                     3.4.2                pyh267e887_2           conda-forge                                  
  numpy                        2.2.3                py310hefbff90_0        conda-forge                                  
  oneccl                       2021.14.1            intel_2                https://software.repos.intel.com/python/conda
  oneccl-bind-pt               2.5.0+xpu            pypi_0                 pypi                                         
  oneccl-devel                 2021.14.1            intel_2                https://software.repos.intel.com/python/conda
  onemkl-sycl-blas             2025.0.1             intel_14               https://software.repos.intel.com/python/conda
  onemkl-sycl-datafitting      2025.0.1             intel_14               https://software.repos.intel.com/python/conda
  onemkl-sycl-dft              2025.0.1             intel_14               https://software.repos.intel.com/python/conda
  onemkl-sycl-lapack           2025.0.1             intel_14               https://software.repos.intel.com/python/conda
  onemkl-sycl-rng              2025.0.1             intel_14               https://software.repos.intel.com/python/conda
  onemkl-sycl-sparse           2025.0.1             intel_14               https://software.repos.intel.com/python/conda
  onemkl-sycl-stats            2025.0.1             intel_14               https://software.repos.intel.com/python/conda
  onemkl-sycl-vm               2025.0.1             intel_14               https://software.repos.intel.com/python/conda
  opencv-python                4.11.0.86            pypi_0                 pypi                                         
  opencv-python-headless       4.11.0.86            pypi_0                 pypi                                         
  openjpeg                     2.5.3                h5fbd93e_0             conda-forge                                  
  openpyxl                     3.1.5                pypi_0                 pypi                                         
  openssl                      3.4.0                h7b32b05_1             conda-forge                                  
  openvla                      0.0.3                pypi_0                 pypi                                         
  opt-einsum                   3.4.0                hd8ed1ab_1             conda-forge                                  
  opt_einsum                   3.4.0                pyhd8ed1ab_1           conda-forge                                  
  optree                       0.14.0               py310h3788b33_1        conda-forge                                  
  packaging                    24.2                 pyhd8ed1ab_2           conda-forge                                  
  pandas                       2.2.3                pypi_0                 pypi                                         
  pillow                       11.1.0               py310h7e6dc6c_0        conda-forge                                  
  pip                          25.0                 pyh8b19718_0           conda-forge                                  
  psutil                       6.1.1                py310ha75aee5_0        conda-forge                                  
  pthread-stubs                0.4                  hb9d3cd8_1002          conda-forge                                  
  pybind11                     2.13.6               pyh1ec8472_2           conda-forge                                  
  pybind11-global              2.13.6               pyh415d2e4_2           conda-forge                                  
  pycocotools                  2.0.8                pypi_0                 pypi                                         
  pycparser                    2.22                 pyh29332c3_1           conda-forge                                  
  pydantic                     2.10.6               pyh3cfb1c2_0           conda-forge                                  
  pydantic-core                2.27.2               py310h505e2c1_0        conda-forge                                  
  pysocks                      1.7.1                pyha55dd90_7           conda-forge                                  
  python                       3.10.16              he725a3c_1_cpython     conda-forge                                  
  python_abi                   3.10                 5_cp310                conda-forge                                  
  pytorch                      2.5.1                py310_xpu_0            https://software.repos.intel.com/python/conda
  rav1e                        0.6.6                he8a937b_2             conda-forge                                  
  readline                     8.2                  h8228510_1             conda-forge                                  
  requests                     2.32.3               pyhd8ed1ab_1           conda-forge                                  
  ruamel.yaml                  0.18.10              py310ha75aee5_0        conda-forge                                  
  ruamel.yaml.clib             0.2.8                py310ha75aee5_1        conda-forge                                  
  schema                       0.7.7                pypi_0                 pypi                                         
  scikit-learn                 1.6.1                pypi_0                 pypi                                         
  scipy                        1.15.1               py310hfa6ec8c_0        conda-forge                                  
  setuptools                   75.8.0               pyhff2d567_0           conda-forge                                  
  sleef                        3.8                  h1b44611_0             conda-forge                                  
  soupsieve                    2.5                  pyhd8ed1ab_1           conda-forge                                  
  svt-av1                      2.3.0                h5888daf_0             conda-forge                                  
  sympy                        1.13.1               pypyh2585a3b_103       conda-forge                                  
  tbb                          2022.0.0             intel_402              https://software.repos.intel.com/python/conda
  tbb-devel                    2022.0.0             intel_402              https://software.repos.intel.com/python/conda
  tcm                          1.2.0                intel_589              https://software.repos.intel.com/python/conda
  tk                           8.6.13               noxft_h4845f30_101     conda-forge                                  
  torch                        2.5.1+cxx11.abi      pypi_0                 pypi                                         
  torchaudio                   2.5.1+cxx11.abi      pypi_0                 pypi                                         
  torchvision                  0.20.1               py310_xpu_0            https://software.repos.intel.com/python/conda
  tqdm                         4.67.1               pyhd8ed1ab_1           conda-forge                                  
  transformers                 4.48.3               pypi_0                 pypi                                         
  typing-extensions            4.12.2               hd8ed1ab_1             conda-forge                                  
  typing_extensions            4.12.2               pyha770c72_1           conda-forge                                  
  tzdata                       2025a                h78e105d_0             conda-forge                                  
  umf                          0.9.1                intel_6                https://software.repos.intel.com/python/conda
  urllib3                      2.3.0                pyhd8ed1ab_0           conda-forge                                  
  wheel                        0.45.1               pyhd8ed1ab_1           conda-forge                                  
  x265                         3.5                  h924138e_3             conda-forge                                  
  xorg-libxau                  1.0.12               hb9d3cd8_0             conda-forge                                  
  xorg-libxdmcp                1.1.5                hb9d3cd8_0             conda-forge                                  
  zstandard                    0.23.0               py310ha39cb0e_1        conda-forge                                  
  zstd                         1.5.6                ha6fb4c9_0             conda-forge 

Reproduce quick scripts

import os
import time
import requests
from PIL import Image
import numpy as np
import torch
from transformers import AutoModelForVision2Seq, AutoProcessor, AutoConfig, AutoImageProcessor, AutoModel
import intel_extension_for_pytorch as ipex
from torch.profiler import profile, record_function, ProfilerActivity
from transformers import BitsAndBytesConfig

# Set environment variables
def set_environment_variables():
    os.environ['TRANSFORMERS_VERBOSITY'] = 'error'
    os.environ['HF_HUB_DISABLE_SSL_VERIFY'] = '1'

def check_model_dtype(model):
    for name, param in model.named_parameters():
        print(f"Parameter: {name}, dtype: {param.dtype}")  

def main():
    set_environment_variables()

    local_model_path = "./openvla/openvla-7b"
    device = 'xpu'
    num_iterations = 10
    quan = False  # 'bf16' or 'int4' 'int8'
    image_num = 1
    save_excel = False
    dtype = torch.bfloat16
    
    # Load Processor & VLA from local path
    processor = AutoProcessor.from_pretrained(local_model_path, trust_remote_code=True, local_files_only=True)

    quantization_config = BitsAndBytesConfig(
        load_in_8bit=True,
        bnb_4bit_quant_type="nf4",
    )
    
    vla = AutoModelForVision2Seq.from_pretrained(
        local_model_path,
        torch_dtype=dtype,
        low_cpu_mem_usage=True,
        trust_remote_code=True,
        local_files_only=True,
        quantization_config=quantization_config,
    ).to(device)

    check_model_dtype(vla)
        
    SYSTEM_PROMPT = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions."
    instruction = 'pick up cup'
    prompt = f"{SYSTEM_PROMPT} USER: What action should the robot take to {instruction.lower()}?"

    for image_id in range(image_num):
        image_url = f"https://rail.eecs.berkeley.edu/datasets/bridge_release/raw/bridge_data_v2/datacol2_toykitchen7/drawer_pnp/01/2023-04-19_09-18-15/raw/traj_group0/traj0/images0/im_{image_id}.jpg"
        image = Image.open(requests.get(image_url, stream=True).raw).resize((256, 256))

        total_time = 0
        for i in range(num_iterations):
            inputs = processor(prompt, image).to(device, dtype=dtype)
            start_time = time.time()
            action = vla.predict_action(**inputs, unnorm_key="bridge_orig", do_sample=False)
            end_time = time.time()
            elapsed_time = end_time - start_time
            total_time += elapsed_time
            print(f"step {i}:{elapsed_time}s")

        average_time = total_time / num_iterations
        print(f"im_{image_id}.jpg | Final action: {action} | Mean time: {average_time:.2f}s")
if __name__ == "__main__":
    main()

Any comments on Bus Error?
Thanks!

[jiqing-feng]

Hi @Xiangyi1996 , The error comes from model loading states, but I ran this model on my XPU device (Intel(R) Data Center GPU Max 1550) and it works well.
|

Can I get login to your node to debug it? You can send your device info to my email: [email protected]. Thanks!