Other Non-Nvidia GPU

[jcubic]

Does this project work on anything other than NVidia?

I have AMD GPU and wanted to test, but the docker command use CUDA that is an NVidia technology.

When I run docker command, I've got this error:

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

[wsxiaoys]

Hi - for non-nvidia GPU, consider using tabby's vulkan backend (blog post: https://tabby.tabbyml.com/blog/2024/05/01/vulkan-support/)

[wsxiaoys]

The error shows that ~/.tabby is not accessible for your current user.

[jcubic]

On Linux, AFAIK docker run as root even when executed as normal user.

After changing the permission, it gives this error:

$ RUST_BACKTRACE=1 ./tabby_x86_64-manylinux2014-vulkan serve --model StarCoder-1B --device vulkan
thread 'main' panicked at /__w/tabby/tabby/crates/tabby-common/src/registry.rs:64:21:
Failed to fetch model organization <TabbyML>: Failed to get JSON

Caused by:
    0: error decoding response body: invalid type: map, expected a sequence at line 178 column 6
    1: invalid type: map, expected a sequence at line 178 column 6
stack backtrace:
   0: rust_begin_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   2: tabby_common::registry::ModelRegistry::new::{{closure}}
   3: tabby_download::download_model::{{closure}}
   4: tabby::services::model::download_model_if_needed::{{closure}}
   5: tabby::main::{{closure}}
   6: tokio::runtime::park::CachedParkThread::block_on
   7: tabby::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[zwpaper]

Hi @jcubic, can you also post which version you were using?

you can get it by

./tabby_x86_64-manylinux2014-vulkan --version

[jcubic]

The one from the article, tabby 0.10.0.

[zwpaper]

related: #3313