Does it support a TPU?

[fwartner]

I got no real GPU on my homelab so i would be interested if there's support for TPU?

[Van-QA]

hi @fwartner,
Unfortunately, we don't support TPU atm due to a lack of hardware for testing purposes.

@hiro-v would it be possible for us to support this user somehow? thank you

[Disane87]

I guess supporting a Google Coral would be pretty beneficial for all folks running this on a central homelab server, since they don’t have a dedicated gpu for these stuff.

additionally a coral is pretty affordable.

[fwartner]

I agree with @Disane87 (Who actually brought me to Jan ^^)..

If needed: I would love to sponsor a Coral TPU for testing purposes!

[mawoka-myblock]

As far as I am aware, the Coral TPU won't help here as it is only made for small models, but LLMs are large, as the name implies and by that too big for the ram on the TPU and streaming the model is also too slow.

[hiro-v]

Hi @mawoka-myblock and @Disane87

Currently we do not have plan for supporting TPU, as our current local inference engines are llama.cpp and nvidia tensorrt llm do not support those. We might problem support Python runtime and we can discover the possibility of running on TPU

There are several reference here:

• NVIDIA Tensorrt-llm of course only supports NVIDIA GPU
• For llama.cpp
  • https://github.com/ggerganov/llama.cpp/issues/1052#issuecomment-1515339426
  • https://github.com/ggerganov/llama.cpp/issues/3253

[dan-menlo]

Let's continue to keep this discussion thread open. I'm intrigued by the potential of Coral TPU.

• We are focused on refactoring our existing engines (e.g. TensorRT-LLM, OpenAI, etc) into an Inference Engine SDK
• This should make it possible for the community to build a Coral TPU Inference Engine pretty easily