Open Discussion: What To Do With AMD?

[Jeffser]

Hello everyone I'm the developer of Alpaca, I've heard you all, AMD ROCm stopped working with Alpaca's Ollama instance in Flatpak.

Why is this happening?

To put it simple, when installing the ROCm extension Alpaca moves ROCm into /app/plugins/AMD/... inside the Flatpak, now, this was fine because there's an env variable that told the Ollama instance where to find the library.

Does this has to do with the instance manager?

Nope, just an unfortunate coincidence.

Ok so what's the problem then?

Well apparently there's either something hardcoded into Ollama that expects the Ollama libraries to be in the same directory as the ROCm stuff or I'm just stupid and can't figure this out.

Now then, why don't you just move the extension to the existing library directory?

Nope can't do, something actually really nice about Flatpak is that the extensions can't modify existing directories, I actually made a version of Alpaca internally that has both the Ollama instance and ROCm preinstalled.

So just preinstall ROCm

Some people don't even like Ollama being included, adding a couple of gigs to people that don't even use the instance or don't have an AMD card is not the best way of fixing this.

What do I want to do?

If it's up to me I would just stop including Ollama altogether, it's been a real pain in the ass to keep it updated and fixing stuff. To be fair Ollama is a really great piece of software that's excellently packaged, I just don't know what I'm doing most of the time.

Even though I don't believe in free software being a democracy I will make an exception here because this is a big change.

So, my lovely users, what do you guys want me to do about this?

Feel free to add other options, call me lazy or whatever, I love reading your comments!

[Thorazin]

let me see if I have this right, you are unable to point Ollama to ROCm because Ollama expects ROCm to be in the same directory?

[Jeffser]

Pretty much that's what I'm seeing. Might be wrong tho

[Thorazin]

Perhaps I am being naive here but that sounds like an error on the part of Ollama which should look for ROCM in the default location, unless it is a "feature" of the flatpak that everything be inside the flatpak sandbox?

[Sirius3331]

That's a difficult choice. GPUs from NVIDIA are too expensive today and AMD got really good with their 9000 series which I will buy later. So I have to maybe use ROCm or the AI Accelerator on the GPU can be used without it. Maybe you can make a inquiry by startup of Alpaca which Instance should be installed or should be used if there are multiple instances installed. It's a huge thing I know but every AMD GPU users will be forced to use a similar app if you stop implementing AMDs ROCm.

[DomiStyle]

The main reason why I use Alpaca is because of how easy it is to get going. Just install Alpaca and the extension and you are good to go with your AMD GPU.

So I think preserving ollama support as an extension is worth doing to keep things accessible for everyone.

[coolkys]

One question: alpaca's ability to communicate with and control an external ollama server(localhost or otherwise) won't be affected by the separation/removal of ollama from alpaca, right?

I was looking forward to the integration of the two projects, but if it is infeasible as you described, I think it is best to manage ollama separately, given that it is possible. I'd like to keep using my 7900XTX for local LLMs, and at the same time I don't want to go back to using the ollama CLI; Alpaca is so nice to use.
Therefore, I'd prefer it if you provided us with the option to install ollama as an extension, and even when/if you decide to discontinue the extension support, we can still use ollama externally as a server, with alpaca as the frontend.

[Jeffser]

The integrated instance and the (theoretical) Ollama extension work almost the same way as an external instance of Ollama in a Docker or system installation.

The difference is that external instances cannot be started or stopped from Alpaca, nor can Alpaca set env variables to them. But Alpaca could still download and create models the same way as it does now.

So, the extension could be started and stopped and it's env variables modified.
An external instance cannot do that, that's the main difference

[coolkys]

Ah, so that's why my separate ollama server doesn't get stopped by the stop button. I imagined that ollama provided a dead-stop API to make that possible, but now I understand that the stop button was for the embedded ollama instance. While that is indeed a nice-to-have, what's most important is making this project manageable for the author. Thanks for your work!

[Jeffser]

Well not exactly, the stop button is supposed to work with any instance, it just disconnects Alpaca from the request, the thing is, Ollama doesn't actually have a way of stopping a request, sometimes it stops sometimes it doesn't.

What I meant by stop is exiting the process

[Jeffser]

My Personal Opinion

I wanted to wait a couple of hours to get some votes before sharing my personal opinion, so here it is.

Alpaca started as an interface for Ollama, it actually didn't include the instance at first. Then it became an easy way of just installing something and chatting with bots directly.

It's been a fun year with a lot of changes for my little project and I think this could probably be a big change.

With the introduction of the instance manager users can now connect to Gemini, ChatGPT, etc etc. Making Alpaca more of an AI hub than just an Ollama client.

To reflect that I think the best approach would be to get rid of the Ollama instance... by default.

So, this is what I want to do:

Make a new extension called com.jeffser.Alpaca.Plugins.Ollama, this will include just Ollama, no ROCm, this would be the option for people who want the instance but have Nvidia GPUs, for example.

Then I would transform the existing com.jeffser.Alpaca.Plugins.AMD to be Ollama with ROCm, making it the option for users with AMD GPUs.

I'm not sure if I can make it so that Flathub detects conflicts if you try to install both extensions or something like that but I hope I can figure it out.

This approach would also make it possible for people who just want a client for an existing Ollama instance or Gemini to have a minimal app with no instance that they won't use, the same thing as the -no-ollama packages that are available in the releases page.

Of course, I would have to make it so that the welcome screen now has an explanation or at least a link to a wiki article in the repo or something.

[AlgorithmArtist]

I do think this is the best way to go, as it would preserve the ease of use (mostly) as well as satisfy the "everything is bloat crowd" / slim down the application :)

[Jeffser]

I GOT IT OMG

So get this, Ollama originally didn't include this library called Numa, so I added it manually to the flatpak, apparently it now includes it, having both just broke ROCm.

I went to bed at like 4 am and couldn't figure this out, THE SOLUTION CAME TO ME IN MY FUCKING DREAM.

I don't know if I should be impressed or scared.

Anyways, this is what I want to do now

com.jeffser.Alpaca.Plugins.Ollama > Just Ollama
com.jeffser.Alpaca.Plugins.AMD > Just ROCm

So if you have AMD you should install both extensions

[MysteriousGitEntity]

THE SOLUTION CAME TO ME IN MY FUCKING DREAM

Wtf... Congrats, and thank you for your dedication to this project! 😄 🎉

[Thorazin]

Ha! Awesome! Glad you figured it out.

[Jeffser]

It works

So I just built Alpaca with it's new manifest and both extensions, everything works, I'm so tired but overjoyed hahahaha

[Jeffser]

Update is Underway

I will take the weekend off but I wanted to push the update!

5.1.0 is coming to Flathub, it is the first version of Alpaca to not include Ollama ever since it was introduced, it has a little popup that will explain everything.

And yeah, the Ollama extension is already out, the nice people at Flathub were really fast and helped me polish it up!

I hope everything works because I won't be able to do any work this weekend, please report any issues and I will take a look at them sunday night!

Thanks again for all the patience!

[rudolf-enum]

It seems that the author has unfortunately given up on this, as per this instantly locked issue: "Stop Using AMD"

The AMD Ollama Docker image works well, for anyone looking to get back up and running again.

[Jeffser]

Is not that I've given up on this, it's just that I have 6 projects to attend to in college.

Trust me, I would love to be able to fix this but it really isn't easy to test.

[Jeffser]

I just released com.jeffser.Alpaca.Plugins.AMD version 0.5.13.1

Can someone test if it works?

My GPU is technically not supported so I can't be sure

[MysteriousGitEntity]

I only just received the update. It did not work:

INFO	[instance_manager.py | start] Starting Alpaca's Ollama instance...
INFO	[instance_manager.py | start] Started Alpaca's Ollama instance
Couldn't find '/home/user/.ollama/id_ed25519'. Generating new private key.
Your new public key is: 

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBbR2r4eW0NtqZ57Aj3Or9WtAB2nETzpFW1gPTgEBulR

2025/03/12 06:28:35 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11435 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2025-03-12T06:28:35.727Z level=INFO source=images.go:432 msg="total blobs: 22"
time=2025-03-12T06:28:35.727Z level=INFO source=images.go:439 msg="total unused blobs removed: 0"
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:	export GIN_MODE=release
 - using code:	gin.SetMode(gin.ReleaseMode)

[GIN-debug] POST   /api/pull                 --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers)
[GIN-debug] POST   /api/generate             --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers)
[GIN-debug] POST   /api/chat                 --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers)
[GIN-debug] POST   /api/embed                --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers)
[GIN-debug] POST   /api/embeddings           --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers)
[GIN-debug] POST   /api/create               --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers)
[GIN-debug] POST   /api/push                 --> github.com/ollama/ollama/server.(*Server).PushHandler-fm (5 handlers)
[GIN-debug] POST   /api/copy                 --> github.com/ollama/ollama/server.(*Server).CopyHandler-fm (5 handlers)
[GIN-debug] DELETE /api/delete               --> github.com/ollama/ollama/server.(*Server).DeleteHandler-fm (5 handlers)
[GIN-debug] POST   /api/show                 --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (5 handlers)
[GIN-debug] POST   /api/blobs/:digest        --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers)
[GIN-debug] HEAD   /api/blobs/:digest        --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers)
[GIN-debug] GET    /api/ps                   --> github.com/ollama/ollama/server.(*Server).PsHandler-fm (5 handlers)
[GIN-debug] POST   /v1/chat/completions      --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers)
[GIN-debug] POST   /v1/completions           --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers)
[GIN-debug] POST   /v1/embeddings            --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers)
[GIN-debug] GET    /v1/models                --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers)
[GIN-debug] GET    /v1/models/:model         --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers)
[GIN-debug] GET    /                         --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] GET    /api/tags                 --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
[GIN-debug] GET    /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
[GIN-debug] HEAD   /                         --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] HEAD   /api/tags                 --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
[GIN-debug] HEAD   /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
time=2025-03-12T06:28:35.728Z level=INFO source=routes.go:1238 msg="Listening on [::]:11435 (version 0.5.7)"
time=2025-03-12T06:28:35.728Z level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 cuda_v11_avx cuda_v12_avx rocm_avx]"
time=2025-03-12T06:28:35.728Z level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
INFO	[instance_manager.py | start] client version is 0.5.7
time=2025-03-12T06:28:35.732Z level=WARN source=amd_linux.go:61 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2025-03-12T06:28:35.733Z level=INFO source=amd_linux.go:388 msg="amdgpu is supported" gpu=0 gpu_type=gfx1102
time=2025-03-12T06:28:35.734Z level=INFO source=amd_linux.go:297 msg="unsupported Radeon iGPU detected skipping" id=1 total="512.0 MiB"
time=2025-03-12T06:28:35.734Z level=INFO source=types.go:131 msg="inference compute" id=0 library=rocm variant="" compute=gfx1102 driver=0.0 name=1002:7480 total="8.0 GiB" available="7.9 GiB"
[GIN] 2025/03/12 - 06:28:35 | 200 |     443.955µs |       127.0.0.1 | GET      "/api/tags"
[GIN] 2025/03/12 - 06:28:35 | 200 |    5.080388ms |       127.0.0.1 | POST     "/api/show"
[GIN] 2025/03/12 - 06:28:35 | 200 |   12.878988ms |       127.0.0.1 | POST     "/api/show"
[GIN] 2025/03/12 - 06:28:35 | 200 |   13.439249ms |       127.0.0.1 | POST     "/api/show"
[GIN] 2025/03/12 - 06:28:35 | 200 |   14.221161ms |       127.0.0.1 | POST     "/api/show"
time=2025-03-12T06:28:39.598Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 gpu=0 parallel=4 available=8441634816 required="6.5 GiB"
time=2025-03-12T06:28:39.598Z level=INFO source=server.go:104 msg="system memory" total="62.0 GiB" free="51.0 GiB" free_swap="4.0 GiB"
time=2025-03-12T06:28:39.599Z level=INFO source=memory.go:356 msg="offload to rocm" projector.weights="595.5 MiB" projector.graph="0 B" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[7.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.5 GiB" memory.required.partial="6.5 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.5 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.6 GiB" memory.weights.nonrepeating="102.6 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="585.0 MiB"
time=2025-03-12T06:28:39.600Z level=INFO source=server.go:376 msg="starting llama server" cmd="/app/plugins/Ollama/lib/ollama/runners/rocm_avx/ollama_llama_server runner --model /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --mmproj /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-72d6f08a42f656d36b356dbe0920675899a99ce21192fd66266fb7d82ed07539 --threads 8 --parallel 4 --port 38881"
time=2025-03-12T06:28:39.600Z level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-03-12T06:28:39.600Z level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
time=2025-03-12T06:28:39.600Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
time=2025-03-12T06:28:39.632Z level=INFO source=runner.go:936 msg="starting go runner"
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory

rocBLAS error: Could not initialize Tensile host: No devices found
time=2025-03-12T06:28:40.101Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error:Could not initialize Tensile host: No devices found"
time=2025-03-12T06:28:40.101Z level=WARN source=server.go:475 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)"
[GIN] 2025/03/12 - 06:28:40 | 500 |  514.421644ms |       127.0.0.1 | POST     "/v1/chat/completions"
INFO	[_client.py | _send_single_request] HTTP Request: POST http://0.0.0.0:11435/v1/chat/completions "HTTP/1.1 500 Internal Server Error"
INFO	[_base_client.py | _retry_request] Retrying request to /chat/completions in 0.470076 seconds
time=2025-03-12T06:28:45.102Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.000635924 model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868
time=2025-03-12T06:28:45.109Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 gpu=0 parallel=4 available=8441634816 required="6.5 GiB"
time=2025-03-12T06:28:45.109Z level=INFO source=server.go:104 msg="system memory" total="62.0 GiB" free="51.0 GiB" free_swap="4.0 GiB"
time=2025-03-12T06:28:45.110Z level=INFO source=memory.go:356 msg="offload to rocm" projector.weights="595.5 MiB" projector.graph="0 B" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[7.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.5 GiB" memory.required.partial="6.5 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.5 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.6 GiB" memory.weights.nonrepeating="102.6 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="585.0 MiB"
time=2025-03-12T06:28:45.110Z level=INFO source=server.go:376 msg="starting llama server" cmd="/app/plugins/Ollama/lib/ollama/runners/rocm_avx/ollama_llama_server runner --model /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --mmproj /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-72d6f08a42f656d36b356dbe0920675899a99ce21192fd66266fb7d82ed07539 --threads 8 --parallel 4 --port 40661"
time=2025-03-12T06:28:45.110Z level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-03-12T06:28:45.110Z level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
time=2025-03-12T06:28:45.111Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
time=2025-03-12T06:28:45.143Z level=INFO source=runner.go:936 msg="starting go runner"
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory

rocBLAS error: Could not initialize Tensile host: No devices found
time=2025-03-12T06:28:45.352Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.250239228 model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868
time=2025-03-12T06:28:45.611Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error:Could not initialize Tensile host: No devices found"
time=2025-03-12T06:28:45.611Z level=WARN source=server.go:475 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)"
time=2025-03-12T06:28:45.611Z level=WARN source=server.go:475 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)"
[GIN] 2025/03/12 - 06:28:45 | 500 |  6.024272209s |       127.0.0.1 | POST     "/v1/chat/completions"
INFO	[_client.py | _send_single_request] HTTP Request: POST http://0.0.0.0:11435/v1/chat/completions "HTTP/1.1 500 Internal Server Error"
INFO	[_base_client.py | _retry_request] Retrying request to /chat/completions in 0.447125 seconds
time=2025-03-12T06:28:50.612Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.000633711 model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868
time=2025-03-12T06:28:50.619Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 gpu=0 parallel=4 available=8441634816 required="6.5 GiB"
time=2025-03-12T06:28:50.619Z level=INFO source=server.go:104 msg="system memory" total="62.0 GiB" free="50.9 GiB" free_swap="4.0 GiB"
time=2025-03-12T06:28:50.619Z level=INFO source=memory.go:356 msg="offload to rocm" projector.weights="595.5 MiB" projector.graph="0 B" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[7.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.5 GiB" memory.required.partial="6.5 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.5 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.6 GiB" memory.weights.nonrepeating="102.6 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="585.0 MiB"
time=2025-03-12T06:28:50.621Z level=INFO source=server.go:376 msg="starting llama server" cmd="/app/plugins/Ollama/lib/ollama/runners/rocm_avx/ollama_llama_server runner --model /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --mmproj /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-72d6f08a42f656d36b356dbe0920675899a99ce21192fd66266fb7d82ed07539 --threads 8 --parallel 4 --port 41175"
time=2025-03-12T06:28:50.621Z level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-03-12T06:28:50.621Z level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
time=2025-03-12T06:28:50.622Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
time=2025-03-12T06:28:50.659Z level=INFO source=runner.go:936 msg="starting go runner"
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory

rocBLAS error: Could not initialize Tensile host: No devices found
time=2025-03-12T06:28:50.862Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.250934921 model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868
time=2025-03-12T06:28:51.123Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error:Could not initialize Tensile host: No devices found"
time=2025-03-12T06:28:51.123Z level=WARN source=server.go:475 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)"
time=2025-03-12T06:28:51.123Z level=WARN source=server.go:475 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)"
[GIN] 2025/03/12 - 06:28:51 | 500 | 10.550092654s |       127.0.0.1 | POST     "/v1/chat/completions"
INFO	[_client.py | _send_single_request] HTTP Request: POST http://0.0.0.0:11435/v1/chat/completions "HTTP/1.1 500 Internal Server Error"
INFO	[_base_client.py | _retry_request] Retrying request to /chat/completions in 0.873115 seconds
time=2025-03-12T06:28:56.124Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.000621959 model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868
time=2025-03-12T06:28:56.131Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 gpu=0 parallel=4 available=8441634816 required="6.5 GiB"
time=2025-03-12T06:28:56.131Z level=INFO source=server.go:104 msg="system memory" total="62.0 GiB" free="50.9 GiB" free_swap="4.0 GiB"
time=2025-03-12T06:28:56.132Z level=INFO source=memory.go:356 msg="offload to rocm" projector.weights="595.5 MiB" projector.graph="0 B" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[7.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.5 GiB" memory.required.partial="6.5 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.5 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.6 GiB" memory.weights.nonrepeating="102.6 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="585.0 MiB"
time=2025-03-12T06:28:56.133Z level=INFO source=server.go:376 msg="starting llama server" cmd="/app/plugins/Ollama/lib/ollama/runners/rocm_avx/ollama_llama_server runner --model /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --mmproj /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-72d6f08a42f656d36b356dbe0920675899a99ce21192fd66266fb7d82ed07539 --threads 8 --parallel 4 --port 40243"
time=2025-03-12T06:28:56.133Z level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-03-12T06:28:56.133Z level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
time=2025-03-12T06:28:56.133Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
time=2025-03-12T06:28:56.168Z level=INFO source=runner.go:936 msg="starting go runner"
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory

rocBLAS error: Could not initialize Tensile host: No devices found
time=2025-03-12T06:28:56.374Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.250377625 model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868
time=2025-03-12T06:28:56.634Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error:Could not initialize Tensile host: No devices found"
time=2025-03-12T06:28:56.635Z level=WARN source=server.go:475 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)"
time=2025-03-12T06:28:56.635Z level=WARN source=server.go:475 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)"
[GIN] 2025/03/12 - 06:28:56 | 500 | 10.573988038s |       127.0.0.1 | POST     "/v1/chat/completions"
time=2025-03-12T06:28:56.635Z level=WARN source=server.go:475 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)"
INFO	[_client.py | _send_single_request] HTTP Request: POST http://0.0.0.0:11435/v1/chat/completions "HTTP/1.1 500 Internal Server Error"
INFO	[_base_client.py | _retry_request] Retrying request to /chat/completions in 0.842430 seconds
time=2025-03-12T06:29:01.635Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.00041729 model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868
time=2025-03-12T06:29:01.643Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 gpu=0 parallel=4 available=8441634816 required="6.5 GiB"
time=2025-03-12T06:29:01.643Z level=INFO source=server.go:104 msg="system memory" total="62.0 GiB" free="50.9 GiB" free_swap="4.0 GiB"
time=2025-03-12T06:29:01.644Z level=INFO source=memory.go:356 msg="offload to rocm" projector.weights="595.5 MiB" projector.graph="0 B" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[7.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.5 GiB" memory.required.partial="6.5 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.5 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.6 GiB" memory.weights.nonrepeating="102.6 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="585.0 MiB"
time=2025-03-12T06:29:01.645Z level=INFO source=server.go:376 msg="starting llama server" cmd="/app/plugins/Ollama/lib/ollama/runners/rocm_avx/ollama_llama_server runner --model /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --mmproj /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-72d6f08a42f656d36b356dbe0920675899a99ce21192fd66266fb7d82ed07539 --threads 8 --parallel 4 --port 40633"
time=2025-03-12T06:29:01.645Z level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-03-12T06:29:01.645Z level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
time=2025-03-12T06:29:01.645Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
time=2025-03-12T06:29:01.680Z level=INFO source=runner.go:936 msg="starting go runner"
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory

rocBLAS error: Could not initialize Tensile host: No devices found
time=2025-03-12T06:29:01.885Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.250743787 model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868
time=2025-03-12T06:29:02.146Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error:Could not initialize Tensile host: No devices found"
time=2025-03-12T06:29:02.146Z level=WARN source=server.go:475 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)"
time=2025-03-12T06:29:02.146Z level=WARN source=server.go:475 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)"
[GIN] 2025/03/12 - 06:29:02 | 500 | 10.147842261s |       127.0.0.1 | POST     "/v1/chat/completions"
time=2025-03-12T06:29:02.146Z level=WARN source=server.go:475 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)"
INFO	[_client.py | _send_single_request] HTTP Request: POST http://0.0.0.0:11435/v1/chat/completions "HTTP/1.1 500 Internal Server Error"
time=2025-03-12T06:29:07.147Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.00119262 model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868
time=2025-03-12T06:29:07.154Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 gpu=0 parallel=4 available=8441634816 required="6.5 GiB"
time=2025-03-12T06:29:07.154Z level=INFO source=server.go:104 msg="system memory" total="62.0 GiB" free="51.0 GiB" free_swap="4.0 GiB"
time=2025-03-12T06:29:07.155Z level=INFO source=memory.go:356 msg="offload to rocm" projector.weights="595.5 MiB" projector.graph="0 B" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[7.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.5 GiB" memory.required.partial="6.5 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.5 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.6 GiB" memory.weights.nonrepeating="102.6 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="585.0 MiB"
time=2025-03-12T06:29:07.156Z level=INFO source=server.go:376 msg="starting llama server" cmd="/app/plugins/Ollama/lib/ollama/runners/rocm_avx/ollama_llama_server runner --model /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --mmproj /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-72d6f08a42f656d36b356dbe0920675899a99ce21192fd66266fb7d82ed07539 --threads 8 --parallel 4 --port 44213"
time=2025-03-12T06:29:07.156Z level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-03-12T06:29:07.156Z level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
time=2025-03-12T06:29:07.156Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
time=2025-03-12T06:29:07.189Z level=INFO source=runner.go:936 msg="starting go runner"
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory

rocBLAS error: Could not initialize Tensile host: No devices found
time=2025-03-12T06:29:07.396Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.250192853 model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868
time=2025-03-12T06:29:07.657Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error:Could not initialize Tensile host: No devices found"
[GIN] 2025/03/12 - 06:29:07 | 500 | 10.178411446s |       127.0.0.1 | POST     "/v1/chat/completions"
time=2025-03-12T06:29:07.657Z level=WARN source=server.go:475 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)"
INFO	[_client.py | _send_single_request] HTTP Request: POST http://0.0.0.0:11435/v1/chat/completions "HTTP/1.1 500 Internal Server Error"
time=2025-03-12T06:29:07.657Z level=WARN source=server.go:475 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)"
time=2025-03-12T06:29:07.657Z level=WARN source=server.go:475 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)"
time=2025-03-12T06:29:07.657Z level=WARN source=server.go:475 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)"
ERROR	[instance_manager.py | generate_message] Error code: 500 - {'error': {'message': 'llama runner process has terminated: error:Could not initialize Tensile host: No devices found', 'type': 'api_error', 'param': None, 'code': None}}
time=2025-03-12T06:29:12.659Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.001110377 model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868
time=2025-03-12T06:29:12.665Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 gpu=0 parallel=4 available=8441634816 required="6.5 GiB"
time=2025-03-12T06:29:12.665Z level=INFO source=server.go:104 msg="system memory" total="62.0 GiB" free="50.9 GiB" free_swap="4.0 GiB"
time=2025-03-12T06:29:12.666Z level=INFO source=memory.go:356 msg="offload to rocm" projector.weights="595.5 MiB" projector.graph="0 B" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[7.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.5 GiB" memory.required.partial="6.5 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.5 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.6 GiB" memory.weights.nonrepeating="102.6 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="585.0 MiB"
time=2025-03-12T06:29:12.667Z level=INFO source=server.go:376 msg="starting llama server" cmd="/app/plugins/Ollama/lib/ollama/runners/rocm_avx/ollama_llama_server runner --model /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --mmproj /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-72d6f08a42f656d36b356dbe0920675899a99ce21192fd66266fb7d82ed07539 --threads 8 --parallel 4 --port 46203"
time=2025-03-12T06:29:12.667Z level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-03-12T06:29:12.668Z level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
time=2025-03-12T06:29:12.668Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
time=2025-03-12T06:29:12.699Z level=INFO source=runner.go:936 msg="starting go runner"
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory

rocBLAS error: Could not initialize Tensile host: No devices found
time=2025-03-12T06:29:12.909Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.251164938 model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868
time=2025-03-12T06:29:13.169Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error:Could not initialize Tensile host: No devices found"
[GIN] 2025/03/12 - 06:29:13 | 500 | 11.019880164s |       127.0.0.1 | POST     "/v1/chat/completions"
INFO	[_client.py | _send_single_request] HTTP Request: POST http://0.0.0.0:11435/v1/chat/completions "HTTP/1.1 500 Internal Server Error"
INFO	[_base_client.py | _retry_request] Retrying request to /chat/completions in 0.406737 seconds
time=2025-03-12T06:29:13.588Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 gpu=0 parallel=4 available=8441634816 required="6.5 GiB"
time=2025-03-12T06:29:13.588Z level=INFO source=server.go:104 msg="system memory" total="62.0 GiB" free="50.9 GiB" free_swap="4.0 GiB"
time=2025-03-12T06:29:13.589Z level=INFO source=memory.go:356 msg="offload to rocm" projector.weights="595.5 MiB" projector.graph="0 B" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[7.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.5 GiB" memory.required.partial="6.5 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.5 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.6 GiB" memory.weights.nonrepeating="102.6 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="585.0 MiB"
time=2025-03-12T06:29:13.590Z level=INFO source=server.go:376 msg="starting llama server" cmd="/app/plugins/Ollama/lib/ollama/runners/rocm_avx/ollama_llama_server runner --model /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --mmproj /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-72d6f08a42f656d36b356dbe0920675899a99ce21192fd66266fb7d82ed07539 --threads 8 --parallel 4 --port 44875"
time=2025-03-12T06:29:13.590Z level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-03-12T06:29:13.590Z level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
time=2025-03-12T06:29:13.590Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
time=2025-03-12T06:29:13.622Z level=INFO source=runner.go:936 msg="starting go runner"
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory

rocBLAS error: Could not initialize Tensile host: No devices found
time=2025-03-12T06:29:14.091Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error:Could not initialize Tensile host: No devices found"
[GIN] 2025/03/12 - 06:29:14 | 500 |  513.206817ms |       127.0.0.1 | POST     "/v1/chat/completions"
INFO	[_client.py | _send_single_request] HTTP Request: POST http://0.0.0.0:11435/v1/chat/completions "HTTP/1.1 500 Internal Server Error"
INFO	[_base_client.py | _retry_request] Retrying request to /chat/completions in 0.849109 seconds
time=2025-03-12T06:29:14.953Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 gpu=0 parallel=4 available=8441634816 required="6.5 GiB"
time=2025-03-12T06:29:14.953Z level=INFO source=server.go:104 msg="system memory" total="62.0 GiB" free="50.9 GiB" free_swap="4.0 GiB"
time=2025-03-12T06:29:14.954Z level=INFO source=memory.go:356 msg="offload to rocm" projector.weights="595.5 MiB" projector.graph="0 B" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[7.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.5 GiB" memory.required.partial="6.5 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.5 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.6 GiB" memory.weights.nonrepeating="102.6 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="585.0 MiB"
time=2025-03-12T06:29:14.954Z level=INFO source=server.go:376 msg="starting llama server" cmd="/app/plugins/Ollama/lib/ollama/runners/rocm_avx/ollama_llama_server runner --model /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --mmproj /home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-72d6f08a42f656d36b356dbe0920675899a99ce21192fd66266fb7d82ed07539 --threads 8 --parallel 4 --port 41257"
time=2025-03-12T06:29:14.954Z level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-03-12T06:29:14.954Z level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
time=2025-03-12T06:29:14.954Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
time=2025-03-12T06:29:14.991Z level=INFO source=runner.go:936 msg="starting go runner"
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory

rocBLAS error: Could not initialize Tensile host: No devices found
time=2025-03-12T06:29:15.455Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error:Could not initialize Tensile host: No devices found"
[GIN] 2025/03/12 - 06:29:15 | 500 |  513.037994ms |       127.0.0.1 | POST     "/v1/chat/completions"
INFO	[_client.py | _send_single_request] HTTP Request: POST http://0.0.0.0:11435/v1/chat/completions "HTTP/1.1 500 Internal Server Error"
ERROR	[instance_manager.py | generate_chat_title] Error code: 500 - {'error': {'message': 'llama runner process has terminated: error:Could not initialize Tensile host: No devices found', 'type': 'api_error', 'param': None, 'code': None}}
time=2025-03-12T06:29:20.456Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.000532664 model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868
time=2025-03-12T06:29:20.706Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.251026562 model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868
time=2025-03-12T06:29:20.956Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.500505765 model=/home/user/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868

[Jeffser]

I forgot that I had to change the Alpaca's manifest too :)

I'll do it tomorrow since I also want to release some bug fixes

[coolkys]

I'm pretty sure people know this, but to those who may stumble upon this issue not knowing what to do, the best way to use AMD GPUs with Alpaca as it currently stands is to use Ollama as a local server:

1. Install Ollama, their installer script should detect ROCm compatible GPUs and install the necessary components.
2. Install models using Ollama CLI. Using the CLI, ensure that the models work(ollama run <model name>).
3. Turn on Ollama server with the command ollama serve
4. On Alpaca, go to Manage Instances > Click + > Select Ollama
5. In instance URL, use http://0.0.0.0:11434 or http://127.0.0.1:11434.
6. Select this instance. Alpaca should be able to pull the models you installed through Ollama(visible in Manage Models>Added)
7. Use Alpaca as normal.

[MysteriousGitEntity]

So happy it all works again

No. Please read the comments 😐

[mags0ft]

Oof, they were automatically collapsed by GitHub. Didn't say a thing... :/

[0chroma]

It seems like the issue may have something to do with running ollama in a rootless container. Seems that when the group mismatches it throws the error we've been seeing. Relevant comment: ollama/ollama#6685 (comment)

[Sirius3331]

Which ROCm version does the Alpaca AMD Support package have?

[mags0ft]

It appears to be using ROCm from Ollama version 0.6.0.