|
2 | 2 | "cells": [ |
3 | 3 | { |
4 | 4 | "cell_type": "markdown", |
5 | | - "id": "db5dcdaa-b22d-46d8-bed4-d89148a5cdc3", |
| 5 | + "id": "c82d5231-3e32-491a-851c-1c1832ee8af0", |
6 | 6 | "metadata": {}, |
7 | 7 | "source": [ |
8 | | - "# Running Large Language Models\n", |
| 8 | + "# Running LLMs with llama.cpp\n", |
9 | 9 | "\n", |
| 10 | + "In this notebook we will use llama.cpp to execute LLMs (Large Language Models). `llama.cpp` enables model loading and inference on a variety of CPU and GPU platforms including Ryzen AI through ROCm and Vulkan.\n", |
10 | 11 | "\n", |
11 | 12 | "## Goals\n", |
12 | 13 | "\n", |
13 | | - "* Learn to use `llama.cpp` to run inference on the AMD GPU" |
14 | | - ] |
15 | | - }, |
16 | | - { |
17 | | - "cell_type": "markdown", |
18 | | - "id": "c82d5231-3e32-491a-851c-1c1832ee8af0", |
19 | | - "metadata": {}, |
20 | | - "source": [ |
21 | | - "\n", |
22 | | - "# LLMs with llama.cpp\n", |
| 14 | + "* Learn to use `llama.cpp` to run inference on the AMD GPU\n", |
23 | 15 | "\n", |
24 | | - "In this notebook we will use llama.cpp to execute LLMs (Large Language Models). `llama.cpp` enables model loading and inference on a variety of CPU and GPU platforms including Ryzen AI through ROCm and Vulkan.\n", |
25 | | - "\n", |
26 | | - "To launch `llama.cpp`, open a separate terminal and run the command below. `llama.cpp` will pull the model, detect the AMD GPU and start a chat dialog. Try asking the model a question." |
| 16 | + "To launch `llama.cpp`, open a terminal and run the commands below. `llama.cpp` will pull the model, detect the AMD GPU and start a chat dialog. Try asking the model a question." |
27 | 17 | ] |
28 | 18 | }, |
29 | 19 | { |
|
51 | 41 | "id": "aff4aafe-e673-44a6-82b6-82ac23e85763", |
52 | 42 | "metadata": {}, |
53 | 43 | "source": [ |
54 | | - "`llama.cpp` also supports a server/client architecture amd can serve up models from a variety of model zoos. We will cover this in the next notebook." |
| 44 | + "`llama.cpp` also supports a server/client architecture and can serve up models from a variety of model zoos. We will cover this in the next notebook." |
55 | 45 | ] |
56 | 46 | }, |
57 | 47 | { |
|
66 | 56 | "On this platform, `llama.cpp` is compiled with ROCm and Vulkan backends. `llama-bench` is a utility that allows you to benchmark models under multiple backends. Run the code below in a terminal to compare the model execution under ROCm and Vulkan.\n", |
67 | 57 | "\n", |
68 | 58 | "```bash\n", |
69 | | - "llama-bench -m /ryzers/.cache/llamacpp/unsloth_Llama-3.2-3B-Instruct-GGUF_Llama-3.2-3B-Instruct-Q4_K_M.gguf -dev ROCm0,Vulkan0\n", |
| 59 | + "llama-bench -m /root/.cache/llama.cpp/unsloth_Qwen3-1.7B-GGUF_Qwen3-1.7B-Q4_K_M.gguf -dev ROCm0,Vulkan0\n", |
70 | 60 | "```\n", |
71 | 61 | "\n", |
72 | 62 | "You should see output similar to:\n", |
|
90 | 80 | }, |
91 | 81 | { |
92 | 82 | "cell_type": "markdown", |
93 | | - "id": "7300b4f1-1cc2-4049-b127-23aa7cb76155", |
| 83 | + "id": "ee2eb148-0f24-4b8e-a335-07d6510cde87", |
94 | 84 | "metadata": {}, |
95 | 85 | "source": [ |
96 | 86 | "The above output displays for each device, ROCm or Vulkan, the performance in tokens/second **(t/s)**. The performance tests are **pp512** and **tg128**. **pp512** is a prompt processing test and indicates how quickly the model can process prompts of 512 tokens. **tg128** is a token generation test which indicates how quickly the model can generate token." |
97 | 87 | ] |
98 | 88 | }, |
| 89 | + { |
| 90 | + "cell_type": "markdown", |
| 91 | + "id": "7300b4f1-1cc2-4049-b127-23aa7cb76155", |
| 92 | + "metadata": {}, |
| 93 | + "source": [ |
| 94 | + "## Key Takeaways\n", |
| 95 | + "\n", |
| 96 | + "Now you know:\n", |
| 97 | + "- How to run an LLM using `llama.cpp`.\n", |
| 98 | + "- How to compare runtimes for executing your LLM.\n" |
| 99 | + ] |
| 100 | + }, |
| 101 | + { |
| 102 | + "cell_type": "markdown", |
| 103 | + "id": "4949cf4d-d5da-4677-a492-f214e1ac70b0", |
| 104 | + "metadata": {}, |
| 105 | + "source": [ |
| 106 | + "## Next Steps\n", |
| 107 | + "\n", |
| 108 | + "(At home) Try downloading and running more models from https://huggingface.co/models. Be sure to select GGUF format in the *libraries* search box.\n", |
| 109 | + "\n", |
| 110 | + "<img src=\"images/hf.png\" width=\"800\" height=\"200\">\n", |
| 111 | + "\n", |
| 112 | + "When you select a model, click \"Use this model\" and click `llama.cpp` to see the command to run.\n", |
| 113 | + "\n", |
| 114 | + "<img src=\"images/model.png\" width=\"300\" height=\"200\">\n", |
| 115 | + "\n" |
| 116 | + ] |
| 117 | + }, |
99 | 118 | { |
100 | 119 | "cell_type": "markdown", |
101 | 120 | "id": "ac2313db-cc41-4589-b2d9-4974abe55ca7", |
102 | 121 | "metadata": {}, |
103 | 122 | "source": [ |
104 | 123 | "\n", |
105 | 124 | "\n", |
| 125 | + "## Cleanup\n", |
| 126 | + "\n", |
| 127 | + "Make sure to `ctrl+c` and close the terminals before moving onto the next notebook.\n", |
| 128 | + "\n", |
106 | 129 | "## References\n", |
107 | 130 | "\n", |
108 | | - "* [Llama.cpp](https://github.com/ggml-org/llama.cpp)" |
| 131 | + "* [Llama.cpp](https://github.com/ggml-org/llama.cpp)\n", |
| 132 | + "* [Huggingface models](https://huggingface.co/models)" |
109 | 133 | ] |
110 | 134 | }, |
111 | 135 | { |
112 | 136 | "cell_type": "markdown", |
113 | 137 | "id": "b285bb33-051c-4edb-bb43-14e4efa00fde", |
114 | 138 | "metadata": {}, |
115 | 139 | "source": [ |
116 | | - "\n", |
| 140 | + "**Continue to:** [3_vlm_ros_integration](3_vlm_ros_integration.ipynb)\n", |
117 | 141 | "\n", |
118 | 142 | "\n", |
119 | 143 | "---\n", |
120 | 144 | "Copyright© 2025 AMD, Inc SPDX-License-Identifier: MIT" |
121 | 145 | ] |
| 146 | + }, |
| 147 | + { |
| 148 | + "cell_type": "code", |
| 149 | + "execution_count": null, |
| 150 | + "id": "ae918e00-eec7-41b4-ab76-8d617e0cfe3b", |
| 151 | + "metadata": {}, |
| 152 | + "outputs": [], |
| 153 | + "source": [] |
122 | 154 | } |
123 | 155 | ], |
124 | 156 | "metadata": { |
|
0 commit comments